Why a fast dedupe matters
Email lists, keywords, order IDs, filenames: any manually-built dataset ends up with duplicates. Problems they cause: sending the same email twice, buying the same keyword in Google Ads, inflating report counts. Excel and Sheets can do it, but spinning up a spreadsheet for 200 lines is overkill. Paste here, click, copy.
Typical cases
- Email lists. Before uploading to Mailchimp, Klaviyo or your CRM, dedupe to protect sender reputation.
- SEO keywords. When you merge lists from multiple tools (Ahrefs, Semrush, Search Console), the same keyword shows up three times.
- Unique IDs. Validate that an export has no duplicate IDs before importing into your database.
- Tags and categories. When consolidating taxonomies across sites or CMS versions.
- Logs and IP addresses. Counting unique IPs that hit a page.
"Ignore case" mode
By default deduplication is exact: "john@email.com" and
"John@email.com" are different. For emails (which are case-insensitive
per RFC 5321), turn this on. For technical IDs where casing matters, leave it off.
Trimming whitespace
A silent cause of "duplicates that don't look like duplicates" is leftover whitespace.
"Boston" and "Boston " are different to the algorithm
without normalization. That's why "trim whitespace" is on by default.
The count matters
After processing, we show how many lines went in and how many came out. If the gap is huge (say, 1000 to 100), the source of duplicates is probably broken — maybe an export with a bad join. Worth investigating before you keep using that list in production.
Privacy for sensitive lists
If your list contains customer emails, internal IDs or any confidential data, this tool is safe: nothing is sent to a server. All processing is local JavaScript. Still, remember that pasting customer data into any web tool should comply with your company's policy and, where applicable, with regulations like GDPR or CCPA.