How to remove duplicate rows from a CSV
Drop the file and choose what makes a row a duplicate: the whole row, or one or more columns. To find repeated customers by email address, select only the email column; to find fully identical rows, select nothing. The first row with each value is kept and later ones are dropped, and the result reports how many were removed.
Removing duplicate rows in Excel is limited by its million-row ceiling and by memory. Here, rows are streamed and compared by a compact hash, so the rows themselves are never held in memory and the file can be far larger than Excel would open.
Comparing by column
- Select the columns that define a duplicate; the others are ignored in the comparison but kept in the output.
- Ignoring leading and trailing spaces catches “a@example.com” and “a@example.com ” as the same.
- Comparison is exact and case-sensitive, so “Ali” and “ali” are different; normalise case first if you want them treated as the same.
- The header row is never treated as a duplicate.
How exact is it?
Rows are compared by a 53-bit hash rather than stored, which keeps memory flat. The chance that two different rows are mistaken for duplicates is roughly n squared divided by 2 to the 54th power, which is negligible below a few million distinct rows and still very small at ten million. For tax or legal data, spot-check the removal count against a second method.
Messy files, handled
- Quoted fields that contain commas, doubled quotes and line breaks are read correctly, so a note with a newline in it does not split into two rows.
- Comma, semicolon, tab and pipe delimiters are detected from the start of the file, including the semicolons European exports use.
- A byte-order mark, and Windows (CRLF) or Unix (LF) line endings, are handled.
- Rows with more or fewer fields than the header are counted and shown, not silently dropped.
- Files with no header row are supported with one checkbox.
- UTF-8 files are supported. A UTF-16 file needs saving as UTF-8 first, and the tool tells you so.
How to use it
- Drop the CSV file onto the box.
- Choose the columns that define a duplicate, or leave none selected for the whole row.
- Click Remove duplicate rows.
- Download the deduplicated file.
Frequently asked questions
Which duplicate is kept?
The first one in the file. Later rows with the same key are removed.
Can I remove duplicates based on one column?
Yes. Select that column, and rows are compared on it alone. Everything else in the row is kept for the first occurrence.
Is the comparison case-sensitive?
Yes. Values must match exactly, apart from optional trimming of leading and trailing spaces when comparing by column.
How large a CSV file can it handle?
The largest file it has been run on is 1 GB, opened and scrolled in a real browser. It streams the file instead of loading it, so memory grows with the number of rows (about 8 bytes each for the index), not with the size of the file. Very large row counts, roughly above a hundred million, may exhaust a browser tab’s memory.
Is my file uploaded anywhere?
No. The file is read in pieces by code running in your browser and is never sent to a server. The site records only an anonymous counter with the operation and rough size buckets, such as “100–500 MB” and “1M–10M rows” — never file names, column names or any data.