Remove Duplicate Lines - Deduplicate Text Online

How to use this tool

1. Paste your text. Paste text into the input box or open a local .txt, .csv, .log, or .tsv file up to 5 MiB. The browser reads the file on your device without uploading it.
2. Configure options. Choose whether the comparison is case-sensitive, trim leading or trailing whitespace, or remove blank and whitespace-only lines. Empty-line removal is off by default.
3. Review the counts. See the original and unique line counts, duplicate removals, and a separate empty-lines-removed count when that option is enabled.
4. Copy the result. Click "Copy result" to copy the deduplicated text to your clipboard.

When to remove duplicate lines

Duplicate lines crop up constantly in real-world text work: log files with repeated entries, email address lists with duplicates from multiple exports, configuration files with copy-paste errors, or word lists merged from multiple sources. Cleaning them manually is tedious and error-prone; this tool does it instantly.

How duplicates are detected

Two lines are considered duplicates when their comparison keys match. The comparison key is determined by the options you choose:

• Case-sensitive (default on): "Hello" and "hello" are different lines. Turn this off to treat them as the same.
• Trim whitespace: with this on, " hello " and "hello" are the same line. The first occurrence is kept with its original spacing.
• Remove empty lines (default off): when enabled, blank lines and lines made only of whitespace are omitted before comparison. Their count is shown separately, so the duplicate count only includes repeated non-empty lines.

Open a local text file

Choose Open file to load a .txt, .csv, .log, or .tsv file up to 5 MiB into the same input box. The browser reads the selected file on your device, and the contents are not uploaded. The Clear button removes both the input text and the selected filename. Larger files can be pasted instead, though very large inputs may run slowly on some devices.

CSV files are handled as newline-separated text: each line is compared as a whole row, and the tool does not parse CSV columns or quoted fields. If a CSV contains quoted values spanning multiple lines, this line-based tool will treat those physical lines separately.

Order is preserved

The first occurrence of each unique line is kept. Later occurrences are removed. The output is in the same order as the original input. This is a stable deduplication.

Duplicate report

Expand the "Show duplicate lines" section to see exactly which lines appeared more than once and how many times each appeared. The list is sorted by frequency, so the most-repeated line appears first.

How the counts work with empty lines

When empty-line removal is off, a blank line and a whitespace-only line are regular values; exact duplicates among them count as duplicate removals. When it is on, blank and whitespace- only lines are removed first and counted separately. The accounting stays clear: original lines equal unique output lines plus duplicate removals plus empty lines removed.

Advanced use cases

• CSV deduplication: open a CSV or paste the entire CSV including headers; the tool treats each row as a line. To deduplicate on a specific column rather than the whole row, sort the CSV by that column first so identical values cluster together.
• URL lists: paste a list of URLs to remove exact duplicates. Note that http:// and https:// versions of the same URL are treated as different values; normalize protocol before pasting if needed.
• Database exports: when exporting query results to a plain-text list for cleanup, paste here to quickly identify repeated IDs, emails, or codes before re-importing.

Keep first vs. keep last

By default this tool keeps the first occurrence of each duplicate and removes all subsequent ones — the standard behavior for most deduplication tasks. Some scenarios require keeping the last occurrence (e.g., a changelog where the most recent entry for a version should win). For keep-last behavior, reverse the list before pasting, run deduplication, then reverse the result.