peartools

Remove Duplicate Lines Online

Paste any list and clear out repeated lines in one click. Keep one copy of each entry, ignore case, trim spaces, and drop blank lines as you go. Everything runs in your browser, so your text stays on your device.

0characters0words0lines
Result
0characters0words0lines

About removing duplicate lines

What does removing duplicate lines do?

Removing duplicate lines takes a list where each entry sits on its own line and keeps only one copy of each. It's the quick way to clean up a messy list of emails, keywords, or IDs without scrolling through it by hand. You can keep the first or last copy of each line, ignore case, trim extra spaces, drop blank lines, and sort what's left, all in one click.

Why use our duplicate line remover

  • ·Free. No signup, no paywall, no limits.
  • ·Runs in your browser. Nothing gets uploaded, your text stays with you.
  • ·Keeps your order. Removes repeats without shuffling the lines you keep.
  • ·Cleans as it removes duplicates. Ignore case, trim spaces, and drop blank lines in the same step.
  • ·Shows the count. Tells you how many duplicate lines were removed.

Duplicate line remover options

Ignore case

Treats uppercase and lowercase as the same, so 'Apple' and 'apple' count as duplicates and only one is kept. Turn it off to keep lines that differ only by capitalization.

BeforeApple↵ apple↵ APPLEAfterApple

Trim lines

Removes spaces from the start and end of each line before comparing. Useful when copied lines have uneven indentation, so 'apple' and ' apple' are treated as the same entry.

Beforeapple↵ apple↵ apple Afterapple

Remove blank lines

Clears out empty lines so they don't fill the result. Without it, one blank line is kept like any other unique entry.

Beforepear↵ ↵ apple↵ ↵ Afterpear↵ apple

Keep first

Keeps the first time a line appears and drops later copies. Turn it off to keep the last copy instead. Either way the order of the remaining lines is preserved.

Beforered↵ blue↵ red↵ greenAfterred↵ blue↵ green

Sort result

Orders the unique lines alphabetically after the duplicates are removed. Leave it off to keep the lines in their original order.

Beforecherry↵ apple↵ cherry↵ bananaAfterapple↵ banana↵ cherry

Why some duplicate lines are not removed

Two lines only count as duplicates when they match exactly, after the options you turned on. If lines look the same but both stay in the result, one of these is usually the reason:

  • ·Spaces at the start or end. A trailing space is invisible but makes the line different. Turn on Trim lines, which also removes non-breaking spaces at the ends.
  • ·Different capitals.Paris and paris are different unless Ignore case is on.
  • ·Hidden characters. Text copied from web pages, PDFs, or AI chats can contain zero-width spaces that you cannot see. Run the text through the AI text cleaner first, then remove duplicates.
  • ·Different punctuation. Curly quotes and straight quotes, or a hyphen and a dash, are different characters. The AI text cleaner can switch them all to plain versions.
  • ·Extra spaces in the middle. Two spaces or a tab between words is not the same as one space. Trim lines only cleans the ends of each line.

Windows and Mac line endings are handled for you, so a list that mixes both still compares correctly.

How to remove duplicates in Excel, Google Sheets, and the terminal

If your list is already in another app, these built-in options also work:

  • ·Excel. Select the column, then choose Data, then Remove Duplicates.
  • ·Google Sheets. Choose Data, then Data cleanup, then Remove duplicates. Or use =UNIQUE(A:A) to list the unique values in another column without changing the original.
  • ·Notepad++. Choose Edit, then Line Operations, then Remove Duplicate Lines.
  • ·Mac and Linux terminal.sort -u list.txt removes duplicates but also sorts the lines. To keep the original order, use awk '!seen[$0]++' list.txt.

When to remove duplicate lines

Dropping repeated lines saves time whenever a list needs to be unique or clean. Here are the most common reasons people use a duplicate line remover:

  • ·Cleaning an email or contact list. Remove repeated addresses so each person only appears once before you send.
  • ·Cleaning up a keyword list. Turn a messy dump of keywords or tags into a clean, unique set.
  • ·Remove log or export data. Strip out repeated rows or IDs pasted from a spreadsheet or log file.
  • ·Cleaning imports and config. Remove duplicate import lines or environment keys so entries stay unique.
  • ·Building a unique word list. Combine several lists, remove the overlaps, and sort what's left in one pass.

Related text tools

Frequently asked questions

How do I remove duplicate lines?
Paste your list into the box above, choose your options, then click Remove duplicates. The tool keeps one copy of each line and drops the rest, and tells you how many lines were removed. You can copy the result or download it as a .txt file.
Does it keep the original order of my lines?
Yes. By default the lines stay in the order you pasted them, with only the repeats removed. Turn on 'Sort result' if you also want the unique lines ordered alphabetically.
Which copy of a duplicate line is kept?
By default the first copy is kept and later ones are removed. Turn off 'Keep first' to keep the last copy of each line instead.
Can I ignore case when finding duplicates?
Yes. Turn on 'Ignore case' so 'Apple' and 'apple' count as the same line. With it off, lines that differ only by capitalization are treated as separate entries.
What about extra spaces around lines?
Turn on 'Trim lines' to remove spaces from the start and end of each line before comparing. That way 'apple' and ' apple' are treated as duplicates instead of two different lines.
Is the remove duplicate lines tool free?
Yes, it's completely free with no signup, no limits, and no watermarks. Use it as often as you like.
Is my text safe and private?
Your text never leaves your browser. All processing runs locally on your device, so nothing is uploaded to a server. Once the page has loaded you can even use it offline.