Skip to: Site menu | Main content


Linux – Removing duplicate lines using uniq utility

I think that one of the biggest advantages Linux has over Windows is the powerful command-line utilities and scripting. You can do some really cool things with Linux.
Of course Windows has its advantages, and naturally Windows XP is what I use on my work PC, so in order to enjoy the power of Linux utilities in XP, I’m using Cygwin.

Long story short – I’ve just discovered the coolest little Linux command that has just saved me precious minutes of tedious work: uniq.
It does the simplest thing – It gets lines of text as input, and outputs these lines of texts, without duplicate lines..!
So simple – Yet how the hell would I do that in Windows?!? I literally spent 15 minutes playing around with Excel and PSPad to try and figure out how to do it.

Then it struck me – A Linux command MUST exist that does just that. A quick Google search, and yep – Linux has uniq.

So although Cygwin is worth its own lengthy post, I’ll do it here since it’s important.
Guys – Whether you use Linux or XP. Think Linux is cool or not. Linux expert or not:
If you are using Windows XP, you MUST install Cygwin, and enjoy all the power of Linux on your Windows XP machine.

3 Responses to “Linux – Removing duplicate lines using uniq utility”

  1. Aviv Says:

    In PSPAD:
    Edit -> Sort -> Check “Remove Duplicates”.

  2. Avi Says:

    Damn – I simply KNEW it was there somewhere…
    Nevermind, I’m still happy about discovering uniq..! ;-)

  3. Rick S. Says:

    Be careful. This is from the “man uniq” –

    Duplicated lines must be adjacent. (Before issuing the uniq command, use the sort command to make all duplicate lines adjacent.)

    Considering this, what really is the point of “uniq”? If you’re supposed to sort the stream before running it, why not just “sort -u” the stream and be done with it?

Leave a Reply