[Unix] Remove duplicate rows from file

[How To] Remove duplicate rows from file

uniq command can be used the consecutive recurring lines in a file.

$ uniq filename > targetfile

There are multiple options available with uniq. They are not limited to and include the following:

uniq -c filename
Prefix lines by the number of occurrences

uniq -d fileanme
Only print duplicate lines

uniq -u fileanme
Only print unique lines

Please note that the uniq command expects the input in sorted format. If the input is not sorted, it will only remove the consecutive recurring rows. To remove recurring rows spread across the file (not necessarily consecutive), one must first sort the input in the below way.

$ sort filename | uniq > targetfile

Happy Scripting!!