Batch to remove duplicate rows from text file

Question

Is it possible to remove duplicate rows from a text file? If yes, how?

Answer 1

The Batch file below do what you want:

@echo off
setlocal EnableDelayedExpansion
set "prevLine="
for /F "delims=" %%a in (theFile.txt) do (
   if "%%a" neq "!prevLine!" (
      echo %%a
      set "prevLine=%%a"
   )
)

If you need a more efficient method, try this Batch-JScript hybrid script that is developed as a filter, that is, similar to Unix uniq program. Save it with .bat extension, like uniq.bat:

@if (@CodeSection == @Batch) @then

@CScript //nologo //E:JScript "%~F0" & goto :EOF

@end

var line, prevLine = "";
while ( ! WScript.Stdin.AtEndOfStream ) {
   line = WScript.Stdin.ReadLine();
   if ( line != prevLine ) {
      WScript.Stdout.WriteLine(line);
      prevLine = line;
   }
}

Both programs were copied from this post.