Batch to remove duplicate rows from text file

Question

Is it possible to remove duplicate rows from a text file? If yes, how?

Answer 1

I have used a fake "array" to accomplish this

@echo off
:: filter out all duplicate ip addresses
REM you file would take place of %1
set file=%1%
if [%1]==[] goto :EOF
setlocal EnableDelayedExpansion
set size=0
set cond=false
set max=0
for /F %%a IN ('type %file%') do (   
      if [!size!]==[0] (
          set cond=true
          set /a size="size+1"
          set arr[!size!]=%%a

      ) ELSE (
                 call :inner
                 if [!cond!]==[true] (
                     set /a size="size+1" 
                     set arr[!size!]=%%a&& ECHO > NUL                      
                 ) 
      )
)
break> %file%
:: destroys old output
for /L %%b in (1,1,!size!) do echo !arr[%%b]!>> %file%
endlocal
goto :eof
:inner
for /L %%b in (1,1,!size!) do (  
          if "%%a" neq "!arr[%%b]!" (set cond=true) ELSE (set cond=false&&goto :break)                                
)
:break

the use of the label for the inner loop is something specific to cmd.exe and is the only way I have been successful nesting for loops within each other. Basically this compares each new value that is being passed as a delimiter and if there is no match then the program will add the value into memory. When it is done it will destroy the target files contents and replace them with the unique strings