Why does my tool output overwrite itself and how do I fix it?

Question

The intent of this question is to provide an answer to the daily questions whose answer is \"you have DOS line endings\" so we can simply close them as duplicates of this on

Answer 1

The problem is that your input file uses DOS line endings of CRLF instead of UNIX line endings of just LF and you are running a UNIX tool on it so the CR remains part of the data being operated on by the UNIX tool. CR is commonly denoted by \r and can be seen as a control-M (^M) when you run cat -vE on the file while LF is \n and appears as $ with cat -vE.

So your input file wasn't really just:

what isgoingon

it was actually:

what isgoingon\r\n

as you can see with cat -v:

$ cat -vE file
what isgoingon^M$

and od -c:

$ od -c file
0000000   w   h   a   t       i   s   g   o   i   n   g   o   n  \r  \n
0000020

so when you run a UNIX tool like awk (which treats \n as the line ending) on the file, the \n is consumed by the act of reading the line, but that leaves the 2 fields as:

Note the \r at the end of the second field. \r means Carriage Return which is literally an instruction to return the cursor to the start of the line so when you do:

print $2, $1

awk will print isgoingon and then will return the cursor to the start of the line before printing what which is why the what appears to overwrite the start of isgoingon.

To fix the problem, do either of these:

dos2unix file
sed 's/\r$//' file
awk '{sub(/\r$/,"")}1' file
perl -pe 's/\r$//' file

Apparently dos2unix is aka frodos in some UNIX variants (e.g. Ubuntu).

Be careful if you decide to use tr -d '\r' as is often suggested as that will delete all \rs in your file, not just those at the end of each line.

Note that GNU awk will let you parse files that have DOS line endings by simply setting RS appropriately:

gawk -v RS='\r\n' '...' file

but other awks will not allow that as POSIX only requires awks to support a single character RS and most other awks will quietly truncate RS='\r\n' to RS='\r'. You may need to add -v BINMODE=3 for gawk to even see the \rs though as the underlying C primitives will strip them on some platforms, e.g. cygwin.

One thing to watch out for is that CSVs created by Windows tools like Excel will use CRLF as the line endings but can have LFs embedded inside a specific field of the CSV, e.g.:

"field1","field2.1
field2.2","field3"

is really:

"field1","field2.1\nfield2.2","field3"\r\n

so if you just convert \r\ns to \ns then you can no longer tell linefeeds within fields from linefeeds as line endings so if you want to do that I recommend converting all of the intra-field linefeeds to something else first, e.g. this would convert all intra-field LFs to tabs and convert all line ending CRLFs to LFs:

gawk -v RS='\r\n' '{gsub(/\n/,"\t")}1' file

Doing similar without GNU awk left as an exercise but with other awks it involves combining lines that do not end in CR as they're read.