How to convert Windows end of line in Unix end of line (CR/LF to LF)

Question

I\'m a Java developer and I\'m using Ubuntu to develop. The project was created in Windows with Eclipse and it\'s using the Windows-1252 encoding.

To convert to UTF-8

Answer 1

Did you try the python script by Bryan Maupin found here ? (I've modified it a little bit to be more generic)

#!/usr/bin/env python

import sys

input_file_name = sys.argv[1]
output_file_name = sys.argv[2]

input_file = open(input_file_name)
output_file = open(output_file_name, 'w')

line_number = 0

for input_line in input_file:
    line_number += 1
    try:  # first try to decode it using cp1252 (Windows, Western Europe)
        output_line = input_line.decode('cp1252').encode('utf8')
    except UnicodeDecodeError, error:  # if there's an error
        sys.stderr.write('ERROR (line %s):\t%s\n' % (line_number, error))  # write to stderr
        try:  # then if that fails, try to decode using latin1 (ISO 8859-1)         
            output_line = input_line.decode('latin1').encode('utf8')
        except UnicodeDecodeError, error:  # if there's an error
            sys.stderr.write('ERROR (line %s):\t%s\n' % (line_number, error))  # write to stderr
            sys.exit(1)  # and just keep going
    output_file.write(output_line)

input_file.close()
output_file.close()

You can use that script with

$ ./cp1252_utf8.py file_cp1252.sql file_utf8.sql