Read CSV file in pyspark with ANSI encoding

问题

I am trying to read in a csv/text file that requires it to be read in using ANSI encoding. However this is not working. Any ideas?

mainDF= spark.read.format("csv")\
                  .option("encoding","ANSI")\
                  .option("header","true")\
                  .option("maxRowsInMemory",1000)\
                  .option("inferSchema","false")\
                  .option("delimiter", "¬")\
                  .load(path)

java.nio.charset.UnsupportedCharsetException: ANSI

The file is over 5GB hence the spark requirement.

I have also tried ANSI in lower case

回答1:

ISO-8859-1 is the same as ANSI so replace that as above

来源：https://stackoverflow.com/questions/59645851/read-csv-file-in-pyspark-with-ansi-encoding