file-format

A customer is sending me a .csv file where the line breaks are made up of the sequence 0xD 0xD 0xA . As far as I know line breaks are either 0xA from Mac or Unix or 0xD 0xA from Windows. Is the 0xD 0xD 0xA any known encoding? Is there any known sequence of savings that corrupts a file's line endings that causes this (I think the customer uses a Mac)? The file doesn't start with any encoding markers, it starts with the text contents directly. The text is displayed correctly if opened with code page 1252. The CRCRLF is known as result of a Windows XP notepad word wrap bug . For future reference,

When I'm using spark, I sometimes run into one huge file in a HIVE table, and I sometimes am trying to process many smaller files in a HIVE table. I understand that when tuning spark jobs, how it works depends on whether or not the files are splittable . In this page from cloudera, it says that we should be aware of whether or not the files are splittable: ...For example, if your data arrives in a few large unsplittable files... How do I know if my file is splittable? How do I know the number of partitions to use if the file is splittable ? Is it better to err on the side of more partitions if

I have a few different JPEG images I've been testing with. As far as I've seen the 0th and first bytes are always 0xFF and 0xD8 . The second and third are usually either 0xFF and 0xE0 ( APP0 ) indicating either a JFIF segment or JFIF extension segment or 0xFF and 0xE1 ( APP1 ) indicating an EXIF segment. My question: is this always the case? Are the 2nd and 3rd bytes always APP0 or APP1? No. There are e.g. several cameras that create JPEGs without these markers, or with other APP markers. The only thing you can rely on is the SOI sequence, FF D8 , not even EOI is produced by all cameras. Also

What is the BMP format for Gray scale Images (especially for 16 bit per pixel) ? The wikipedia just talks about colour images for bmp. Update: Just for an update and information for future visitors, I am going for PGM as this is uncompressed and can support 16 bit gray-scale. Another option was to use PNG , but it compresses the data (which is not what I want) as discussed here . Also note that the image may appear distorted, since most of the monitors support 256 colors and not 4096 for 16 bit. So the Image will be saturated. It was though surprising to know that BMP is almost helpless in

What does the following mean ? diff -rBNu src.orig/java/org/apache/nutch/analysis/NutchAnalysisConstants.java src/java/org/apache/nutch/analysis/NutchAnalysisConstants.java --- src.orig/java/org/apache/nutch/analysis/NutchAnalysisConstants.java 2009-03-10 11:34:01.000000000 -0700 +++ src/java/org/apache/nutch/analysis/NutchAnalysisConstants.java 2009-03-10 14:11:55.000000000 -0700 @@ -4,9 +4,12 @@ + int CJK = 21; + int DIGIT = 22; int DEFAULT = 0; String[] tokenImage = { "<EOF>", + "\"OR\"", "<WORD>", "<ACRONYM>", "<SIGRAM>", @@ -39,6 +42,8 @@ "\"\\\"\"", "\":\"", "\"/\"", + "\"(\"", + "\")\""

I am starting work on a new piece of software that will end up needing some robust and expandable file IO. There are a lot of formats out there. XML, JSON, INI, etc. However, there are always plusses and minuses so I thought I would ask for some community input. Here are some rough requirements: The format is a "standard"...I don't want to reinvent the wheel if I don't have to. It doesn't have to be a formal IEEE standard, but something you could Google and get some information on as a new user, may have some support tools (editors) beyond vi. (Though the software users will generally be