发表新帖
发表新帖

What is “ANSI as UTF-8” and how can I make fputcsv() generate UTF-8 w/BOM?

前端 未结
关注
4 588
离开以前
离开以前 2020-12-08 11:11

I made a PHP script that generates CSV files that were previously generated by another process. And then, the CSV files have to be imported by yet another process.

T

相关标签:
4条回答
  • 渐次进展
    2020-12-08 11:24

    It is worth noting that ANSI as UTF-8, i.e. UTF-8 without the BOM is useful if you are formatting your PHP files as UTF-8. If your PHP file is outputting html to the browser then the BOM is included in the HTML output which the w3c validator explicitly warns against:

    Byte-Order Mark found in UTF-8 File.

    The Unicode Byte-Order Mark (BOM) in UTF-8 encoded files is known to cause problems for some text editors and older browsers. You may want to consider avoiding its use until it is better supported.

    Further to this, I spotted that the BOM confuses Firefox's Firebug which now thinks that all your <head> content is actually in the <body> tag.

    0 讨论(0)
  • 走了就别回头了
    2020-12-08 11:29

    There's nothing wrong with the file. "ANSI as UTF-8" means there's no BOM but Notepad++ has definitely identified the encoding as UTF-8 by analyzing byte patterns. I tested this by creating a file with Russian, Greek and Polish text in it and saving it as UTF-8 without a BOM. Here it is: 

    # Russian
Следующая

# Greek
Επόμενη

# Polish
Więcej

    I did this in a different editor (EditPad Pro) and used hex mode to make sure the BOM wasn't there. When I opened it in NPP it showed the encoding as "ANSI as UTF-8" and all of the characters displayed correctly. Then, still in hex mode, I removed the first byte of the first Russian character. When I opened it in NPP again, it showed the encoding as "ANSI" and displayed the non-ASCII parts of the text as mojibake:

    ; Russian
¡Ð»ÐµÐ´ÑƒÑŽÑ‰Ð°Ñ

; Greek
Επόμενη

; Polish
Więcej

    Back to EditPad, and this time I added a BOM but didn't repair the Cyrillic character. This time NPP reported the encoding as "UTF-8" and everything displayed correctly except that first Russian character, as shown below. "A1" is the hex representation of what should have been the second byte of that character in UTF-8. It was displayed in an inverted color scheme to indicate an error.

    # Russian
A1ледующая

# Greek
Επόμενη

# Polish
Więcej

    To summarize: In the absence of a BOM, Notepad++ looks for bytes that can't represent ASCII characters because their values are greater than 127 (or 7F hex). If it finds any, but they all conform to the patterns required by UTF-8, it decodes the file as UTF-8 and reports the encoding in the status bar as "ANSI as UTF-8".

    But if it finds even one byte that doesn't toe the UTF-8 line, it decodes the file as "ANSI", meaning the default single-byte encoding for the underlying platform. If your file had been corrupted, that's what you would be seeing.

    EDIT: Although your file is valid without it, you could add a BOM by manually writing the three bytes "EF BB BF" at the very beginning of the file--but there should be a better way. How are you generating the content now? Because it is UTF-8, with at least one non-ASCII character in there somewhere; otherwise, NPP would report it as "ANSI".

    Another possibility to consider: if you have any influence over the process that consumes your CSV file, maybe you can configure it to expect UTF-8 without a BOM. Technically, any software that can decode UTF-8 with a BOM but not without one is broken. The Unicode Consortium actually discourages use of the UTF-8 BOM, not that anyone's listening.

    0 讨论(0)
  • 野趣味
    2020-12-08 11:44

    According to the Notepad++ related threads here and here, 'ANSI as UTF-8' indicates UTF-8 without BOM, while a plain 'UTF-8' means UTF-8 with BOM. So maybe the process reading the CSV needs the Byte-order mark to correctly read the CSV as UTF-8.

    But before going into that, make sure that your script actually writes UTF-8! When you open the new CSVs in Notepad++ (and it says 'ANSI as UTF-8'), are all 'special' characters displayed correctly? If not, you need to adapt your script to actually write UTF-8, if yes, check for the BOM difference.

    0 讨论(0)
  • 陌清茗
    2020-12-08 11:44

    Try changing your PHP script to UTF-8 too. Sometimes it is necessary (despite it can be bypassed) to have the script in the same char encoding of the data.

    Similar problem: PHP: Explode using special characters

    0 讨论(0)
 看不清?
提交回复
热议问题