Working with files and utf8 in PHP

Question

Lets say I have a file called foo.txt encoded in utf8:

aoeu  
qjkx
ñpyf

And I want to get an array that contains all the lines in that file

Answer 1

In UTF-8, ñ is encoded as two bytes. Normally in PHP all string operations are byte-based, so when you preg_split the input it splits up the first byte and the second byte into separate array items. Neither the first byte on its own nor the second byte on its own will match both bytes together as found in $allowed_letters, so it'll never match ñ.

As Yanick posted, the solution is to add the u modifier. This makes PHP's regex engine treat both the pattern and the input line as Unicode characters instead of bytes. It's lucky that PHP has special Unicode support here; elsewhere PHP's Unicode support is extremely spotty.

A simpler and quicker way than splitting would be to compare each line against a character-group regex. Again, this must be a u regex.

if(preg_match('/^[aoeuñpyf]+$/u', $line))
    $lines[]= $line;

Answer 2

It sounds like you've already got your answer, but it is important to recognize that unicode characters can be stored in multiple ways. Unicode normalization* is a process which can help ensure comparisons work as expected.

• http://en.wikipedia.org/wiki/Unicode_equivalence

Answer 3

If you are running Windows, the OS does not save files in UTF-8, but in cp1251 (or something...) by default you need to save the file in that format explicitly or run each line in utf8_encode() before performing your check. I.e.:

$line=utf8_encode(fgets($f));

If you are sure that the file is UTF-8 encoded, is your PHP file also UTF-8 encoded?

If everything is UTF-8, then this is what you need :

foreach(preg_split("//u",$line,-1,PREG_SPLIT_NO_EMPTY) as $letter){
   // ...
}

(append u for unicode chars)

However, let me suggest a yet faster way to perform your check :

$allowed_letters=array("a","o","e","u","ñ","p","y","f");

$lines=array();
$f=fopen("foo.txt","r");
while(!feof($f)){
    $line=fgets($f);

    $line = str_split(rtrim($line));
    if (count(array_intersect($line, $allowed_letters)) == count($line)) {
            $lines[] = $line;
    }
}
fclose($f);

(add space chars to allow space characters as well, and remove the rtrim($line))