Using str_word_count for UTF8 texts

Question

I have this text:

$text  = \"Başka, küskün otomobil kaçtı buraya küskün otomobil neden kaçtı
          kaçtı buraya, oraya KISMEN @here #there J.J.Johanson h         


        

Answer 1

I think you're sort of on the right track with explode, but that doesn't handle regex.

Change your code to:

$namePattern = '/[\s,:?!]+/u';
$wordsArray = preg_split($namePattern, $text, -1, PREG_SPLIT_NO_EMPTY);
$wordsArray2 = array_count_values($wordsArray);
arsort($wordsArray2);
print_r($wordsArray2);

Of course you may need to tweak the regex ($regexPattern) to meet your needs.

Fiddle: http://ideone.com/JoIJqv

Answer 2

You will never have a prefect solution of word-count, because word-count concept is not exists or too difficult in some languages. UTF8 or not does not matter.

Japanese and Chinese are not space tokenism language. They even don't have a static word list, you have to read the whole sentence before find verb and noun.

If you want to support multiple languages, you will need language specific tokenizer engine. You may research full-text index, tokenizer, CJK-tokenizer, CJK-analyzer for more information.

If you only want to support limited selected languages, just improve your regex patters with more and more cases.