Using str_word_count for UTF8 texts

Question

I have this text:

$text  = \"Başka, küskün otomobil kaçtı buraya küskün otomobil neden kaçtı
          kaçtı buraya, oraya KISMEN @here #there J.J.Johanson h         


        

Answer 1

You will never have a prefect solution of word-count, because word-count concept is not exists or too difficult in some languages. UTF8 or not does not matter.

Japanese and Chinese are not space tokenism language. They even don't have a static word list, you have to read the whole sentence before find verb and noun.

If you want to support multiple languages, you will need language specific tokenizer engine. You may research full-text index, tokenizer, CJK-tokenizer, CJK-analyzer for more information.

If you only want to support limited selected languages, just improve your regex patters with more and more cases.