ios app compile sqlite fts with icu,but it cant get the perfect answer when i input a letter like “z”

Question

In sqlite I:

1. Perform a create virtual MyTable (tokenize =icu ,id text,subject text,abstract text)
2. Then successfully insert info MyT

Answer 1

You may also try FMDB's FMSimpleTokenizer. FMSimpleTokenizer uses build-in CFStringTokenizer and according to apple document "CFStringTokenizer allows you to tokenize strings into words, sentences or paragraphs in a language-neutral way. It supports languages such as Japanese and Chinese that do not delimit words by spaces"

If you check FMSimpleTokenizer code, you will find that is done by calling CFStringTokenizerAdvanceToNextToken & CFStringTokenizerGetCurrentTokenRange.

One interesting "fact" is how CFStringTokenizer tokenizes the Chinese words, for example "欢迎使用" will be tokenize into "欢迎" & "使用", which totally makes sense, but if you search "迎", you will be surprised to see no result at all!

In that case you probably need to write a tokenizer like Hai Feng Kao's sqlite tokenizer.