I've a little problem.
I'm using NodeJS as backend. Now, an user has a field "biography", where the user can write something about himself.
Suppose that this field has 220 maxlength, and suppose this as input:
๐ถ๐ป๐ฆ๐ป๐ง๐ป๐จ๐ป๐ฉ๐ป๐ฑ๐ปโโ๏ธ๐ฑ๐ป๐ด๐ป๐ต๐ป๐ฒ๐ป๐ณ๐ปโโ๏ธ๐ณ๐ป๐ฎ๐ปโโ๏ธ๐ฎ๐ป๐ท๐ปโโ๏ธ๐ท๐ป๐๐ปโโ๏ธ๐๐ป๐ต๐ปโโ๏ธ๐ฉ๐ปโโ๏ธ๐จ๐ปโโ๏ธ๐ฉ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ๐จ๐ปโ๐พ
As you can see there aren't 220 emojis (there are 37 emojis), but if I do in my nodejs server
console.log(bio.length)
where bio is the input text, I got 221. How could I "parse" the string input to get the correct length? Is it a problem about unicode?
SOLVED
I used this library: https://github.com/orling/grapheme-splitter
I tried that:
var Grapheme = require('grapheme-splitter');
var splitter = new Grapheme();
console.log(splitter.splitGraphemes(bio).length);
and the length is 37. It works very well!
str.length gives the count of UTF-16 units. Unicode-proof way to get string length in codepoints (in characters) is [...str].length as iterable protocol split the string to codepoints.
See also:
What every JavaScript developer should know about Unicode
JavaScript has a Unicode problem
function fancyCount2(str){
const joiner = "\u{200D}";
const split = str.split(joiner);
let count = 0;
for(const s of split){
//removing the variation selectors
const num = Array.from(s.split(/[\ufe00-\ufe0f]/).join("")).length;
count += num;
}
//assuming the joiners are used appropriately
return count / split.length;
}
TL;DR there are solutions, but they donโt work in every case. Unicode can feel like a dark art.
There seems to be limitations in various solutions I have seen presented, with the issue going beyond emojis and covering other characters in the Unicode range. Consider รฉ can be stored as รฉ or e + โ, if using combing characters. This can even lead to two strings that look the same not being equal. Also note, in certain cases a single emoji can be 11 characters when stored and as a result 22 bytes, assuming UTF16.
The way this is handled and how characters are combined, or displayed, can even vary between browsers and operating systems. So, while you may think you cracked it, there is a risk another environment breaks this. Be sure to test where it matters.
Now, there is the front-end vs back-end problem: you solved the character count problem so it works well for human users, now your single emoji blows right past the allocated field size in the database. Less of an issue with databases such as mongo, but can be one with SQL databases, where field allocation was conservative. This means how you solve your problem will depend where the hardest limitation comes in.
Note, that a basic solution does involve converting a string to an array and getting the length, accepting limitations:
Array.from(str)
This will fall apart when characters are combined and dealing with astral planes.
A few high level approaches, that take into account limitations:
- use approaches that solve the front-end issue, as best as possible, and then ensure storage issues are resolved
- be more conservative with the advertised front-end limits, if the database or other storage canโt be adjusted
- limit the character types that can be entered
- clearly indicate limitations of the length calculation
Additionally, given the complexity of the issue it may be worth seeing if there is a popular JS library that already deals with this? I did not find one at the time of writing. Hopefully this is something that would become core to Javascript at some point.
Other pages to read:
As you can see from the below example, this is to do with unicode encoding,
There's some great resources such as the one I took this example from.
https://blog.jonnew.com/posts/poo-dot-length-equals-two
console.log("๐ฉโโค๏ธโ๐โ๐ฉ".length === 11);ๆฅๆบ๏ผhttps://stackoverflow.com/questions/54369513/how-to-count-the-correct-length-of-a-string-with-emojis-in-javascript