How to count the correct length of a string with emojis in javascript?

问题

I've a little problem.

I'm using NodeJS as backend. Now, an user has a field "biography", where the user can write something about himself.

Suppose that this field has 220 maxlength, and suppose this as input:

👶🏻👦🏻👧🏻👨🏻👩🏻👱🏻‍♀️👱🏻👴🏻👵🏻👲🏻👳🏻‍♀️👳🏻👮🏻‍♀️👮🏻👷🏻‍♀️👷🏻💂🏻‍♀️💂🏻🕵🏻‍♀️👩🏻‍⚕️👨🏻‍⚕️👩🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾👨🏻‍🌾

As you can see there aren't 220 emojis (there are 37 emojis), but if I do in my nodejs server

console.log(bio.length)

where bio is the input text, I got 221. How could I "parse" the string input to get the correct length? Is it a problem about unicode?

SOLVED

I used this library: https://github.com/orling/grapheme-splitter

I tried that:

var Grapheme = require('grapheme-splitter');
var splitter = new Grapheme();
console.log(splitter.splitGraphemes(bio).length);

and the length is 37. It works very well!

回答1:

str.length gives the count of UTF-16 units. Unicode-proof way to get string length in codepoints (in characters) is [...str].length as iterable protocol split the string to codepoints.

回答2:

function fancyCount2(str){
  const joiner = "\u{200D}";
  const split = str.split(joiner);
  let count = 0;

  for(const s of split){
    //removing the variation selectors
    const num = Array.from(s.split(/[\ufe00-\ufe0f]/).join("")).length;
    count += num;
  }

  //assuming the joiners are used appropriately
  return count / split.length;
}

回答3:

TL;DR there are solutions, but they don’t work in every case. Unicode can feel like a dark art.

There seems to be limitations in various solutions I have seen presented, with the issue going beyond emojis and covering other characters in the Unicode range. Consider é can be stored as é or e + ‘, if using combing characters. This can even lead to two strings that look the same not being equal. Also note, in certain cases a single emoji can be 11 characters when stored and as a result 22 bytes, assuming UTF16.

The way this is handled and how characters are combined, or displayed, can even vary between browsers and operating systems. So, while you may think you cracked it, there is a risk another environment breaks this. Be sure to test where it matters.

Now, there is the front-end vs back-end problem: you solved the character count problem so it works well for human users, now your single emoji blows right past the allocated field size in the database. Less of an issue with databases such as mongo, but can be one with SQL databases, where field allocation was conservative. This means how you solve your problem will depend where the hardest limitation comes in.

Note, that a basic solution does involve converting a string to an array and getting the length, accepting limitations:

Array.from(str)

This will fall apart when characters are combined and dealing with astral planes.

A few high level approaches, that take into account limitations:

use approaches that solve the front-end issue, as best as possible, and then ensure storage issues are resolved
be more conservative with the advertised front-end limits, if the database or other storage can’t be adjusted
limit the character types that can be entered
clearly indicate limitations of the length calculation

Additionally, given the complexity of the issue it may be worth seeing if there is a popular JS library that already deals with this? I did not find one at the time of writing. Hopefully this is something that would become core to Javascript at some point.

回答4:

As you can see from the below example, this is to do with unicode encoding,

There's some great resources such as the one I took this example from.

https://blog.jonnew.com/posts/poo-dot-length-equals-two

console.log("👩‍❤️‍💋‍👩".length === 11);

来源：https://stackoverflow.com/questions/54369513/how-to-count-the-correct-length-of-a-string-with-emojis-in-javascript

标签

javascript

node.js

emoji