Compute MD5 hash of a UTF8 string

Question

I have an SQL table in which I store large string values that must be unique. In order to ensure the uniqueness, I have a unique index on a column in which I store a string

Answer 1

SQL Server does not natively support using UTF-8 strings, and it hasn't for quite a while. As you noticed, NCHAR and NVARCHAR use UCS-2 rather than UTF-8.

If you are insistent on using the HASHBYTES function, you must be able to pass the UTF-8 byte[] as VARBINARY from your C# code to preserve the encoding. HASHBYTES accepts VARBINARY in place of NVARCHAR. This could be accomplished with a CLR function that accepts NVARCHAR and returns the results of Encoding.UTF8.GetBytes as VARBINARY.

With that being said, I strongly suggest keeping these types of business rules isolated within your application rather than the database. Especially since the application is already performing this logic.