Incorrect string value: '\xEF\xBF\xBD' for column

Question

I have a table I need to handle various characters. The characters include Ø, ® etc.

I have set my table to utf-8 as the default collation, all columns use table de

Answer 1

To those who have a similar problem using PHP, try the function utf8_encode($string). It just works!

Answer 2

\xEF\xBF\xBD is the UTF-8 encoding for the unicode character U+FFFD. This is a special character, also known as the "Replacement character". A quote from the wikipedia page about the special unicode characters:

The replacement character � (often a black diamond with a white question mark) is a symbol found in the Unicode standard at codepoint U+FFFD in the Specials table. It is used to indicate problems when a system is not able to decode a stream of data to a correct symbol. It is most commonly seen when a font does not contain a character, but is also seen when the data is invalid and does not match any character:

So it looks like your data source contains corrupted data. It is also possible that you try to read the data using the wrong encoding. Where do the lines come from?

If you can't fix the data, and your input indeed contains invalid characters, you could just remove the replacement characters:

lines[n] = lines[n].Replace("\xFFFD", "");

Answer 3

Mattmanser is right, never write a sql query by concatenating the parameters directly in the query. An example of parametrized query is:

string lastname = "Doe";
double height = 6.1;
DateTime date = new DateTime(1978,4,18);

var connection = new MySqlConnection(connStr);

try
{
    connection.Open();

    var command = new MySqlCommand(
        "SELECT * FROM tblPerson WHERE LastName = @Name AND Height > @Height AND BirthDate < @BirthDate", connection);

    command.Parameters.AddWithValue("@Name", lastname);
    command.Parameters.AddWithValue("@Height", height);
    command.Parameters.AddWithValue("@Name", birthDate);

    MySqlDataReader reader = command.ExecuteReader();
    ...
}
finally
{
    connection.Close();
}