I am trying to use this command:
sed -i \'s#\\{test1\\}#test2#\' /example/myfile.txt
To replace instances of {test1} with
You aren't escaping the curly braces at all. In sed, the default regular expressions are BREs, where \{ and \} indicate a range expression. Since test1 isn't a range, your BRE is incorrect.
To fix it, you can either drop the backslashes (braces aren't special in BREs) or keep it the same and tell sed to use EREs (-r flag with GNU sed, -E flag with BSD/MacOSX sed).
sed -i 's#{test1}#test2#' /example/myfile.txt
You don't need escape {}