How to escape history expansion exclamation mark ! inside a double quoted string?

Question

EDIT: the command substitution is not necessary for the surprising behavior, although it is the most common use case. The same question applies to just echo \"\'!b\'\"

Answer 1

If History Expansion is enabled, you can only echo the ! character if it is put in single quotes, escaped or if followed by a whitespace character, carriage return, or =.

From man bash:

   Only backslash (\) and single quotes can  quote  the  history
   expansion character.

   Several  characters inhibit history expansion if found immediately fol-
   lowing the history expansion character, even if it is unquoted:  space,
   tab,  newline,  carriage return, and =.

I believe the key word here is “Only”. The examples provided in the question only consider the outer most quotes being double quotes.