Why does embedding vector multiplied by a constant in Transformer model?

Question

I am learning to apply Transform model proposed by Attention Is All You Need from tensorflow official document Transformer model for language understanding

Answer 1

I believe the reason for this scaling has nothing to do with the scale applied at the attention layers. It is likely because the transformer shares the weights of the embedding layer and the output softmax. The scales you would use for the embeddings is different than the scale you use for a fully connected layer.

Some implementations of the transformer use this scaling even though they don't actually share the embedding weights at the output layer, but that is probably kept there for consistency (or by mistake). Just make sure that the initialization of your embeddings is consistent.