Color similarity/distance in RGBA color space

Question

How to compute similarity between two colors in RGBA color space? (where the background color is unknown of course)

I need to remap an RGBA image to

Answer 1

First of all, a very interesting problem :)
I don't have a full solution (at least not yet), but there are 2 obvious extreme cases we should consider:
When Δa==0 the problem is similiar to RGB space
When Δa==1 the problem is only on the alpha 1-dim space
So the formula (which is very similar to the one you stated) that would satisfy that is:
(Δr² + Δg² + Δb²) × (1-(1-Δa)²) + Δa² or (Δr² + Δg² + Δb²) × (1-Δa²) + Δa²

In any case, it would probably be something like (Δr² + Δg² + Δb²) × f(Δa) + Δa²

If I were you, I would try to simulate it with various RGBA pairs and various background colors to find the best f(Δa) function. Not very mathematic, but will give you a close enough answer