The common mixing of RGB colors is very different from mixing colors for paintings, it\'s mixing of light instead mixing of pigments.
For example:
Bl
Check this implementation for additive, substractive and other mixing alghoritms.
Is fully functional (writed in java), so you can test whatever colors you need to mix, and see if it fit your needs.
As other responses pointed, Blue + Yellow (exactly Cyan + Yellow) is Green on substractive CMYK alghoritm. See by yourself
I think your problem with combining hues is that you are doing it by adding together the two angles and dividing by two. As you've noticed, the result often makes no sense. I think you'd be better off converting the angles to Cartesian coordinates on the unit circle, averaging those, and finding the angle of the resulting point (ignoring the magnitude).
There is code to mix colors in a realistic way in krita: https://projects.kde.org/projects/calligra/repository/revisions/master/show/krita/plugins/extensions/painterlyframework.
Note that the code including the illuminants file is GPLv2+. It can convert from RGB to wavelengths, do composition and convert back.
Wondering if calculation of inversion of the RGB value work. Since it's about subtraction of lights, technically the subtraction part can be calculated by simple math.
For example cyan + yellow
cyan = 0x00ffff yellow = 0xffff00
Their inversions are 0xff0000 and 0x0000ff, meaning they absorbed red and blue lights completely. Their 1:1 mixture should absorb half of red and blue lights (since the other half of the mixture can still reflects some red and blue light), which is consistent with (0xff0000 + 0x00ffff) / 2 = 0x7f007f. Now we subtract the value from 0xffffff we have 0x80ff80 which is green!
there are two different possibilities combining colors:
additive mixing (like RGB)
subtractive mixing (like CMYK)
So in subtractive color mixing the result is what you expected, but there is no blue, instead there is cyan:
Yellow + cyan = green
In general subtractive color mixing is just "taking away" (filtering) from white while additive color mixing is adding up from black. (base colors of subtractive are inverse from additive: red ->cyan; green->magenta; blue->yellow)
So if you start with white screen applying filters:
min( white (255,255,255), yellow (255,255,0), cyan (0,255,255)) = green (0,255,0)
The correct answer is NO, because there is no correct working model of how "color mixing in the real world" really works. It is FAR too complex and conditional and not really at all like the simple Red-Blue-Yellow stuff that we learned in school (it in fact requires all of Chemistry and a lot of Physics and Biology to resolve).
However, the simplistic answer is: YES, use subtractive mixing rather than Additive mixing.
The color-mixing that we learned in grade school is based on pigment combinations which are a form of subtractive color mixing (very simplistically). That is the more colors that we add together, the darker it becomes because each pigment subtracts a little bit more light.
On the other hand, almost all computer color-schemes are additive in that they are based on combining light waves (very simplistically), so they get brighter, because each color adds a little bit more light.
The RGB+ scheme is somewhat, the additive complement to the subtractive scheme that we learned in most US elementary schools (which is RBY-). However, they do not match up exactly and it can be difficult to convert between them (researching now ...)
OK, if you just want to switch from additive combinations in RGB to subtractive ones, you can use the following reverse-bayesan type formula to combine two colors:
NewColor.R = (Color1.R * Color2.R)/255
NewColor.G = (Color1.G * Color2.G)/255
NewColor.B = (Color1.B * Color2.B)/255
Adjusting for the difference in the chromatic poles (G to Y, then back to G) is a lot harder ...
It has been pointed out that this produces Black for the example problem, and technically this is correct for a true subtractive system, however, if you want more diluting/subtractive system, you could try this instead:
NewColor.R = 255 - SQRT(((255-Color1.R)^2 + (255-Color2.R)^2)/2)
NewColor.G = 255 - SQRT(((255-Color1.G)^2 + (255-Color2.G)^2)/2)
NewColor.B = 255 - SQRT(((255-Color1.B)^2 + (255-Color2.B)^2)/2)
This produces a dark grey instead of Black. But to get Yellow or anything close, you still have to fix the color-scheme's pole-alignment problem.