We\'re trying to do the following in Mathematica - RMagick remove white background from image and make it transparent.
But with actual photos it ends up looking lous
Just replace any pixel that is "almost close to white" with a pixel of the same RGB color and a Sigmoid gradient on the transparency channel. You can apply linear transition from solid to transparent, but Sinusoid or Sigmoid or Tanh look more natural, depending on the sharpness of edge you are looking for, they rapidly move away from the medium to either solid or transparent, but not in stepwise/binary manner, which is what you have now.
Think of it this way:
Let's say R,G,B are each 0.0-1.0, then let's represent white as a single number as R+G+B=1.0*3=3.0.
Taking a little bit of each color out makes it a little "off-white", but taking a little of all 3 is taking it a lot more off than a little off any one. Let's say that you allow a 10% reduction on any one channel: 1.0*.10 = .1, Now spread this loss across all three and bind it between 0 and 1 for alpha channel, if it's less than .1, such that (loss=0.9)=>0 and (loss=1.0)=>1:
threshold=.10;
maxLoss=1.0*threshold;
loss=3.0-(R+G+B);
alpha=If[loss>maxLoss,0,loss/maxLoss];
(* linear scaling is used above *)
(* or use 1/(1 + Exp[-10(loss - 0.5maxLoss)/maxLoss]) to set sigmoid alpha *)
(* Log decay: Log[maxLoss]/Log[loss]
(for loss and maxLoss <1, when using RGB 0-255, divide by 255 to use this one *)
setNewPixel[R,G,B,alpha];
For reference:
maxLoss = .1;
Plot[{ 1/(1 + Exp[-10(loss - 0.5maxLoss)/maxLoss]),
Log[maxLoss]/Log[loss],
loss/maxLoss
}, {loss, 0, maxLoss}]
The only danger (or benefit?) you have in this, is that this does not care about whites which actually ARE part of the photo. It removes all whites. So that if you have a picture of white car, it'll end up having transparent patches in it. But from your example, that seems to be a desired effect.