My app works with photos and videos of people, which I want to cartoonify. So I need an algorithm to do it manually (we use c++/Qt for our product, which has image manipulat
If there's some set of parameters which achieve the desired effect in the GIMP's Cartoon filter (or some other combination of filters) it can be run in a batch processing mode.
You might want to check out Freestyle, an open-source (Google Summer of Code, even) project to implement a non-photorealistic renderer for Blender. Here's an example of its output, in cartoon-mode:
(source: sourceforge.net)
Here's some algorithms to play with:
These are fairly basic and all very easy to implement. Keep in mind that median and box blur filters can be implemented with linear time complexity w.r.t. the kernel radius.
More edits:
Once you get the idea of Huang's algorithm, implementing a box blur filter is a delicious piece of cake.
Reading material:
Other reading materials include Gonzalez & Woods' Digital Image Processing (seems to be an older edition) for segmentation and edge tracing. 8-way edge tracing can be really hard to bend your head around (choosing between on-pixel or between-pixel edges and how to latch onto edges). I'd be happy to share some code, but the hundred-liners don't exactly fit smoothly in here.
Not sure if this will help, but this tutorial for Photoshop suggests doing the following:
Here's the result.
I imagine that you could do something similar in your program.
You could try rotoscopy, like toonyphotos.com does:
I have not done this myself, but thinking about two steps that might give the image a cartoonish look.
Detect edges, and draw a fairly fairly thick line (a few pixels) on those edges.
Decrease the number of colours in your image.