Computer Vision: Masking a human hand

Question

I\'d like to detect my hand from a live video stream and create a mask of my hand. However I\'m reaching quite a poor result, as you can see from the picture.

My goa

Answer 1

A simple and powerful option is histogram backprojection. For example, create a 2D histogram using H and S (from HSV color space) or a* and b* (from La*b* color space), using pixels from different training images of your hand. Then use [cv2.calcBackProject][1] to classify the pixels in your stream. It's very fast and you should get 25 to 30 fps easily, I guess. Note this is a way to learn the color distribution of your object of interest. The same method can be used in other situations.

Answer 2

There are many ways to perform pixel-wise threshold to separate "skin pixels" from "non-skin pixels", and there are papers based on virtually any colorspace (even with RGB). So, my answer is simply based on the paper Face Segmentation Using Skin-Color Map in Videophone Applications by Chai and Ngan. They worked with the YCbCr colorspace and got quite nice results, the paper also mentions a threshold that worked well for them:

(Cb in [77, 127]) and (Cr in [133, 173])

The thresholds for the Y channel are not specified, but there are papers that mention Y > 80. For your single image, Y in the whole range is fine, i.e. it doesn't matter for actually distinguishing skin.

Here is the input, the binary image according to the thresholds mentioned, and the resulting image after discarding small components.

import sys
import numpy
import cv2

im = cv2.imread(sys.argv[1])
im_ycrcb = cv2.cvtColor(im, cv2.COLOR_BGR2YCR_CB)

skin_ycrcb_mint = numpy.array((0, 133, 77))
skin_ycrcb_maxt = numpy.array((255, 173, 127))
skin_ycrcb = cv2.inRange(im_ycrcb, skin_ycrcb_mint, skin_ycrcb_maxt)
cv2.imwrite(sys.argv[2], skin_ycrcb) # Second image

contours, _ = cv2.findContours(skin_ycrcb, cv2.RETR_EXTERNAL, 
        cv2.CHAIN_APPROX_SIMPLE)
for i, c in enumerate(contours):
    area = cv2.contourArea(c)
    if area > 1000:
        cv2.drawContours(im, contours, i, (255, 0, 0), 3)
cv2.imwrite(sys.argv[3], im)         # Final image

Lastly, there are a quite decent amount of papers that do not rely on individual pixel-wise classification for this task. Instead, they start from a base of labeled images that are known to contain either skin pixels or non-skin pixels. From that they train, for example, a SVM and then distinguish other inputs based on this classifier.