Kinect + Python - Fill depth for shadows

问题

The Kinect camera returns a depth image for the whole view. Due to the way the image is captured, some small areas are invisible to the camera. For those areas 2047 is returned.

I want to fill those areas with the value that is left of them - which is the most likely value for that area. I have the depth as numpy uint16 array. The trivial solution would be:

for x in xrange(depth.shape[1]):
  for y in xrange(depth.shape[0]):
    if depth[y,x] == 2047 and x > 0:
      depth[y,x] = depth[y,x-1]

This takes around 16 seconds to execute (Raspberry 2) per 640 x 480 frame.

I came up with a solution using indexes:

w = numpy.where(depth == 2047)
w = zip(w[0], w[1])
for index in w:
  if depth[index] == 2047 and index[1] > 0:
    depth[index] = depth[index[0],index[1] - 1]

This takes around 0.6 seconds to execute for a test frame. Much faster but still far from perfect. Index computation and zip only take 0.04 seconds, so the main performance killer is the loop.

I reduced it to 0.3 seconds by using item():

for index in w:
  if depth.item(index) == 2047 and index[1] > 0:
    depth.itemset(index, depth.item(index[0],index[1] - 1))

Can this be improved further using only python (+numpy/opencv)? Compared to how fast simple filtering is, it should be possible to be faster than 0.05s

回答1:

You have islands going behind the places where the elements in the input array are 2. The job here is to fill the shadows with the element right before starting the shadows. So, one way would be to find out the start and stop places of those islands and put x and -x at those places respectively, where x is the element right before starting of each island. Then, do cumsum along the rows, which would effectively fill the shodow-islands with x. That's all there is for a vectorized solution! Here's the implementation -

# Get mask of places to be updated
mask = np.zeros(np.array(depth.shape) + [0,1],dtype=bool)
mask[:,1:-1] = depth[:,1:] == 2047

# Get differentiation along the second axis and thus island start and stops
diffs = np.diff(mask.astype(int),axis=1)
start_mask = diffs == 1
stop_mask = diffs == -1

# Get a mapping array that has island places filled with the start-1 element
map_arr = np.zeros_like(diffs)
map_arr[start_mask] = depth[start_mask]
map_arr[stop_mask] = -depth[start_mask]
map_filled_arr = map_arr.cumsum(1)[:,:-1]

# Use mask created earlier to selectively set elements from map array
valid_mask = mask[:,1:-1]
depth[:,1:][valid_mask] = map_filled_arr[valid_mask]

Benchmarking

Define functions :

def fill_depth_original(depth):
    for x in xrange(depth.shape[1]):
        for y in xrange(depth.shape[0]):
            if depth[y,x] == 2047 and x > 0:
                depth[y,x] = depth[y,x-1]

def fill_depth_original_v2(depth):
    w = np.where(depth == 2047)
    w = zip(w[0], w[1])
    for index in w:
      if depth[index] == 2047 and index[1] > 0:
        depth[index] = depth[index[0],index[1] - 1]

def fill_depth_vectorized(depth):

    mask = np.zeros(np.array(depth.shape) + [0,1],dtype=bool)
    mask[:,1:-1] = depth[:,1:] == 2047

    diffs = np.diff(mask.astype(int),axis=1)
    start_mask = diffs == 1
    stop_mask = diffs == -1

    map_arr = np.zeros_like(diffs)
    map_arr[start_mask] = depth[start_mask]
    map_arr[stop_mask] = -depth[start_mask]
    map_filled_arr = map_arr.cumsum(1)[:,:-1]

    valid_mask = mask[:,1:-1]
    depth[:,1:][valid_mask] = map_filled_arr[valid_mask]

Runtime tests and verify outputs :

In [303]: # Create a random array and get a copy for profiling vectorized method
     ...: depth = np.random.randint(2047-150,2047+150,(500,500))
     ...: depthc1 = depth.copy()
     ...: depthc2 = depth.copy()
     ...: 

In [304]: fill_depth_original(depth)
     ...: fill_depth_original_v2(depthc1)
     ...: fill_depth_vectorized(depthc2)
     ...: 

In [305]: np.allclose(depth,depthc1)
Out[305]: True

In [306]: np.allclose(depth,depthc2)
Out[306]: True

In [307]: # Create a random array and get a copy for profiling vectorized method
     ...: depth = np.random.randint(2047-150,2047+150,(500,500))
     ...: depthc1 = depth.copy()
     ...: depthc2 = depth.copy()
     ...: 

In [308]: %timeit fill_depth_original(depth)
     ...: %timeit fill_depth_original_v2(depthc1)
     ...: %timeit fill_depth_vectorized(depthc2)
     ...: 
10 loops, best of 3: 89.6 ms per loop
1000 loops, best of 3: 1.47 ms per loop
100 loops, best of 3: 10.3 ms per loop

So, the second approach listed in the question still looks like winning!

来源：https://stackoverflow.com/questions/35231524/kinect-python-fill-depth-for-shadows

标签

python

OpenCV

numpy

kinect