Mobile Vision API - concatenate new detector object to continue frame processing

Question

I want to use the new face detection feature that the vision API provides along with additional frame processing in an application. For this, I need to have access to the ca

Answer 1

Yes, it is possible. You'd need to create your own subclass of Detector which wraps FaceDetector and executes your extra frame processing code in the detect method. It would look something like this:

class MyFaceDetector extends Detector {
  private Detector mDelegate;

  MyFaceDetector(Detector delegate) {
    mDelegate = delegate;
  }

  public SparseArray detect(Frame frame) {
    // *** add your custom frame processing code here
    return mDelegate.detect(frame);
  }

  public boolean isOperational() {
    return mDelegate.isOperational();
  }

  public boolean setFocus(int id) {
    return mDelegate.setFocus(id);
  }
}

You'd wrap the face detector with your class, and pass your class into the camera source. It would look something like this:

    FaceDetector faceDetector = new FaceDetector.Builder(context)
            .build();
    MyFaceDetector myFaceDetector = new MyFaceDetector(faceDetector);

    myFaceDetector.setProcessor(/* include your processor here */);

    mCameraSource = new CameraSource.Builder(context, myFaceDetector)
            .build();

Your detector will be called first with the raw frame data.

Note that the image may not be upright, if the device is rotated. You can get the orientation through the frame's metadata.getRotation method.

One word of caution: once the detect method returns, you should not access the frame pixel data. Since the camera source recycles image buffers, the contents of the frame object will be eventually overridden once the method returns.

EDIT: (additional notes) You could also avoid the boilerplate code of MyFaceDetector using a MultiDetector like this:

MultiDetector multiDetector = new MultiDetector.Builder()
    .add(new FaceDetector.Builder(context)
                .build())
    .add(new YourReallyOwnDetector())
    .build();

Also note the use of FaceTrackerFactory in conjuction with MultiProcessor described there.