1. 技术文章
  2. Using the onFrameAvailable() in Jacobi Google Tango API

Using the onFrameAvailable() in Jacobi Google Tango API

由 喜欢而已 提交于 2019-11-30 03:58:27

OK, I figured out a way to make it work.

Update: My working solution is here:

https://github.com/stevehenderson/GoogleTango_AR_VideoCapture

I essentially set up a "man (renderer) in the middle" attack on the rendering pipeline. This approach intercepts the SetRenderer call from the TangoCameraPreview base class, and allows one to get access to the base renderer's OnDraw() method and the GL context. I then add additional methods to this extended renderer that allow reading of the GL buffer.

General approach

1) Extend the TangoCameraPreview class (e.g. in my example ReadableTangoCameraPreview). Override the setRenderer(GLSurfaceView.Renderer renderer), keeping a reference to the base renderer, and replacing the renderer with your own "wrapped" GLSUrface.Renderer renderer that will add methods to render the backbuffer to an image on the device.

2) Create your own GLSurfaceView.Renderer Interface (e.g. my ScreenGrabRenderer class ) that implements all the GLSurfaceView.Renderer methods, passing them on to the base renderer captured in Step 1. Also, add a few new methods to "cue" when you want to grab the image.

3) Implement the ScreenGrabRenderer described in step 2 above.

4) Use a callback interface (my TangoCameraScreengrabCallback) to communicate when an image has been copied

It works pretty well, and allows one to grab the camera bits in an image without rooting the device.

Note: I haven't had the need to closely synchronize my captured images with the point cloud. So I haven't checked the latency. For best results, you may need to invoke the C methods proposed by Mark.

Here's what each of my classes looks like..

///Main Activity Class where bulk of Tango code is
.
.
.

// Create our Preview view and set it as the content of our activity.
mTangoCameraPreview = new ReadableTangoCameraPreview(getActivity());

RelativeLayout preview = (RelativeLayout) view.findViewById(R.id.camera_preview);
preview.addView(mTangoCameraPreview);


.
.
.
//When you want to take a snapshot, call the takeSnapShotMethod()
//(you can make this respond to a button)
mTangoCameraPreview.takeSnapShot();
.
.
.
.
.
//Main Tango Listeners
@Override
public void onFrameAvailable(final int cameraId) {
    // Update the UI with TangoPose information
    runOnUiThread(new Runnable() {
        @Override
        public void run() {
            if (cameraId == TangoCameraIntrinsics.TANGO_CAMERA_COLOR) {
                tangoCameraPreview.onFrameAvailable();
            }       
        }
    });
}

ReadableTangoCameraPreview Class

public class ReadableTangoCameraPreview extends TangoCameraPreview implements  TangoCameraScreengrabCallback  {

    Activity mainActivity;
    private static final String TAG = ReadableTangoCameraPreview.class.getSimpleName();

    //An intercept renderer
    ScreenGrabRenderer screenGrabRenderer;

    private boolean takeSnapShot = false;

    @Override
    public void setRenderer(GLSurfaceView.Renderer renderer) {  
        //Create our "man in the middle"
        screenGrabRenderer= new ScreenGrabRenderer(renderer);

        //Set it's call back
        screenGrabRenderer.setTangoCameraScreengrabCallback(this);

        //Tell the TangoCameraPreview class to use this intermediate renderer
        super.setRenderer(screenGrabRenderer);
        Log.i(TAG,"Intercepted the renderer!!!");       
    }


    /**
     * Set a trigger for snapshot.  Call this from main activity
     * in response to a use input
     */
    public void takeSnapShot() {
        takeSnapShot = true;
    }   

    @Override
    public void onFrameAvailable() {
        super.onFrameAvailable();
        if(takeSnapShot) {
            //screenGrabWithRoot();
            screenGrabRenderer.grabNextScreen(0,0,this.getWidth(),this.getHeight());
            takeSnapShot = false;           
        }
    }

    public ReadableTangoCameraPreview(Activity context) {
        super(context); 
        mainActivity = context;     

    }

    public void newPhoto(String aNewPhotoPath) {
        //This gets called when a new photo was  grabbed created in the renderer
        Log.i(TAG,"New image available at" + aNewPhotoPath);    
    }

}

ScreenGrabRenderer Interface

(Overloads the TangoCameraPreview default Renderer)

/**
 * This is an intermediate class that intercepts all calls to the TangoCameraPreview's
 * default renderer.
 * 
 * It simply passes all render calls through to the default renderer.
 * 
 * When required, it can also use the renderer methods to dump a copy of the frame to a bitmap
 * 
 * @author henderso
 *
 */
public class ScreenGrabRenderer implements GLSurfaceView.Renderer  {


    TangoCameraScreengrabCallback mTangoCameraScreengrabCallback;

    GLSurfaceView.Renderer tangoCameraRenderer;
    private static final String TAG = ScreenGrabRenderer.class.getSimpleName();

    private String lastFileName = "unset";

    boolean grabNextScreen = false;

    int grabX = 0;
    int grabY = 0;
    int grabWidth = 640;
    int grabHeight = 320;

    public void setTangoCameraScreengrabCallback(TangoCameraScreengrabCallback aTangoCameraScreengrabCallback) {
        mTangoCameraScreengrabCallback = aTangoCameraScreengrabCallback;
    }

    /**
     * Cue the renderer to grab the next screen.  This is a signal that will
     * be detected inside the onDrawFrame() method
     * 
     * @param b
     */
    public void grabNextScreen(int x, int y, int w, int h) {
        grabNextScreen = true;
        grabX=x;
        grabY=y;
        grabWidth=w;
        grabHeight=h;
    }

    @Override
    public void onSurfaceCreated(GL10 gl, EGLConfig config) {
        tangoCameraRenderer.onSurfaceCreated(gl, config);

    }

    @Override
    public void onSurfaceChanged(GL10 gl, int width, int height) {
        tangoCameraRenderer.onSurfaceChanged(gl, width, height);        
    }

    @Override
    public void onDrawFrame(GL10 gl) {
        tangoCameraRenderer.onDrawFrame(gl);    
        if(grabNextScreen) {
            screenGrab(gl);
            grabNextScreen=false;
        }
    }


    /**
     * 
     * Creates a bitmap given a certain dimension and an OpenGL context
     *  
     * This code was lifted from here:
     * 
     * http://stackoverflow.com/questions/5514149/capture-screen-of-glsurfaceview-to-bitmap 
     */
    private Bitmap createBitmapFromGLSurface(int x, int y, int w, int h, GL10 gl)
            throws OutOfMemoryError {
        int bitmapBuffer[] = new int[w * h];
        int bitmapSource[] = new int[w * h];
        IntBuffer intBuffer = IntBuffer.wrap(bitmapBuffer);
        intBuffer.position(0);

        try {
            gl.glReadPixels(x, y, w, h, GL10.GL_RGBA, GL10.GL_UNSIGNED_BYTE, intBuffer);
            int offset1, offset2;
            for (int i = 0; i < h; i++) {
                offset1 = i * w;
                offset2 = (h - i - 1) * w;
                for (int j = 0; j < w; j++) {
                    int texturePixel = bitmapBuffer[offset1 + j];
                    int blue = (texturePixel >> 16) & 0xff;
                    int red = (texturePixel << 16) & 0x00ff0000;
                    int pixel = (texturePixel & 0xff00ff00) | red | blue;
                    bitmapSource[offset2 + j] = pixel;
                }
            }
        } catch (GLException e) {
            Log.e(TAG,e.toString());
            return null;
        }

        return Bitmap.createBitmap(bitmapSource, w, h, Bitmap.Config.ARGB_8888);
    }


    /**
     * Writes a copy of the GLSurface backbuffer to storage
     */
    private void screenGrab(GL10 gl) {
        long fileprefix = System.currentTimeMillis();
        String targetPath =Environment.getExternalStorageDirectory()  + "/RavenEye/Photos/";
        String imageFileName = fileprefix + ".png";   
        String fullPath = "error";

        Bitmap image = createBitmapFromGLSurface(grabX,grabY,grabWidth,grabHeight,gl);
        if(!(new File(targetPath)).exists()) {
            new File(targetPath).mkdirs();
        }
        try {           
            File targetDirectory = new File(targetPath);
            File photo=new File(targetDirectory, imageFileName);
            FileOutputStream fos=new FileOutputStream(photo.getPath());
            image.compress(CompressFormat.PNG, 100, fos);          
            fos.flush();
            fos.close();
            fullPath =targetPath + imageFileName;
            Log.i(TAG, "Grabbed an image in target path:" + fullPath);      

            ///Notify the outer class(es)
            if(mTangoCameraScreengrabCallback != null) {
                mTangoCameraScreengrabCallback.newPhoto(fullPath);
            } else {
                Log.i(TAG, "Callback not set properly..");
            }

        } catch (FileNotFoundException e) {
            Log.e(TAG,"Exception " + e);
            e.printStackTrace();
        } catch (IOException e) {
            Log.e(TAG,"Exception " + e);
            e.printStackTrace();
        }   
        lastFileName = fullPath;
    }


    /**
     * Constructor
     * @param baseRenderer
     */
    public ScreenGrabRenderer(GLSurfaceView.Renderer baseRenderer) {
        tangoCameraRenderer = baseRenderer;     
    }
}

TangoCameraScreengrabCallback Interface (not required unless you want to pass info back from the screen grab renderer)

    /*
     * The TangoCameraScreengrabCallback is a generic interface that provides callback mechanism 
     * to an implementing activity.
     * 
     */
    interface TangoCameraScreengrabCallback {
        public void newPhoto(String aNewPhotoPath);
    }

I haven't tried on the latest release, but it was the absence of this functionality that drove me to the C API where I could get image data - a recent post, I think on the G+ page, seemed to indicate that the Unity API now returns image data as well - for a company that wants to keep scolding us when we don't use Java, it certainly is an odd lag :-)

标签
google-project-tango
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!