Rendering SurfaceTexture to Unity Texture2D

Question

I came up with simillar questions earlier, but they weren\'t good clarified and right now I would like to take an advice what\'s wrong I\'m doing in my code.

So what

Answer 1

You can't call surfaceTexture.updateTexImage(); in onFrameAvailable, call it in DrawFrame() .

And in Unity3D:

void Update()
    {
        androidStreamerObj.Call("DrawFrame");
        GL.InvalidateState(); // ADD it
    }

Answer 2

Few people know this trick. I'd like to give you some brief and I think you can figure out the rest:

1. First you need a ImageReader, it can accept surface that you want to read, and it has a callback ImageReader.OnImageAvailableListener once the image is ready your code can get called.
2. Use ImageReader.acquireLatestImage() to get a Image
3. Use Image.getHardwareBuffer() to get a HardwareBuffer

4. Pass the HardwareBuffer to your JNI function and update your texture

   //Target your texture
   glBindTexture(GL_TEXTURE_2D, textureName);
   // Get native AHardwareBuffer
   AHardwareBuffer *hwbuffer = AHardwareBuffer_fromHardwareBuffer(env, hardwareBuffer);
   // Create EGLClientBuffer from the AHardwareBuffer.
   EGLClientBuffer native_buffer = eglGetNativeClientBufferANDROID(hwbuffer);
   // Destroy last created EGLImageKHR
   if (cachedImages.find(textureName) != cachedImages.end()){
       eglDestroyImageKHR(eglGetCurrentDisplay(), cachedImages[textureName]);
   }
   // Begin to make new EGLImageKHR
   EGLImageKHR image {EGL_NO_IMAGE_KHR};
   EGLint attrs[] = {
           EGL_IMAGE_PRESERVED_KHR,
           EGL_TRUE,
           EGL_NONE,
   };
   // Create EGLImage from EGLClientBuffer.
   image = eglCreateImageKHR(eglGetCurrentDisplay(), EGL_NO_CONTEXT, EGL_NATIVE_BUFFER_ANDROID, native_buffer, attrs);
   if (image == EGL_NO_IMAGE_KHR) {
       LOGE("Failed to create EGLImage.");
       return false;
   }
   // Cache the image
   cachedImages[textureName] = image;
   // Get glEGLImageTargetTexture2DOES
   if (!isGlEGLImageTargetTexture2DOESInited) {
       glEGLImageTargetTexture2DOES = (PFNGLEGLIMAGETARGETTEXTURE2DOESPROC) eglGetProcAddress("glEGLImageTargetTexture2DOES");
       isGlEGLImageTargetTexture2DOESInited = true;
   }
   if(glEGLImageTargetTexture2DOES == NULL){
       LOGE("Error: Failed to find glEGLImageTargetTexture2DOES at %s:%in", __FILE__, __LINE__);
       return false;
   }
   // Allocate the OpenGL texture using the EGLImage.
   glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, image);
   //Not GL_TEXTURE_EXTERNAL_OES
   //glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, image);
   
   glBindTexture(GL_TEXTURE_2D, 0);
   

5. Now you have updated texturename, which is you created in your code before(from native or Android EGL or Unity)

The whole process is like:

1. ImageReader's callback sets a image ready flag,
2. Unity's Update() check if there is image ready
3. Update the texture by using the code above.