Accessing Color Frames with Unity and Tango 'Leibniz'

Question

I'm just starting to tinker with Tango and Unity. Unfortunately there doesn't seem to be any documentation or examples on how to access color data in Unity with the latest release.

I've been using the motion tracking example from GitHub (https://github.com/googlesamples/tango-examples-unity) as a starting point, trying to read incoming color frames the same way pose and depth data are read. I'm assuming the best way is to go through the "ITangoVideoOverlay" interface and "OnTangoImageAvailableEventHandler" callback.

All I am trying to do right now is to get the "OnTangoImageAvailableEventHandler" callback working and I can't quite figure it out. I have "Enable Video Overlay" checked on the Tango Manager and the following script connected to a GUI Text object for debugging.

using System.Collections;
using UnityEngine;
using UnityEngine.UI;
using Tango;
using System;

public class VideoController : MonoBehaviour, ITangoVideoOverlay
{

    private TangoApplication m_tangoApplication;

    private bool debugB = false;
    public Text debugText;

    // Use this for initialization
    void Start () {
        m_tangoApplication = FindObjectOfType<TangoApplication>();
        m_tangoApplication.Register(this);
    }

    // Update is called once per frame
    void Update () {
        if (debugB)
            debugText.text = "TRUE";
        else
            debugText.text = "FALSE";
    }

    // No Unity API
    public void OnTangoImageAvailableEventHandler(Tango.TangoEnums.TangoCameraId id, Tango.TangoUnityImageData image)
    {
        debugB = true;
    }
}

Is there some initialization of the camera that I am missing? Or is the preferred method still to use the VideoOverlayListener like in this older code: Getting color data in Unity

I know it's also possible to directly access the camera through Unity (disabling depth). But I would like to learn the "proper way" first.

Thank you for your time!

Update 04/28/15 - Latest version of the script, callback works! Still needs a conversion to RGB color

This script was written as an addition to Google's Tango Motion Tracking example on GitHub. Attach the script to a Unity camera and then link the public field "m_viewScreen" to a mesh object (like a plane) for the video texture to display on.

using System.Collections;
using UnityEngine;
using Tango;
using System;

public class VideoController : MonoBehaviour
{
    private TangoApplication m_tangoApplication;
    private Texture2D m_texture;
    private Material m_screenMaterial;
    private MeshFilter m_meshFilter;
    private bool m_readyToDraw = false;

    // Link to a mesh object for displaying texture
    public GameObject m_viewScreen;

    // Use this for initialization
    void Start ()
    {
        // Tango initilization
        m_tangoApplication = FindObjectOfType<TangoApplication>();
        m_tangoApplication.Register(this);
        m_tangoApplication.RegisterPermissionsCallback(_OnTangoApplicationPermissionsEvent);

        // Initialize view object Material
        m_meshFilter = m_viewScreen.GetComponent<MeshFilter> ();
        m_screenMaterial = new Material(Shader.Find("Mobile/Unlit (Supports Lightmap)"));

        // Begin to texture to webcam
        m_texture = m_tangoApplication.GetVideoOverlayTexture();
        m_texture.Apply();

        if (m_screenMaterial != null)
        {
            // Apply the texture
            m_screenMaterial.mainTexture = m_texture;
            m_meshFilter.GetComponent<MeshRenderer>().material = m_screenMaterial;

            // Connect the texture to the camera
            if (m_tangoApplication.m_useExperimentalVideoOverlay)
            {
                VideoOverlayProvider.ExperimentalConnectTexture(TangoEnums.TangoCameraId.TANGO_CAMERA_COLOR, m_texture.GetNativeTextureID(), _OnUnityFrameAvailable);
            }
            else
            {
                VideoOverlayProvider.ConnectTexture(TangoEnums.TangoCameraId.TANGO_CAMERA_COLOR, m_texture.GetNativeTextureID());
            }
        }
    }

    private void _OnTangoApplicationPermissionsEvent(bool permissionsGranted)
    {
        m_readyToDraw = true;
    }

    private void _OnUnityFrameAvailable(System.IntPtr callbackContext, Tango.TangoEnums.TangoCameraId cameraId)
    {
        // Do fun stuff here!

    }

    void OnPreRender()
    {
        if (m_readyToDraw)
        {
            VideoOverlayProvider.RenderLatestFrame(TangoEnums.TangoCameraId.TANGO_CAMERA_COLOR);
            GL.InvalidateState();
        }
    }

    void Update ()
    {
        // Do other fun stuff here!
    }
}

Answer 1

From what I could find, they changed the whole system so it's easier to access the image, where all the image grabbing is handle by the Tango object.

In your Start after grabbing the Tango App, try this:

m_texture = m_tangoApplication.GetVideoOverlayTexture();
m_texture.Apply();

if (m_screenMaterial != null)
{
    // Apply the texture
    m_screenMaterial.mainTexture = m_texture;
    m_meshFilter.GetComponent<MeshRenderer>().material = m_screenMaterial;

    // Connect the texture to the camera
    if (m_tangoApplication.m_useExperimentalVideoOverlay)
    {
        VideoOverlayProvider.ExperimentalConnectTexture(TangoEnums.TangoCameraId.TANGO_CAMERA_COLOR, m_texture.GetNativeTextureID(), _OnUnityFrameAvailable);
    }
    else
    {
        VideoOverlayProvider.ConnectTexture(TangoEnums.TangoCameraId.TANGO_CAMERA_COLOR, m_texture.GetNativeTextureID());
    }

}

and you'll need this somewhere else for the callback event:

private void _OnUnityFrameAvailable(System.IntPtr callbackContext, Tango.TangoEnums.TangoCameraId cameraId)
{
    // Do fun stuff here!
}

But it doesn't really need to do anything. Of course, the image is still in YUV-NV12 format so you'll need to convert it to RGB (or wait til their next release which should fix it).

Edit: Oops! Forgot you need one more call to actually update the texture on the AR material:

In Start() after grabbing the TangoApp:

m_tangoApplication.RegisterPermissionsCallback(_OnTangoApplicationPermissionsEvent);

Then:

private void _OnTangoApplicationPermissionsEvent(bool permissionsGranted)
{
    m_readyToDraw = true;
}

void OnPreRender()
{
    if (m_readyToDraw)
    {
        VideoOverlayProvider.RenderLatestFrame(TangoEnums.TangoCameraId.TANGO_CAMERA_COLOR);
        GL.InvalidateState();
    }
}

Hope that works now!