Getting color data in Java Tango SDK (or C SDK)

Question

I just got my tablet, where I previously had a phone. As most of you probably know, the phone's SDK allowed capturing of superframes via the android camera callback. If properly parsed, the superframes contained all of the relevant sensor data.

In Archimedes, I tried the following. I made an activity that implements CameraPreviewListener:

public class MainActivity extends Activity implements CameraPreviewListener
{
    // Inside of this class we manage another object that implements PreviewCallback
    ...
}

This allows capture of a camera image, just like it would on any other Android device. (Note that on a Peanut phone, this provided superframes; this just provides a raw RGB buffer on a tablet.) Likewise, a simple implementation of callbacks from the Tango service works just fine.

public class MainActivity extends Activity
{
    private Tango mTango;
    protected void onActivityResult(int requestCode, int resultCode, Intent data)
    {
        ...
        setTangoListeners();
    }
    private void setTangoListeners()
    {
        mTango.connectListener(framePairs, new OnTangoUpdateListener() {

        @Override
        public void onPoseAvailable(final TangoPoseData pose) 
        {
            System.out.println("Pose data received.");
        }
        @Override
        public void onXyzIjAvailable(final TangoXyzIjData xyzIj)
        {
            System.out.println("Cloud data received.");
        }
        @Override
        public void onTangoEvent(final TangoEvent event)
        {
            ...
        }
    }
}

However, when I try to combine the two together, like so,

public class MainActivity extends Activity implements CameraPreviewListener
{
    // Inside of this class we manage another object that implements PreviewCallback
    private Tango mTango;
    protected void onActivityResult(int requestCode, int resultCode, Intent data)
    {
        ...
        setTangoListeners();
    }
    private void setTangoListeners()
    {
        mTango.connectListener(framePairs, new OnTangoUpdateListener() {

        @Override
        public void onPoseAvailable(final TangoPoseData pose) 
        {
            System.out.println("Pose data received.");
        }
        @Override
        public void onXyzIjAvailable(final TangoXyzIjData xyzIj)
        {
            System.out.println("Cloud data received.");
        }
        @Override
        public void onTangoEvent(final TangoEvent event)
        {
            ...
        }
    }
}

Something strange happens. The camera callback fires just fine, and I get onPoseAvailable callbacks as well. However, I no longer receive any callbacks to onXyzIjAvailable.

So my questions are:

Am I correct in presuming that the Tango service requires access to the Camera callback to produce PointCloud data?
If so, is there anyway around this so that I can get RGB buffer and Pointcloud at roughly the same time? (Yes, I know that calibration is not trivial.)
If there is no solution to 2, are there any future SDK updates planned that will allow such a thing?
I haven't explored the C SDK yet. Maybe there's a way to do it in there, and if so, does anybody have any experience to lend?

Ideally, I would like access to the raw RGB buffer, intensity image, and fisheye image; I don't really care how, as long as it works.

which tablet you are holding? Is this "Yellowstone" architecture? — Nikolaos Giotis

r4ravi2008 r4ravi2008 · Accepted Answer · 2014-11-06T20:50:45

Tango Service does require camera access to produce point cloud data. You can set the camera permissions in the android manifest file of your app.
Tango Service needs a handle on the RGB-IR camera to receive callbacks from onXyzIjAvailable. From what I observed from your code it looks like you are trying to use Android Camera APIs along with Tango Service. What this means is RGB-IR camera is already consumed by the Android Camera APIs meaning Tango service cannot use the RGB IR camera anymore to produce the point cloud data. But, you will still receive onPoseAvailable callbacks as motion tracking uses FishEye camera for its Pose estimates.

This example shows you a way use connectSurface call to render RGB data on to a Android Surface using Tango Service. Using this call doesn't affect onXyzIjAvailable callbacks as it is a part of Tango Service API.

tl:dr: Do not use Android Camera APIs with Tango Service. Instead use the connectSurface(int cameraId, Surface surface) function to connect to the camera you want.

Getting color data in Java Tango SDK (or C SDK)

1 Answers