SurfaceTexture in Android plugin doesn't work in Unity

Question

I can't get the texture tied to a SurfaceTexture to display in Unity.

Update 4: Based on the pipeline in update 1 (surface->external texture via surface texture -> fbo -> texture 2d) I know the SurfaceTexture isn't properly converting its surface to a texture. I can get correctly drawn pictures from its surface via pixelcopy and I can confirm my FBO drawing to texture2d pipeline works with some test colors. So the question is, why can't the SurfaceTexture convert its surface to a texture?

I generate a Texture in Java and pass its pointer back to Unity:

public void initGLTexture()
{
    Log.d("Unity", "initGLTexture");
    int textures[] = new int[1];
    GLES20.glGenTextures(1, textures, 0);
    mTextureId = textures[0];

    GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mTextureId);
    GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
    GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
    GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}

I create a SurfaceTexture from the id (in Java):

mSurfaceTexture = new SurfaceTexture(mTextureId);
mSurfaceTexture.setDefaultBufferSize(512, 512);

I use a third-party library, GeckoView, to render onto the Surface of the SurfaceTexture. I call the following method from Unity's OnRenderObject() to keep all GL rendering on the same thread:

mSurfaceTexture.updateTexImage();

I know the above code allows proper drawing onto the surface.

I call the following in Unity to load the texture:

_imageTexture2D = Texture2D.CreateExternalTexture(
        512,512,TextureFormat.RGBA32,false,true,(IntPtr) mTextureId);
_rawImage.texture = _imageTexture2D;

Why does the RawImage with the texture applied show only this sprite-looking thing, which should be a webpage?

Update 1: So I've been working on the hypothesis of: use Gecko to draw to the Surface, and use a SurfaceTexture to render this surface to a GL_TEXTURE_EXTERNAL_OES. Since I can't display this on Unity (not sure why) I am drawing this texture to a frame buffer and copying the pixels in the framebuffer to a GL_TEXTURE_2D. I am getting a web page in the texture_2d (in the emulator with an imageview and glReadPixels). However, when I import the work into Unity to test if the pipeline is okay thus far I just get a black screen. I CAN get images of the surface via the PixelCopy api.

Here is my FBO overview code - my rendering code comes from grafika's texture2D program:

    // bind display buffer
    GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
    GlUtil.checkGlError("glbindframebuffer");

    // unbind external texture to make sure it's fresh
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
    GlUtil.checkGlError("glunbindexternaltex");

   // bind source texture (done in drawFrame as well )
    GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mOffscreenTextureId);
    GlUtil.checkGlError("glBindFramebuffer");

    // draw to frame buffer
    GLES20.glClearColor(0.0f, 0.0f, 0.0f, 1.0f);    // again, only really need to
    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);     //  clear pixels outside rect

    mFullScreen.drawFrame(mOffscreenTextureId, mIdentityMatrix);

    // unbind source texture
    GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);
    GlUtil.checkGlError("glBindTexture2d");

    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
    GlUtil.checkGlError("glunbindexternaltex");

    // make sure we're still bound to fbo
    GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
    GlUtil.checkGlError("glBindTexture2d");

    // copy pixels from frame buffer to display texture
     GLES20.glCopyTexImage2D(GLES20.GL_TEXTURE_2D,0,GLES20.GL_RGBA,0,0,512,512,0);

    // read pixels from the display buffer to imageview for debugging
    BitmapDisplay.mBitmap = SavePixels(0,0,512,512);

    // unbind texture
    GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);

Here's my player settings > other:

Update 2: Possible pipeline to try: call the draw function of the external texture to FBO (attached to Unity's texture_2d) in C++ via this interface.

Update 3: Calling the Java functions from native code that are responsible for drawing the texture from the SurfaceTexture to the FBO to Unity's texture via the GL.IssuePluginEvent produce a black texture as in the first update. It will show images in the emulator but not in Unity.

@Shaman do you have one of the pipelines' logs in mind? e.g. 1. try and display the external texture attached to the surface texture, 2. the external texture-> fbo -> teture_2d for display, 3. the same as 2 but with a textureview attached to the surface texture — pale bone
Alright, can you show me Player settings -> Other settings? (Unity) — Hamid Yusifli

Robson de Paula Robson de Paula · Accepted Answer · 2019-01-22T15:33:21

I had to do a similar task a couple of months ago and found out that the correct pipeline is creating a texture in Unity, obtaining a native pointer in C and finally updating it in the Java layer.

Please take a look at this sample project, it should give you a different perspective.

https://github.com/robsondepaula/unity-android-native-camera

Regards

SurfaceTexture in Android plugin doesn't work in Unity

1 Answers