Passing Metal texture2d_array to SceneKit shader modifier

Question

I want to create shader modifier for SCNMaterial (with SCNShaderModifierEntryPointSurface) and pass Metal texture2d_array as custom input. To achieve that I try to use key-value coding setting parameter of type SCNMaterialProperty. From SceneKit manual on SCNMaterialProperty property contents:

You can set a value for this property using any of the following types:

...
A texture (SKTexture, MDLTexture, MTLTexture, or GLKTextureInfo)
...

So, I construct MTLTexture:

Create MTLTextureDescriptor with type MTLTextureType2DArray
Create MTLTexture with MTLDevice's newTextureWithDescriptor:
Fill texture with data using [texture replaceRegion:mipmapLevel:withBytes:bytesPerRow:];

Then I create SCNMaterialProperty, assign my created MTLTexture to its contents and pass it to material:

[material setValue:aTexture forKey:@"tex"];

Then I attach this shader modifier:

#pragma arguments
texture2d_array tex;

#pragma body
// do sampling from 1st slice
constexpr sampler smp(coord::normalized, address::repeat, filter::nearest);
_surface.diffuse = tex.sample(smp, _surface.diffuseTexcoord, 0);

And I get:

Backtrace:

I also tried to pass texture directly with [material setValue:aTexture forKey:@"tex"]; But then I'm getting error:

/Library/Caches/com.apple.xbs/Sources/Metal/Metal-56.6/ToolsLayers/Debug/MTLDebugRenderCommandEncoder.mm:1600: 
failed assertion `textureType mismatch at texture binding at index 8 for tex[0].'

Also, I tried passing MTLTexture2D in SCNMaterialProperty contents and It fails with exception in C3DImageGetImage type. Passing MTLTexture2D directly via [material setValue:forKey] results in everything white - wrong texture being sampled.

When analysing 'Capture GPU Frame' output, I see that wrong texture bound during draw call.

Maybe someone managed to pass texture2d_array to shader modifier? Is it possible at all? I could use metal command buffer to bind manually, but how to get access to it on per-material basis?

Thank you all.

Have you been able to access a simple texture2d (instead of a texture2d_array) within a shader modifier using this approach? I'm curious as I couldn't seem to get it to work, instead I'm assigning my MTLTexture's to SceneKits various materials (diffuse, specular, normal, etc) as a hack'ish workaround. — lock
@lock It seems Apple always interpreting material property contents as image when passing as custom input for shader modifier (but I confirm that passing MTLTexture2D to diffuse, specular etc works). When assigning MTLTexture directly without wrapping in SCNMaterialProperty I get no error but all white color - 'Capture GPU Frame' output shows that wrong (white? 4x4) texture bound during draw call. Looks like bug or lack of functionality in SCNMaterialProperty. I could bind texture myself, but I can't figure out how to access SCNRenderContext when drawing nodes that uses certain materials. — Ef Dot
Thanks @EfDot. Your findings seem to line up with what I've seen, especially the white 4x4 texture. I did some looking into the SCNRenderContext to see how I could possibly pass in another MTLTexture (through the render command encoder), without success unfortunately. An SCNProgram will work, but then you have to rewrite your entire shader. — lock
@lock you welcome. I see no problems in writing whole custom shader using SCNProgram (since any error in shader modifier dumps it to stderr), but when I tried to do that, I loose frustum culling (which I believe is bug in SceneKit). — Ef Dot

masaldana2 masaldana2 · Accepted Answer · 2019-10-22T18:56:50

        #include <metal_stdlib>
        #include <metal_texture>
        using namespace metal;
        #pragma arguments
        texture2d<float> mask;

        #pragma body
        constexpr sampler quadSampler(address::clamp_to_edge, filter::linear);

        _surface.diffuse = mask.sample(quadSampler, _surface.diffuseTexcoord);

then on swift side

let mask = SCNMaterialProperty(contents: MDLTexture(named: "gameFrame.png")!)
doorFrame.geometry?.firstMaterial?.setValue(mask, forKey: "mask")

Passing Metal texture2d_array to SceneKit shader modifier

1 Answers