iOS10 + SceneKit: render a video with custom shader

4
votes

How can I render a video as a diffuse texture on a node in SceneKit while using custom shader? This sounds like a simple thing to do but:

  • SceneKit elements don't support AVPlayerLayer as input texture (SO)
  • alternatives are:
    • using a SpriteKit scene with a SKVideoNode but:
      • Since iOS 10, rendering with openGL fails as soon as a SpriteKit scene is involved (SO) I've filled a radar
      • I could not figure out how to pass uniforms to a shader modifier when using Metal (SO) Things were working perfectly with openGL (on iOS 9, when SpriteKit was working) There's surprisingly very little documentation out there on that.
      • SCNProgram seems to fails as soon as a video is involved in the SceneKit scene (SO) Things work fine with the same setup without video
    • transforming the video output to a CGImage, passing it to a CALayer, and passing this layer to the SceneKit video node (no SpriteKit involved) This is detailed here, but not a viable solution as soon as the video is a bit large, because all the output is getting copied to image and this is costly..

I'm still quite reluctant to get out of SceneKit, since it's handling a lot of the annoying low level things with opengl/metal, but I can't get things to work right now...

As an additional bad news, it looks like SpriteKit has memory leak when used with Metal, and that it keeps alive the SceneView, and therefore all the parent views and everything attached to them...

You can find a reproduction of this issue here, and a simpler one here:

1
videoopengl-essprite-kitscenekitmetal
Last year SpriteKit went one step forward (must have been something improved) and about three steps back (iOS 9). It took many months before Apple began addressing the problems. They didn't ever publicly acknowledge the problems despite many specific, recurring and matching complaints. – Confused
iOS 10 looks like it's SceneKit's turn to take a few steps back, and one forward (PBR). SceneKit has 1/20th (or less) of SpriteKit users. So it might take a whole year for them to get around to even looking at problems. Having said that, two of the guys that work on SceneKit are active in these forums. – Confused
Do you know how can I even have an idea if the issue is coming from SceneKit or SpriteKit? – Guig
Not really, depends too much on how you're doing things, and how obscure the right approach is to achieving this. The Apple documentation for Sprite Kit is poor. The documentation for SceneKit is atrocious. – Confused
It's no direct help but... if you're into it, one more commit and we could start documenting anything we learn about SceneKit. I'm not a big believer in SO.Docs, but this is one use case it might be good for: stackoverflow.com/documentation/scenekit/commit – Confused

1 Answers

3
votes

Not being able to solve this issue, I wrote my own library that mimics some functionalities of SceneKit: https://github.com/team-pie/DDDKit