Unity3D - 2D static fog of war for line of sight implementation

Question

Unity beginner here!

So I'm trying to get some static-fog-of-war/line-of-sight for my game.

Currently I have 2 cameras:

The first one renders the scene.
The second one has black background and a white polygon which represents the line of sight.

The 1st camera renders first, so far everything works. (layers are set properly so that the second camera has only the line of sight polygon)

Then I wan't to do some post processing on the second camera by setting all white pixels of that camera to transparent but this is where I'm lost.

Using that code to apply the shader:

void OnRenderImage(RenderTexture src, RenderTexture dest)
{
    Graphics.Blit(src, dest, shader);
}

The problem is that when I apply a shader on the second camera (tested with depthOnly flag on second camera), the shader also modify the renderTexture of the first camera. It looks like the cameras' render textures are combined before the shader applies.

What am I missing here?

Draco18s no longer trusts SE Draco18s no longer trusts SE · Accepted Answer · 2017-06-07T15:45:29

You should use a shader for this instead. OnRenderImage requires piping the data back to the CPU and processing it linearly there (slow!!!¹) rather than in parallel on the GPU.

This will take a little bit of research to do properly, but the concept is pretty straight forward. You need to use the stencil mask to get the desired effect. The "white polygon" you have writes data (literally setting a bit to 1 instead of 0) to the stencil mask and then the rest of the scene gets rendered, but only where the stencil mask is a 1.

Basically think of the line of sight polygon as telling the camera where it's allowed to draw other things. Which is what a mask is. The stencil mask is just a specific grouping of bits that are available during rendering that aren't data that gets drawn to the screen, but rather persist across rendering passes. So:

Pass 1: draw the LOS object to the mask buffer.
Pass 2: render the scene only where the buffer has been modified.

I wrote up an outline shader that uses the stencil mask, which you could use as an example and get started. Mind, that shader was writen to apply only to a single object (and "creates" new geometry), but the same process should still work for doing what you're attempting (I just haven't tried to do it myself).

Looks like this thread has a (partial) shader that has the effect you're looking for.

I created two shaders of the standard affair and modified as follows: Hidden Shader: Just under the auto generated shader line "LOD 200", add:
 Stencil {
 Ref 1
 Comp Equal
 }
Portal Shader: Just under the auto generated "LOD 200", add:
 ZWrite Off
 ColorMask 0

 Pass {
     Stencil {
         Ref 1
         Comp always
         Pass replace
     }
 }
This is essentially the same concept I discussed above.

Alternatively...

Render the LOS to a separate camera as a black and white image to a render texture
Apply this render texture to a plane that sits above your scene
Apply a custom shader to this plane that converts black-white to black-transparent. A fragment shader that does this will be pretty easy to write. You'd just have to modify the section that returns the color of the texture to return the value of the Red channel as the output Alpha value.

¹I recently did an image effect that I couldn't figure out how to convert to a shader that I could pipe into the Blit function and the end result ran at 15 frames per second even at moderate resolutions (1024x768) and was crash-happy. Getting the compiled executable to run for longer than about 45 seconds was a gamble, so I wouldn't recommend following that path. It will seem OK in the editor, but then you'll get a surprise when you build it for distribution.

Unity3D - 2D static fog of war for line of sight implementation

1 Answers