How to setup geometry shaders in HLSL?

Question

I am trying to add a geometry shader stage to my pipeline, however when i add the geometry shader function, even the vertex shader compilation gives errors for some reason (the Vertex, Pixel and Geometry shaders are in the same file). the code inside the geometry shader seems to be causing the problem, because as soon as i deleted it, the vertex shader compiled successfully.

I haven't ever dealt with geometry shaders in any API, but it seemed to me that setting-up a geometry shader should be as simple as the rest, there are some geometry shader examples for DX11 on the internet, but not many. At least all those that i was able find were only showing the hlsl side.


cbuffer TickConstantBuffer : register(b0)
{
    matrix World;
    matrix View;
}

cbuffer OnEvent : register(b1)
{
    matrix Projection;
}

float4 VS( float4 Pos : POSITION ) : SV_POSITION
{
    float4 OutPos;

    OutPos = mul(Pos, View);
    OutPos = mul(OutPos, Projection);

    return Pos;
}

[maxvertexcount(4)]
void GS( point float4 input[1], inout TriangleStream<float4> OutputStream )
{
    float4 v;
    OutputStream.Append(float4(input[0].x + 0.5, input[0].y + 0.5, input[0].z, input[0].w));
    OutputStream.Append(float4(input[0].x - 0.5, input[0].y + 0.5, input[0].z, input[0].w));
    OutputStream.Append(float4(input[0].x - 0.5, input[0].y - 0.5, input[0].z, input[0].w));
    OutputStream.Append(float4(input[0].x + 0.5, input[0].y - 0.5, input[0].z, input[0].w));
}

float4 PS( float4 Pos : SV_POSITION ) : SV_Target
{
    return float4( 1.0f, 1.0f, 0.0f, 1.0f );
}

It could be that there's a mismatch between the vertex shader's output format and the geometry shader's input format. — rashmatash
can anyone help me with this <stackoverflow.com/questions/62947257/…> — Danish Mohammad

Alex Alex · Accepted Answer · 2019-08-27T20:01:30

Your geometry shader has no semantics associated with it. As a result, Direct3D has no idea how to pass the outputs of your vertex shader to your geometry shader, nor how to pass the outputs of your geometry shader to your pixel shader.

To fix this, all of your parameters need to have semantics associated with them. For input, this is easy: change

point float4 input[1]

to

point float4 input[1] : SV_POSITION

This will link up the output of the vertex shader with the geometry shader's input.

To link from the geometry shader to the pixel shader requires a bit more work. Since you've defined the pixel shader as taking a value with SV_POSITION, you need to output a value with that semantic attached. However, you have to attach that semantic to each element of the TriangleStream, and you can't attach a semantic to a type, only a variable, so something like TriangleStream<float4 : SV_POSITION> won't compile.

To solve this issue, we'll create a structure that contains a member variable with the SV_POSITION semantic:

struct GSOutput
{
    float4 pos : SV_POSITION;
};

and change our TriangleStream from

inout TriangleStream<float4> OutputStream

to

inout TriangleStream<GSOutput> OutputStream

Finally, we need to modify our method of generating the points, as we are no longer simply adding them to the list:

GSOutput gsout;
gsout.pos = float4(input[0].x + 0.5, input[0].y + 0.5, input[0].z, input[0].w);
OutputStream.Append(gsout);
gsout.pos = float4(input[0].x - 0.5, input[0].y + 0.5, input[0].z, input[0].w);
OutputStream.Append(gsout); 
gsout.pos = float4(input[0].x - 0.5, input[0].y - 0.5, input[0].z, input[0].w);
OutputStream.Append(gsout); 
gsout.pos = float4(input[0].x + 0.5, input[0].y - 0.5, input[0].z, input[0].w);
OutputStream.Append(gsout);

At this point, the compiler recognizes the file as being completely valid HLSL code, and compiles it fine.

How to setup geometry shaders in HLSL?

1 Answers