glsl vector*matrix different to hlsl

4
votes

I have two (identical) shaders, one in hlsl and one in glsl. In the pixel shader, I am multiplying a vector by a matrix for normal transformations. The code is essentially:

HLSL

float3 v = ...;
float3x3 m = ...;
float3 n = mul(v, m);

GLSL

vec3 v = ...;
mat3 m = ...;
vec3 n = v * m;

This should do a row vector multiplication, yet in glsl it doesn't. If I explicitly type out the algorithm, it works for both. Both the glsl and hlsl spec, from what I can tell, says they should do a row vector multiply if the vector is on the left hand side, which it is.

The other confusing thing is that I multiply a vector by a matrix in the vertex shader with the vector on the left, yet that works fine in both glsl and hlsl. This leads me to guess that it is only an issue in the fragment/pixel shader.

I pass the matrix from the vertex shader to the fragment shader using:

out vec3 out_vs_TangentToWorldX;
out vec3 out_vs_TangentToWorldY;
out vec3 out_vs_TangentToWorldZ;

out_vs_TangentToWorldX = tangent * world3D;
out_vs_TangentToWorldY = binormal * world3D;
out_vs_TangentToWorldZ = normal * world3D;

and in the fragment shader I reconstruct it with:

in vec3 out_vs_TangentToWorldX;
in vec3 out_vs_TangentToWorldY;
in vec3 out_vs_TangentToWorldZ;

mat3 tangentToWorld;
tangentToWorld[0] = out_vs_TangentToWorldX;
tangentToWorld[1] = out_vs_TangentToWorldY;
tangentToWorld[2] = out_vs_TangentToWorldZ;
1
matrixvectorglslshaderhlsl
How are you passing the matrix to the pixel shader? – CynicismRising
in/out attributes from the vertex shader – Programmdude
Is it a per vertex matrix? If each vertex outputs a different matrix the hw would try to interpolate it across each triangle, probably incorrectly. Is it possible to pass it as a uniform? – CynicismRising
It is a per vertex matrix, but its also a per vertex matrix in hlsl too. That wouldn't explain why writing the algorithm for row vector multiplication works, yet doing the v * m command itself doesn't. – Programmdude
Did you tried to swap the operands in GLSL? I'm not sure anymore, but I thought, that I read some time ago, that HLSL is working with row-major matrices and GLSL with col-major or vice versa. – Gnietschow

1 Answers

4
votes

HLSL matrices are row-major, GLSL are column-major. So if you pass your matrix into GLSL shader using the same memory layout as you pass it into HLSL, then your HLSL rows will become GLSL columns. And you should use column-major multiplication in your GLSL shader to get same effect as in HLSL.

Just use

vec3 n = m * v;