GLSL(330) modulo returns unexpected value

3
votes

I am currently working with GLSL 330 and came across some odd behavior of the mod() function. Im working under windows 8 with a Radeon HD 6470M. I can not recreate this behavior on my desktop PC which uses windows 7 and a GeForce GTX 260.

Here is my test code:

float testvalf = -126;
vec2 testval = vec2(-126, -126);

float modtest1 = mod(testvalf, 63.0);   //returns 63
float modtest2 = mod(testval.x, 63.0);  //returns 63
float modtest3 = mod(-126, 63.0);   //returns 0

Edit:

Here are some more test results done after IceCools suggestion below.

int y = 63;
int inttestval = -126;
ivec2 intvectest(-126, -126);
float floattestval = -125.9;


float modtest4 = mod(inttestval, 63); //returns 63
float modtest5 = mod(intvectest, 63); //returns vec2(63.0, 63.0)
float modtest6 = mod(intvectest.x, 63); //returns 63
float modtest7 = mod(floor(floattestval), 63); //returns 63
float modtest8 = mod(inttestval, y); //returns 63
float modtest9 = mod(-126, y); //returns 63

I updated my drivers and tested again, same results. Once again not reproducable on the desktop. According to the GLSL docs on mod the possible parameter combinations are (GenType, float) and (GenType, GenType) (no double, since we're < 4.0). Also the return type is forced to float but that shouldn't matter for this problem.

3
openglglslmoduloopengl-3
Could you test if int x = -126/63 returns -3? It sounds weird, but int division isn't that trivial, maybe the ATI implementation downgrades it to float division, and casts the result back to int. And as we see at modtest1 (int)(-126.0/63.0) is -3 - IceCool
I just tested -126/63, both by assigning it to a variable and by just testing the return value in an if clause. Both times the result was -2. I also tested int((-126.0/63.0)) which also returns -2 as expected. I'm slowly running out of ideas, your float precision error makes the most logical sense and I can't think of anything else that might be causing it. - wacki
Seems like something other than a precision problem is happening, since for ALL possible precisions and ALL possitble values of x and y, the invariant !(abs(mod(x, y)) >= abs(y)) should ALWAYS be true. Breaking that breaks what mod is for in graphics progams. (the weird !...>= is so that its always true for NaN values as well). - Chris Dodd

3 Answers

3
votes

I don't know that if you did it on intention but -126 is an int not a float, and the code might not be doing what you expect.

By the way about the modulo: Notice that 2 different functions are called:

The first two line:

float mod(float, float);

The last line:

int mod(int, float);

If I'm right mod is calculated like:

genType mod(genType x, float y){
return x - y*floor(x/y);
}

Now note, that if x/y evaluates -2.0 it will return 0, but if it evaluates as -2.00000001 then 63.0 will be returned. That difference is not impossible between int/float and float/float division.

So the reason is might be just the fact that you are using ints and floats mixed.

2
votes

I think I have found the answer.

One thing I've been wrong about is that mangsl's keyword for genType doesn't mean a generic type, like in a c++ template.

GenType is shorthand for float, vec2, vec3, and vec4 (see link - ctrl+f genType).

Btw genType naming is like:

genType - floats
genDType - doubles
genIType - ints
genBType - bools

Which means that genType mod(genType, float) implies that there is no function like int mod(int, float).

All the code above have been calling float mod(float, float) (thankfully there is implicit typecast for function parameters, so mod(int, int) works too, but actually mod(float, float) is called).

Just as a proof:

int x = mod(-126, 63);
Doesn't compile: error C7011: implicit cast from "float" to "int"

It only doesn't work because it returns float, so it works like this:

float x = mod(-126, 63);

Therefore float mod(float, float) is called.

So we are back at the original problem:

  • float division is inaccurate
  • int to float cast is inaccurate
  • It shouldn't be a problem on most GPU, as floats are considered equal if the difference between them is less than 10^-5 (it may vary with hardware, but this is the case for my GPU). So floor(-2.0000001) is -2. Highp floats are far more accurate than this.
  • Therefore either you are not using highp floats (precision highp float; should fix it then) or your GPU has stricter limit for float equality, or some of the functions are returning less accurate value.

  • If all else fails try:

    #extension BlackMagic : enable

0
votes

Maybe some driver setting is forcing default float precision to be mediump.

If this happens, all your defined variables will be mediump, however, numbers typed in the code will still remain highp. Consider this code:

precision mediump float;
float x = 0.4121551, y = 0.4121552;
x == y; // true
0.4121551 == 0.4121552; // false, as highp they still differ.

So that mod(-126,63.0) could be still precise enough to return the correct value, as its working with high precision floats, however if you give a variable (like at all the other cases), which will only be mediump, the function won't have enough precision to calculate the correct value, and as you look at your tests, this is what's happening:

  • All the functions that take at least one variable are not precise enough
  • The only function call that takes 2 typed numbers return the correct value.