Fragment shader seems to floor() imprecisely

Question

I'm trying to interpolate coordinates in my fragment shader. Unfortunately if close to the upper edge the interpolated value of fVertexInteger seems to be rounded up instead of beeing floored. This happens above approximately fVertexInteger >= x.97.

Example:

floor(64.7) returns 64.0 --> correct
floor(64.98) returns 65.0 --> incorrect

The same happens on ceiling close above x.0, where ceil(65.02) returns 65.0 instead of 66.0.

Q: Any ideas how to solve this?

Note:

GL ES 2.0 with GLSL 1.0
highp floats are not supported in fragment shaders on my hardware
flat varying hasn't been a solution, because I'm drawing TRIANGLE_STRIP and can't redeclare the provoking vertex (only OpenGL 3.2+)

Fragment Shader:

varying float fVertexInteger;
varying float fVertexFraction;
void main() {

    // Fix vertex integer
    fixedVertexInteger = floor(fVertexInteger);

    // Fragment color
    gl_FragColor = vec4(
        fixedVertexInteger / 65025.0,
        fract(fixedVertexInteger / 255.0),
        fVertexFraction,
        1.0 );

}

Christian Rau Christian Rau · Accepted Answer · 2013-11-13T12:06:17

I don't know completely what the precision of a mediump float actually is in OpenGL ES on Android, but the most likely explanation is, that the literal (or the varying value in the actual code) 64.98 is already rounded to 65 (using round to nearest) in the first place, before it is even put into the floor.

Fragment shader seems to floor() imprecisely

1 Answers