GLSL Vertex Shader causes either flashing colors or all red

Question

I'm writing my first vertex shader for a (here it comes) homework assignment and can't get it to function properly.

I need to implement a vertex shader (and only a vertex shader) than completely mimics the fixed function pipeline vertex shader in openGL, and use the FFP fragment shader (so write nothing for a FS). I am aware of built-in uniform variables, and I'm using them to calculate a vertice's final color based on the openGL lighting equation. I've renamed some of the values for readability's sake (normal and lightVec are normalized):

    //given as part of the assignment, not modifable
    vec4 position = gl_ModelViewMatrix * gl_Vertex;
    gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;

    //my (partial) code below here

    lightVec = (lightPos - position).xyz;
    distance = length(lightVec);
    vec3 normal = gl_NormalMatrix * gl_Normal;

    //I've stored a lot of the values in variables I defined locally
    //to make the code easier to read.
    //eg - constAtten = gl_LightSource[].constantAttenuation

    atten = 1/max((constAtten + distance*linearAtten + distance*distance*quadAtten),1); 
    ambient = ambInt * matAmbInt * atten;
    diffuse = difCol * matDifInt * max(dot(normal, lightVec),0.0) * atten;
    specular = specInt * matSpecInt * atten * pow(max(dot(normal, (gl_LightSource[i].halfVector).xyz),0.0), gl_FrontMaterial.shininess);

I'm summing ambient, diffuse, and specular for each light in the scene and storing it as "sum", followed by:

    gl_FrontColor = gl_FrontMaterial.emission + sum;
    gl_FrontColor.a = 1.0;

The result is crazy flashing colors every time I move the scene's camera. This is openGL v1.2.

Edit: Link to picture

http://i305.photobucket.com/albums/nn208/wolverine1190/shadercomp_zpsc546ac31.png

Whenever I move the camera, the colors on the left change. Could that possibly indicate an incorrectly calculated normal, or possibly the use camera coordinates somewhere I shouldn't have?

=================================================================

Resolved

I've fixed the issue. Not exactly sure why it works now, but it does. I scrapped everything and re-wrote the code from scratch with the same approach and calculations. What I did differently is:

1) I set values with constructors instead of just assignment. Eg - instead of vec3 test = someOtherVec3, I did vec3 test = vec3(someOtherVec3).

2) When normalizing I assigned the normalized result of a variable to itself instead of just calling normalize(). Eg - instead of normalize(normal), normal = normalize(normal);

3) I added to gl_FrontColor directly at each step instead of storing intermediate values in sum and adding sum to gl_FrontColor at the end.

Other than that, everything stayed the same. I can't say for sure why that fixed things, so if someone does know please comment with an explanation.

There is no fragment shader that I'm writing. The default fixed function fragment shader is used. — user2125891
Post a picture of the problem. It might ring a bell for someone. — Ben Jackson

Ben Jackson Ben Jackson · Accepted Answer · 2013-03-02T08:04:41

From your description I wonder if your colors are overflowing. What happens if you try:

    gl_FrontColor = clamp(gl_FrontMaterial.emission + sum, 0.0, 1.0);

GLSL Vertex Shader causes either flashing colors or all red

Resolved

1 Answers