Calculating normals on terrain mesh

Question

I'm trying to optimize my terrain by reducing the triangle count while keeping as much detail as possible. The reduction worked fine, I cut the number of vertices by a 5th without much visual loss. There's a problem with the calculations of the normals on this new asymmetrical mesh.

I have normals per vertex and here's the snippet for calculating normals:

private void calcNormal(Vector<Triangle_dt> triangles, Point_dt point) {

    Vec3 normal = new Vec3(0, 0, 0);
    for (Triangle_dt triangle : triangles) {
        Vec3 a = getPos(triangle.p1());
        Vec3 b = getPos(triangle.p2());
        Vec3 c = getPos(triangle.p3());

        Vec3 AB = b.subtract(a);
        Vec3 AC = c.subtract(a);

        normal = normal.add(AB.cross(AC));
    }

    setNormal(point, normal.getUnitVector());
}

Where triangles are the triangles connected to the vertex (point). I Add all the triangle normals together (without normalizing to make the final vector weighted by triangle area) and then finally normalizing the end result.

I believe the calculations are correct but there are annoying artifacts in the result (it's lit with directional light):

As you can see there's unwanted lines where the vertices are sparse. It's caused due to small clusters of points close together but far from the next set of points (see next picture below). Any idea of how to prevent this? Here's the same view with point rendering:

Do you have the original height map? Then it is better to sample normals per fragment from a terrain normal-map texture. — Yakov Galka
I do, that might be worth a shot, I'm afraid it'll be a bit heavy to load (the 1-dimensional height map is 166 MB saved in plain text) — Johan
(1) don't store in text!? (2) using 4 bytes/vertex texture might be cheaper than 3 floats/vertex you use now. (3) generally if you have a huge terrain model then you're better to load it in tiles according to the distance from camera. (4) using uniform tessellation with resolution as a function of camera distance and tiled textures of heights and normals might be easier to implement and give you better results than what you do now. — Yakov Galka
@ybungalobill (1) Well that was only for debugging purpose, it was just to give you an idea of the size of the heightmap. Thanks a lot for your help and suggestions, I ended up doing what you said, rendering a normal map from the original height map (see my answer below for details). Spektre: That's kinda what I allready did, it works great on your example where the vertices are regularly spaced but it didn't make it smooth enough in my example due to the irregular triangle sizes. — Johan

Johan Johan · Accepted Answer · 2017-05-29T20:53:06

Thanks to ybungalobill I did the following to make it work:

Created a normal map from the original heighmap (symmetrical grid) using the following code:

Calculating normals from height map

// Calculating normals from height map
public void calcNormals() {
    Vec3 up = new Vec3(0, 1, 0);
    float sizeFactor = 1.0f / (8.0f * cellSize);
    normals = new Vec3[rows * cols];

    for (int row = 0; row < rows; row++) {
        for (int col = 0; col < cols; col++) {
            Vec3 normal = up;

            if (col > 0 && row > 0 && col < cols - 1 && row < rows - 1) {
                float nw = getValue(row - 1, col - 1);
                float n = getValue(row - 1, col);
                float ne = getValue(row - 1, col + 1);
                float e = getValue(row, col + 1);
                float se = getValue(row + 1, col + 1);
                float s = getValue(row + 1, col);
                float sw = getValue(row + 1, col - 1);
                float w = getValue(row, col - 1);

                float dydx = ((ne + 2 * e + se) - (nw + 2 * w + sw)) * sizeFactor;
                float dydz = ((sw + 2 * s + se) - (nw + 2 * n + ne)) * sizeFactor;

                normal = new Vec3(-dydx, 1.0f, -dydz).getUnitVector();
            }

            normals[row * cols + col] = normal;
        }
    }
}

Creating image from normals

public static BufferedImage getNormalMap(Terrain terrain) {
    Vec3[] normals = terrain.getNormals();
    float[] pixels = new float[normals.length * 3];

    for (int i = 0; i < normals.length; i++) {
        Vec3 normal = normals[i];
        float x = (1.0f + normal.x) * 0.5f;
        float y = (1.0f + normal.y) * 0.5f;
        float z = (1.0f + normal.z) * 0.5f;
        pixels[i * 3] = x * MAX;
        pixels[i * 3 + 1] = y * MAX;
        pixels[i * 3 + 2] = z * MAX;
    }

    BufferedImage img = new BufferedImage(cols, rows, BufferedImage.TYPE_INT_RGB);
    WritableRaster imgRaster = img.getRaster();
    imgRaster.setPixels(0, 0, cols, rows, pixels);
    return img;
}

Applied the image in the fragment shader and calculated the texture coordinates using the vertex position from the vertex shader:

Part of fragment shader:

void main() {
    vec3 newNormal = texture(normalMap, vec2(worldPos0.x / maxX, worldPos0.z / maxZ)).xyz;
    newNormal = (2.0 * newNormal) - 1.0;
    outputColor = calcColor(normalize(newNormal));
}

The result is the following:

Same view with point rendering:

In other words: few vertices but visually high detail terrain

Calculating normals on terrain mesh

1 Answers