How can I use screen space coordinates directly with OpenGL?

Question

I've follow the guide I find on https://learnopengl.com/ and managed to render a triangle on the window, but this whole process just seems over complicated to me. Been searching the internet, found many similar questions but they all seem outdated and none of them gives a satisfied answer.

I've read the guide Hello Triangle from the site mentioned above which has a picture explaining the graphics pipeline, with that I got the idea about why a seemingly simple task as drawing a triangle shape on screen takes so many steps.

I have also read the guide Coordinate Systems from the same site, which tells me what is the "strange (to me) coordinate" that OpenGL uses (NDC) and why it uses that.

(Here's the picture from the guide mentioned above which I think would be useful for describing my question.)

---

The question is: Can I use the final SCREEN SPACE coordinates directly?

All I want is do some 2D rendering (no z-axis) and the screen size is known (fixed), as such I don't see any reason why should I use a normalized coordinate system instead of a special one bound to my screen.

eg: on a 320x240 screen, (0,0) represents the top-left pixel and (319,239) represents the bottom-right pixel. It doesn't need to be exactly what I describe it, the idea is every integer coordinate = a corresponding pixel on the screen.

I know it's possible to setup such a coordinate system for my own use, but the coordinates would be transformed all around and in the end back to screen space - which is what I have in the first place. All these just seems to be wasted work to me, also is't it gonna introduce precision lost when the coordinates get transformed?

---

Quote from the guide Coordinate Systems (picture above):

4. After the coordinates are in view space we want to project them to clip coordinates. Clip coordinates are processed to the -1.0 and 1.0 range and determine which vertices will end up on the screen.

So consider on a 1024x768 screen, I define the Clip coordinates as (0,0) to (1024,678), where:

(0,0)--------(1,0)--------(2,0)  
  |            |            |    
  |   First    |            |    
  |   Pixel    |            |    
  |            |            |    
(0,1)--------(1,1)--------(2,1)        . . .
  |            |            |    
  |            |            |    
  |            |            |    
  |            |            |    
(0,2)--------(1,2)--------(2,2)  

              .
              .
              .
                                          (1022,766)---(1023,766)---(1024,766)
                                               |            |            |
                                               |            |            |
                                               |            |            |
                                               |            |            |
                                          (1022,767)---(1023,767)---(1024,767)
                                               |            |            |
                                               |            |   Last     |
                                               |            |   Pixel    |
                                               |            |            |
                                          (1022,768)---(1023,768)---(1024,768)

Let's say I want to put a picture at Pixel(11,11), so the clip coordinates for that would be Clip(11.5,11.5) this coordinate is then processed to the -1.0 and 1.0 range:

11.5f * 2 / 1024 - 1.0f = -0.977539063f // x
11.5f * 2 /  768 - 1.0f = -0.970052063f // y

And I have NDC(-0.977539063f,-0.970052063f)

5. And lastly we transform the clip coordinates to screen coordinates in a process we call viewport transform that transforms the coordinates from -1.0 and 1.0 to the coordinate range defined by glViewport. The resulting coordinates are then sent to the rasterizer to turn them into fragments.

So take the NDC coordinate and transform it back to screen coordinate:

(-0.977539063f + 1.0f) * 1024 / 2 = 11.5f        // exact
(-0.970052063f + 1.0f) *  768 / 2 = 11.5000076f  // error

The x-axis is accurate as 1024 is power of 2, but since 768 isn't so y-axis is off. The error is very little, but it's not exactly 11.5f, so I guess there would be some sort of blending happen instead of 1:1 representation of the original picture?

---

To avoid the rounding error mentioned above, I did something like this:

First I set the Viewport size to a size larger than my window, and make both width and height a power of two:

GL.Viewport(0, 240 - 256, 512, 256); // Window Size is 320x240

Then I setup the coordinates of vertices like this:

float[] vertices = {
    //  x       y
      0.5f,   0.5f, 0.0f, // top-left
    319.5f,   0.5f, 0.0f, // top-right
    319.5f, 239.5f, 0.0f, // bottom-right
      0.5f, 239.5f, 0.0f, // bottom-left
};

And I convert them manually in the vertex shader:

#version 330 core
layout (location = 0) in vec3 aPos;

void main()
{
    gl_Position = vec4(aPos.x * 2 / 512 - 1.0, 0.0 - (aPos.y * 2 / 256 - 1.0), 0.0, 1.0);
}

Finally I draw a quad, the result is:

This seem to produce a correct result (the quad has a 320x240 size), but I wonder if it's necessary to do all these.

1. What's the drawback of my approach?
2. Is there a better way to achieve what I did?

---

It seems rendering in wireframe mode hides the problem. I tried to apply texture to my quad (actually I switched to 2 triangles), and I got different result on different GPU and non of them seem correct to me:

Left: Intel HD4000 | Right: Nvidia GT635M (optimus)

I set GL.ClearColor to white and disabled the texture.

While both result fills the window client area (320x240), Intel give me a square sized 319x239 placed to the top left while Nvidia gives me a square sized 319x239 placed on the bottom left.

This is what it looks like with texture turned on.

The texture:

(I have it flipped vertically so I can load it easier in code)

The vertices:

float[] vertices_with_texture = {
    //  x       y           texture x     texture y
      0.5f,   0.5f,        0.5f / 512, 511.5f / 512, // top-left
    319.5f,   0.5f,      319.5f / 512, 511.5f / 512, // top-right
    319.5f, 239.5f,      319.5f / 512, 271.5f / 512, // bottom-right ( 511.5f - height 240 = 271.5f)
      0.5f, 239.5f,        0.5f / 512, 271.5f / 512, // bottom-left
};

Now I'm completely stuck.

I thought I'm placing the quad's edges on exact pixel center (.5) and I'm sampling the texture also at exact pixel center (.5), yet two cards give me two different result and non of them is correct (zoom in and you can see the center of the image is slightly blurred, not a clear checker-board pattern)

What am I missing?

---

I think I figured out what to do now, I've posted the solutions as an answer and leave this question here for reference.

Answer 1

Okay.... I think I finally get everything work as I expect. The problem was that I forgot pixels have size - it's so obvious now that I can't understand why I missed that.

In this answer, I'll refer the Clip Space coordinates as Clip(x,y), where x/y ranges from 0 to screen width/height respectively. For example, on a 320x240 screen the Clip Space is from Clip(0,0) to Clip(320,240)

---

Mistake 1:

When trying to draw a 10 pixels square, I drew it from Clip(0.5,0.5) to Clip(9.5,9.5) .

It's true that those are the coordinates for the pixel centers of the begin (top-left) pixel and end (bottom-right) pixel of the square, but the real space the square takes is not from the pixel center of it's begin pixel and end pixel.

Instead, the real space said square takes is from the top-left corner of the begin pixel to the bottom-right corner of the end pixel. So, the correct coordinates I should use for the square is Clip(0,0) - Clip(10, 10)

---

Mistake 2:

Since I got the size of the square wrong, I was trying to map the texture wrong as well. Since now I have the square size fixed, I'll just fix the coordinates for the textures accordingly as well.

However, I found a better solution: Rectangle Texture , from which I quote:

When using rectangle samplers, all texture lookup functions automatically use non-normalized texture coordinates. This means that the values of the texture coordinates span (0..W, 0..H) across the texture, where the (W,H) refers to its dimensions in texels, rather than (0..1, 0..1).

This is VERY convenient for 2D rendering, for one I don't have to do any coordinates transforms, and for a bonus I don't need to flip the texture vertically anymore.

I tried it and it works just as I expected, but I come across a new problem: bleeding on edge when the texture is not placed on exact pixel grids.

---

Solving the bleeding:

If I use different textures for every square, I can avoid this problem by having the sampler clamp everything outside the texture with TextureWrapMode.ClampToEdge

However, I'm using a texture atlas, a.k.a "sprite sheet". I've searched the internet found solutions like:

1. Manually add padding to each sprite, basically leave safe spaces for bleeding error.

   This is straight forward but I really don't like it as I'll loose the ability to tightly pack my texture, and it makes calculating the texture coordinates more complex also it just a waste of space anyway.

2. Use GL_NEAREST for GL_TEXTURE_MIN_FILTER and shift the coordinates by 0.5/0.375

   This is very easy to code and it works fine for pixel arts - I don't want liner filter to make them blurry any way. But I'd also like to keep the ability to display a picture and move it around smoothly rather than jumping from pixel to pixel, so I need to have the ability to use GL_LINEAR.

One solution: Manually clamp the texture coordinates.

It's basically the same idea as TextureWrapMode.ClampToEdge, but at a per sprite basic rather than only on the edges of entire sprite sheet. I coded the fragment shader like this (just a proof-of-concept, I certainly need to improve it):

#version 330 core
out vec4 FragColor;
in vec2 TexCoord;

uniform sampler2DRect texture1;

void main()
{
    vec2 mTexCoord;
    mTexCoord.x = TexCoord.x <= 0.5 ? 0.5 : TexCoord.x >= 319.5 ? 319.5 : TexCoord.x;
    mTexCoord.y = TexCoord.y <= 0.5 ? 0.5 : TexCoord.y >= 239.5 ? 239.5 : TexCoord.y;
    
    FragColor = texture(texture1, mTexCoord);
}

(Yes, the "sprite" I use in this case is 320x240 which takes my entire screen.)

It's soooo easy to code thanks to the fact that Rectangle Texture uses non-normalized coordinates. It works well enough and I just called it a day and wrapped it up here.

Another solution (untested yet): Use Array Texture instead of texture atlas

The idea is simple, just set TextureWrapMode.ClampToEdge and have the sampler do it's job. I haven't look further into it but it seems to work in concept. Yet I really like the way coordinates work with Rectangle Texture and would like to keep it if possible.

---

Rounding Error

When trying to animate my square on the screen, I come to a very strange result: (notice the lower left corner of the square when the values on the left reads X.5)

This only happens on my iGPU (Intel HD4000) and not the dGPU (Nvidia GT635M via optimus). It looks like this because I fitted all coordinates to pixel center (.5) in fragment shader:

#version 330 core
out vec4 FragColor;
in vec2 TexCoord;

uniform sampler2DRect texture1;

void main()
{
    vec2 mTexCoord;

    // Clamp to spirte
    mTexCoord.x = clamp(TexCoord.x, 0.5, 319.5);
    mTexCoord.y = clamp(TexCoord.y, 0.5, 239.5);

    // Snap to pixel
    mTexCoord.xy = trunc(mTexCoord.xy) + 0.5;
    
    FragColor = texture(texture1, mTexCoord);
}

My best guess is that the iGPU and dGPU rounds differently when converting coordinates to NDC (and back to screen coordinates)

Using a quad and a texture, both sized power-of-2 would avoid this problem. There's also workarounds like adding a small amount (0.0001 is enough on my laptop) to mTexCoord.xy before trunc.

Update: the solution

Okay, after a good sleep I come up with a relatively simple solution.

1. When dealing with pictures, nothing needs to be changed (let linear filter do it's job) Since there's always gonna be rounding error, I basically just give up at this point and live with it. It won't be noticeable to human eyes any way.

2. When trying to fit the pixels in texture into pixel grid on screen, in addition to snapping the texture coordinates in fragment shader (as seen above), I also have to pre-shift the texture coordinates in vertex shader:

#version 330 core
layout (location = 0) in vec2 aPos;
layout (location = 1) in vec2 aTexCoord;

out vec2 TexCoord;
uniform mat4 projection;

void main()
{
    TexCoord = aTexCoord + mod(aPos , 1);
    gl_Position = projection * vec4(aPos.xy, 0.0, 1.0);
}

The idea is simple: when the square is placed at Clip(10.5, 10.5), shift the texture coordinate to Pixel(0.5, 0.5) as well. Of course this would mean the end coordinate gets shifted out of sprite region Pixel(320.5, 320.5) but this is fixed in fragment shader using clamp so no need to worry about it. Any coordinates in between (like Pixel(0.1, 0.1)) wound be snapped to Pixel(0.5, 0.5) by fragment shader as well, which creates a pixel-to-pixel result.

This gives me a consistent result across my iGPU(intel) and dGPU(Nvidia):

Positioned at Clip(10.5,10.5), notice the artifacts in lower-left corner is gone.

---

To sum it up:

1. Setup the Clip Space coordinates as a 1:1 represent to the screen.

2. Remember that pixels have size when calculating the size of a sprite.

3. Use the correct coordinates for textures and fix the bleeding on sprite edges.

4. When trying to snap texture pixels to screen pixels, take extra care about vertex coordinates in addition to texture coordinates.

Answer 2

Can I use the final SCREEN SPACE coordinates directly

No you can't. You have to transform the coordinates. Either on the CPU or in the shader (GPU).

If you want to use window coordinates, then you've to set an Orthographic projection matrix, which transforms the coordinates from x: [-1.0, 1.0], y: [-1.0, 1.0] (normalized device space) to your window coordinates x: [0, 320], y: [240, 0].

e.g. by the use of glm::ortho

glm::mat4 projection = glm::orhto(0, 320.0f, 240.0f, 0, -1.0f, 1.0f);

e.g. OpenTK Matrix4.CreateOrthographic

OpenTK.Matrix4 projection = 
    OpenTK.Matrix4.CreateOrthographic(0, 320.0f, 240.0f, 0, -1.0f, 1.0f);

In the vertex shader, the vertex coordinate has to be multiplied by the projection matrix

in vec3 vertex;

uniform mat4 projection;

void main()
{
    gl_Position = projection * vec4(vertex.xyz, 1.0);
}

---

For the sake of completeness Legacy OpenGL, glOrtho:
(do not use the old and deprecated legacy OpenGL functionality)

glOrtho(0.0, 320.0, 240.0, 0.0, -1.0, 1.0); 

Answer 3

As others have mentioned you need an orthographic projection matrix. You can either implement it yourself by following guides such as these: http://learnwebgl.brown37.net/08_projections/projections_ortho.html Or maybe the framework you're using already has it.

If you set the right/left/top/bottom values to match your screen resolution then a point of coordinate x,y (z being irrelevant, you can use a 2d vector with a 4x4 matrix) will become the same x,y on the screen.

You can move the view around like a camera if you multiply this projection matrix by a translation matrix (first the translation then the projection). Then you pass this matrix to your shader and multiply the position of the vertex by it.