I read lots of information about getting depth with fragment shader.
such as
http://www.opengl.org/discussion_boards/ubbthreads.php?ubb=showflat&Number=234519
but I still don't know whether or not the gl_FragCoord.z is linear.
GLSL specification said its range is [0,1] in screen sapce without mentioning it's linear or not.
I think linearity it is vital since I will use the rendered model to match depth map from Kinect.
Then if it is not linear, how to linearlize it in the world space?
but I still don't know whether or not the gl_FragCoord.z is linear.
Whether gl_FragCoord.z is linear or not depends on, the projection matrix.
While for Orthographic Projection gl_FragCoord.z is linear, for Perspective Projection it is not linear.
In general, the depth (gl_FragCoord.z and gl_FragDepth) is calculated as follows (see GLSL gl_FragCoord.z Calculation and Setting gl_FragDepth):
float ndc_depth = clip_space_pos.z / clip_space_pos.w;
float depth = (((farZ-nearZ) * ndc_depth) + nearZ + farZ) / 2.0;
The projection matrix describes the mapping from 3D points of a scene, to 2D points of the viewport. It transforms from eye space to the clip space, and the coordinates in the clip space are transformed to the normalized device coordinates (NDC) by dividing with the w component of the clip coordinates
At Orthographic Projection the coordinates in the eye space are linearly mapped to normalized device coordinates.
Orthographic Projection Matrix:
r = right, l = left, b = bottom, t = top, n = near, f = far
2/(r-l) 0 0 0
0 2/(t-b) 0 0
0 0 -2/(f-n) 0
-(r+l)/(r-l) -(t+b)/(t-b) -(f+n)/(f-n) 1
At Orthographic Projection, the Z component is calculated by the linear function:
z_ndc = z_eye * -2/(f-n) - (f+n)/(f-n)
At Perspective Projection the projection matrix describes the mapping from 3D points in the world as they are seen from of a pinhole camera, to 2D points of the viewport.
The eye space coordinates in the camera frustum (a truncated pyramid) are mapped to a cube (the normalized device coordinates).
Perspective Projection Matrix:
r = right, l = left, b = bottom, t = top, n = near, f = far
2*n/(r-l) 0 0 0
0 2*n/(t-b) 0 0
(r+l)/(r-l) (t+b)/(t-b) -(f+n)/(f-n) -1
0 0 -2*f*n/(f-n) 0
At Perspective Projection, the Z component is calculated by the rational function:
z_ndc = ( -z_eye * (f+n)/(f-n) - 2*f*n/(f-n) ) / -z_eye
Since the normalized device coordinates are in range (-1,-1,-1) to (1,1,1) the Z-coordinate has to be mapped to the depth buffer range [0,1]:
depth = (z_ndc + 1) / 2
Then if it is not linear, how to linearize it in the world space?
To convert form the depth of the depth buffer to the original Z-coordinate, the projection (Orthographic or Perspective), and the near plane and far plane has to be known.
Orthographic Projection
n = near, f = far
z_eye = depth * (f-n) + n;
Perspective Projection
n = near, f = far
z_ndc = 2.0 * depth - 1.0;
z_eye = 2.0 * n * f / (f + n - z_ndc * (f - n));
If the perspective projection matrix is known this can be done as follows:
A = prj_mat[2][2]
B = prj_mat[3][2]
z_eye = B / (A + z_ndc)
See also the answer to
How to recover view space position given view space depth value and ndc xy
Assuming a usual perspective projection matrix, once the perspective division (by gl_Position.w) step is performed the depth loses its linearity, so gl_FragCoord.z is not linear. For a more detailed explanation read @Dreamer's answer.
To revert back to linear you should perform 2 steps:
1) Transform the variable gl_FragCoord.z to normalized devices coordinates in the range [-1, 1]
z = gl_FragCoord.z * 2.0 - 1.0
2) Apply the inverse of the projection matrix (IP). (You can use arbitrary values for x and y), and normalize for the last component.
unprojected = IP * vec4(0, 0, z, 1.0)
unprojected /= unprojected.w
you will obtain a point in view space (or camera space, you name it) that has a linear z between znear and zfar.
Whether gl_FragCoord.z is linear or not depends on your transformation matrix. gl_FragCoord.z is determined by computing gl_Position.z / gl_Position.w for all vertices of your triangle and then interpolating the result over all fragments of that triangle.
So gl_FragCoord.z is linear when your transformation matrix assigns a constant value to gl_Position.w (which usually happens with ortho projection matrices) and is non-linear when gl_Position.w depends on the x, y, or z coordinate of your input vector (which happens with perspective projection matrices).
Its up to you to decide if you want linear Z or not, everythings relies on your projection matrix. You may read this:
http://www.songho.ca/opengl/gl_projectionmatrix.html
Which explains very well how projection matrices works. It may be better to have non-linear Z in order to have better precision in the foreground and less in the backgrounds, depth artifacts are less visible when far away...
