Map a texture that already has perspective to a 3D object

Question

Usually, textures are like photographs taken from a top-down perspective. The theoretical camera that takes the texture image is pointing downwards and perpendicular to the plane. This texture is then mapped or projected onto arbitrary objects.

What if I start with a photograph that is already taken at an angle, and use that as a texture? Imagine I am standing outside, and pointing the camera slightly downwards, and I take a picture of the grass. Now, I have a 2D plane in my program and would like to use the grass texture on the plane. If the camera in my program has the same position wrt. the plane as my real camera did wrt. the grass, then the pixels from the photo get mapped directly 1:1 to the screen, of course. But I'd like to be able to move the virtual camera around a bit.

Since the photograph of the grass is taken at an angle (and not from above), it has much more detail to the bottom (towards the viewer) than at the top (away from the viewer). I could probably transform it to a "regular" texture (square, taken from above), but I would loose the benefit that it has more details.

I couldn't find any resources on this, and I don't even know what this kind of texture is called. Pre-projected texture? Perspective texture? I'm looking for the name of this technique if it has one, and some hints to get started, like how to do it in OpenGL or Unity, but I'm not tied to a specific technology.

Answer 1

That problem you are describing where you lose detail is fundamental to computer graphics and signal processing in general.

Your image was constructed by sampling all of the photons that hit a light sensor over a certain period of time (exposure). Originally, these photons arrived at different times from different locations in 3D space, but as you sampled them on a single 2D plane over a period of time, you already lost a lot of information (2 dimensions worth).

Now you can warp the image to alter perspective but that produces an uneven sample rate distribution in the new signal (image). The warped signal is going to have a higher sample frequency than necessary in some places and lower in others, so it creates over/under-sampling. Oversampling is generally not a huge problem. Undersampling, on the other hand, can result in another significant loss of information. The situation is more commonly referred to simply as aliasing.

In this case, since you cannot generate the image over again from your new perspective the simplest solution is interpolation. One approach would involve analyzing the angle of the image you are re-constructing and picking a reasonably shaped cross-section of extra pixels to sample and average (anisotropic filtering does just this). Filtering like this is an educated guess, that tries to make sense of missing information by assuming that sample points not in the original image follow a discernible pattern given enough neighboring samples. You are never going to make up for the lost information, simply mask it.

In any event, the general concept you are grasping at here is known as Homography. You deal with this more often than you might think; the easiest example I can give you is Google street view.

For a better visual overview, have a look at the article this diagram is from: