it will be scaled from it's size to bigger.
No, it most certainly will not. This is a common misunderstanding about textures. Textures are nothing more than fancy look-up tables. Textures are not "scaled" relative to one another; it is merely a question of what texture coordinates you use.
In all likelihood, what you are doing is using the same texture coordinate for the screen texture as your normal map. Since we're likely talking about normalized texture coordinates, this means that you're mapping the same [0, 1] range to both of them.
To get the effect you're talking about, you need to compute texture coordinates for your normal texture that do what you need them to do. So if you have a texture coordinate that is relative to the screen texture, you must transform it into the space you want it in for the normal texture.
There are a couple of ways to do it. The manual way is to compute the ratio of the textures' sizes to one another on the CPU, then pass it to the shader. Using the numbers you've given, the size ratios would be:
(1152.0/256.0, 864.0/256.0) = (4.5, 3.375).
Make sure that this is done in floating-point math. Once done, simply pass this along in a uniform and multiply the texture coordinates by this ratio in the shader before sampling:
uniform vec2 textureRatio;
void main() {
//Get the texture coordinate.
vec4 screenTexColor = texture(screenTex, texCoord);
vec2 normTexCoord = textureRatio * texCoord;
vec4 normalValue = texture(normalTex, normTexCoord);
//Do something with these.
}
The automatic way in GLSL is to do this directly in the shader. This requires GLSL 1.30 or better. Basically, you use the available features of the language to compute the ratio:
void main() {
//Get the texture coordinate.
vec2 textureRatio = textureSize(screenTex) / textureSize(normalTex);
vec4 screenTexColor = texture(screenTex, texCoord);
vec2 normTexCoord = textureRatio * texCoord;
vec4 normalValue = texture(normalTex, normTexCoord);
//Do something with these.
}
In both of these cases, I'm assuming that your GL_TEXTURE_WRAP_S/T are set to GL_REPEAT with appropriate texture or sampler parameters.
Do note that computing the ratio on the CPU and passing it as a uniform will likely be faster than computing it in the shader. Particularly for fragment shaders, which will be run a very great deal.