You misunderstood the role of vertex shaders in the rendering pipeline. As the name suggests, the vertex shader is executed once for each input vertex. In your case, you have 4 vertices defining your quad, so the vertex shader will be executed only 4 times. Or maybe 6 times, if you specify the vertices shared by the two triangles twice.
The output values of the vertex shader (varying
variables) are then interpolated across the triangles, and those interpolated values become the matching inputs to the fragment shader.
If you want to deform your geometry in the vertex shader, the most direct approach is that you specify more vertices. You can subdivide the quad into a grid of vertices. A 3x3 subdivision would looks something like this:
___________
|\ |\ |\ |
| \ | \ | \ |
|__\|__\|__\|
|\ |\ |\ |
| \ | \ | \ |
|__\|__\|__\|
|\ |\ |\ |
| \ | \ | \ |
|__\|__\|__\|
This is just for illustration, in reality you will most likely need a much finer subdivision depending on the curvature you're trying to achieve, and how large the final rendering is.
Calculating the vertex coordinates for this is trivial. They are just fractions of the entire quad range, with all of them being in the same plane.
With relatively recent versions of desktop OpenGL you could use a tessellation shader to generate the interior vertices, while still only specifying the 4 original vertices. But tessellation shaders are only available as an extension in OpenGL ES, and not widely supported yet.
If I understand your drawing correctly, I think there's also a small error in your calculation. I believe it should be this:
if(try.x > 0.0 ) {
try[2] = try.x - 100.0;
} else {
try[2] = -try.x - 100.0;
}
Or easier:
try[2] = abs(try.x) - 100.0;