Let's say I have 3 vectors, forward = Vector(1,0,0), up = Vector(0,1,0), right = Vector(0,0,1).
Now I replace the up vector by something else, but forward and right should stay in relation to the new up vector as they have to the old one.
e.g. if the new up vector is Vector(1,0,0), forward should be Vector(0,-1,0) and right should still be Vector(0,0,1).
What mathematical formula can be used for this?
You can not do it without a rotation axis. Even in your simplified (axis aligned) case where one vector is changed to the opposite direction you need a rotation axis:
Both (actually there are four)
forward( 0,-1, 0), up(1,0,0), right(0, 0, 1) and
forward( 0, 1, 0), up(1,0,0), right(0, 0,-1)
are valid solutions.
However having the rotation from Vector(0,1,0) to Vector(1,0,0), it (may) implicitly defines the rotation axis(0, 0, 1) and angle PI/2. Hence you can build a rotation matrix and multiply that matrix with the two other vectors.
