Let X(i, alpha) be the X-coodrinate of vertex i after rotation by angle alpha.
We always assume -PI <= alpha <= PI .
Let i=rightmost(alpha) if X(i, alpha)>=X(j, alpha) for all j. Let i=second_rightmost(alpha) if X(i, alpha)>=X(j,alpha) for all j except one. Similarly define leftmost() and second_leftmost().
Let us prove the following: if X(i, alpha) >= X(j, alpha), and X(i, beta) >= X(j, beta), and beta-alpha < PI, then X(i, gamma) >= X(j, gamma) for all gamma in [alpha, beta]. Indeed, X(i, alpha)=x[i]*cos(alpha)-y[i] * sin(alpha), where (x[i], y[i]) are initial position of the vertex i. Therefore, X(i, a) - X(j, a) = c1*cos(a)-c2*sin(a), where c1=x[i]-x[j],c2=y[i]-y[j]. Let f(a)=X(i,a)-Y(i,a). The function f is continuous, and changes its sign when tan(a)=c1/c2, that is, a=atan2(c1,c2)+PI*n. If beta-alpha
Now we have:
- if rightmost(alpha)=rightmost(beta) and beta-alpha < PI, then rightmost(x)=rightmost(alpha) for all alpha < x < beta.
- if i=rightmost(alpha)=second_rightmost(beta), and j=rightmost(beta)=second_rightmost(alpha), and beta-alpha < PI, then for all x between alpha and beta, rightnost(x) is either i or j, and the point of change is atan2(y[i]-y[j], x[i]-x[j]).
This is enough to obtain with binary search at which interval which point is the rightmost. By sign inversion of angles, we get the leftmost point intervals. Since we know at which interval each point is leftmost, and each is rightmost, we can calculate the values at the borders between intervals, and select the minimal one.