The first observation is that any edge of a rectangle must touch one of the points. Edges that didn't touch a point could be pulled back, resulting in less area.
Given n points, there are thus n selections total for left1, right1, bottom1, top1, left2, right2, bottom2 and top2. This gives a simple O(n^8) algorithm already: try all possible assignments and remember the one giving the least total area (right1 - left1)(top1 - bottom1) + (right2 - left2)(top2 - bottom2). Indeed, you can skip any combinations with right < left or top < bottom. This gives a speedup, though it does not change the O(n^8) bound.
Another observation is that the edges should stay within the minimum and maximum X and Y bounds of all points. Find the minimum and maximum X and Y values of any points. Call these minX, maxX, minY and maxY. At least one of your rectangles will need to have its left, right, bottom and top edges, respectively, at those levels.
Because minx, maxX, minY and maxY must be assigned to one of the two rectangles, and there are exactly 2^4 = 16 ways to do this, you can try each of the four possible assignments with the remaining coordinates assigned as above. This gives an O(n^4) algorithm: O(n) to get minX, maxX, minY and maxY, and then O(n^4) to fill in the four unassigned variables for each of 16 assignments of minX, maxX, minY and maxY to the eight edge coordinates.
We have so far ignored the requirement that rectangles not overlap. To accommodate that, we must ensure at least one of the following four conditions holds true:
- a horizontal line at Y coordinate h with top1 <= h <= bottom2
- a horizontal line at Y coordinate h with top2 <= h <= bottom1
- a vertical line at X coordinate w with right1 <= h <= left2
- a vertical line at X coordinate w with right2 <= h <= left1
The two rectangles overlap if and only if all four of these conditions are simultaneously false. This allows us to skip over candidate solutions, giving a speedup but not changing the asymptotic bound O(n^4). Note that we need to check this condition specifically since, otherwise, optimal solutions might have overlap (exercise: show such an example).
Let's try to shave some more time off of this. Assume we have non-overlapping rectangles by condition #1 above. Then there are n choices for h; we can try each of these n choices and then determine the area of the resulting selections by finding the minimum and maximum coordinates of points in the resulting halves. By trying all n selections for h, we can determine the "best case" vertical split. We need not try condition #2, since the only difference is in the ordering of the rectangles which is arbitrary. We must also try condition #3 with a horizontal split. This suggests an O(n^2) algorithm:
- For each point, choose h = point.y
- Separate the points into groups with point.y <= h and point.y > h.
- Find the minimum and maximum X and Y coordinates of both subsets of points.
- Compute the sum of the areas of the two rectangles.
- Remember the minimum area obtained from the above and the corresponding h.
- Repeat, but using w and X coordinates.
- Determine whether minimum area was obtained for a vertical or horizontal split
- Return the corresponding rectangles as the answer
Can we do even better? This is O(n^2) and not O(n) because for each choice of h
and w
we need to then find the minimum and maximum coordinates of each subgroup. This assumes a linear scan through both subgroups. We don't actually need to do this for the min and max X/Y coordinates when scanning horizontally/vertically, since those will be known. What we need is a solution to this problem:
Given n points and a value h, find the maximum X coordinate of any point whose Y coordinate is no greater than h.
The obvious solution I give above is O(n^2), but you might be able to find an O(n log n) solution by clever application of sorting or maybe even an O(n) solution by some even more clever method. I will not attempt this.
Our solution is O(n^2); the theoretically optimal solution is Omega(n) since you must at least look at all the points. So we're pretty close but there is room for improvement.