Here's a recursive solution that works for dungeons of arbirary size and relatively few monsters.
If there is one monster (x, y) in the dungeon (n, w, s, e), there are two cases. Case 1 is trivial: The monster is outside the dungeon. Then the maximum rectangle is the dungeon. (Dungeons are always rectangular, right?).
In Case 2, the maximum rectangle is one of the rectangles north, west, south or east of the monster:
NNNNNNNNNN ....EEEEEE .......... WWW.......
NNNNNNNNNN ....EEEEEE .......... WWW.......
NNNNNNNNNN ....EEEEEE .......... WWW.......
NNNNNNNNNN ....EEEEEE .......... WWW.......
NNNNNNNNNN ....EEEEEE .......... WWW.......
...@...... ...@EEEEEE ...@...... WWW@......
.......... ....EEEEEE SSSSSSSSSS WWW.......
.......... ....EEEEEE SSSSSSSSSS WWW.......
.......... ....EEEEEE SSSSSSSSSS WWW.......
Now apply this reasoning recursively for your list of monster locations and keep track of the maximum so far. Here's some pseudo code:
max_area = 0
max_rect = Null
sub find_max_rect(Dungeon, Monsters)
if area(Dunegon) <= max_area:
return # save time for small dungeons
if Monsters is empty:
if area(Dungeon) > max:
max_rect = Dungeon
max_area = area(Dungeon)
else
M = Monsters.pop() # Remove head from list
if M in Dungeon:
find_max_rect(Subrect(Dungeon, M, North), Monsters)
find_max_rect(Subrect(Dungeon, M, East), Monsters)
find_max_rect(Subrect(Dungeon, M, South), Monsters)
find_max_rect(Subrect(Dungeon, M, West), Monsters)
else:
find_max_rect(Dungeon, Monsters)
Note: I've actually made a glaring mistake in the sketches above: @
represents, of course, the player and not a monster.