My answer will only work for a closed mesh, but it will handle the case of concave and convex holes.
For the sake of explanation, lets imagine a 2D mesh.
Calculate a bounding box for the mesh. In our example the bounding box needs to store min and max values for the X and Y axis, and a corresponding vertex index for each value:
struct BoundingBox
{
float minX,maxX,minY,maxY;
int vminX,vmaxX,vminY,vmaxY;
}
Iterate over every vertex in the mesh, growing the bounding box as you add each point. When a vertex is responsible for changing one of the min/max values, store or overwrite the corresponding vmin/vmax value with the vertex mesh index.
E.g.
BoundingBox bounds;
bounds.minX = verts[0].X;
bounds.maxX = verts[0].X;
bounds.minY = verts[0].Y;
bounds.maxY = verts[0].Y;
bounds.vminX = bounds.vmaxX = bounds.vminY = bounds.vmaxY = 0;
for (int i = 1; i < numVerts; i++)
{
Vertex v = verts[i];
if (v.X < bounds.minX) { bounds.minX = v.X; bounds.vminX = i; }
if (v.X > bounds.maxX) { bounds.maxX = v.X; bounds.vmaxX = i; }
if (v.Y < bounds.minY) { bounds.minY = v.Y; bounds.vminY = i; }
if (v.Y > bounds.maxY) { bounds.maxY = v.Y; bounds.vmaxY = i; }
}
Now iterate over your boundaries until you find one which contains ALL the vertices you gathered in the bounding box. This is your outer boundary. The remaining boundaries are holes within the mesh.