I have a 3D mesh which represents a surface with some rough boundaries which I would like to smooth:
I am using a half edge data structure for storing the geometry so I can easily iterate over the boundary edges, vertices and faces. I can also quite easily determine whether a given pair of edges is a convex/concave using a dot and cross product.
What would be the best approach for smoothing the edges out, so they form a continuous, curvy line, rather then the sharp pattern seen in the pictures?