I am creating a custom NSWindow with no title bar and am using NSBorderlessWindowMask to make it completely borderless. The problem I have with this however is that the window has sharp edges. As well as this there is no resize control.
How would I give a borderless window rounded corners?
This is not a duplicate of this question as that question was more about removing the title bar and it currently holds no answers.
You can make the window totally transparent and handle drawing everything yourself. The sample I have is for an OpenGL view, but it should work for a Quartz view or Cocoa view as well.
Add the following to the initializer of your NSWindow subclass where you create the new window using the NSBorderlessWindowMask constant.
[self setOpaque:NO];
[self setBackgroundColor:[NSColor clearColor]];
You will probably have to draw the resize control yourself. The sample I took this from is a full screen window so resizing isn't necessary.
Good Luck.
The easiest way to get a window with rounded corners is to place a NSBox into the window as these boxes have customizable rounded corners and customizable borders. If you then set the window to non-opaque and the background color to transparent ("clear color"), you have a NSWindow with rounded corners that draws a normal window shadow (even on older systems where such a window would otherwise not have a shadow). Most of it can be done in Interface Builder. See here for details.
