In the VS setup world, setup.exe is a bootstrapper that installs prerequisites and then runs your MSI. If you have no prerequisites you don't need setup.exe.
There are other tools that let you customize the UI - it's just that VS doesn't support that. Tools like WiX and (full) InstallShield let you build your own UI within the capabilities of MSI dialogs. In the case where there is just one MSI file being installed that's usually all that people do.
Note that MSI dialogs are nothing to do with .NET dialogs or controls - that's why they are very basic. There's no C# in there, just the controls and dialogs that MSI supports internally. You don't get to use your own C# dialogs at all.
In the case where there are multiple MSI files that you want to install as if it's a single product (SQL Server, Office, Visual Studio too I believe) then you need a wrapper (WiX has Burn bootstrapper, InstallShield has a chainer) to make it look like a single setup. If this is the case OR you want you own completely custom C# dialog UI then you write your own - or build on the WiX framework like these guys:
http://www.wrightfully.com/part-1-of-writing-your-own-net-based-installer-with-wix-overview/
http://bryanpjohnston.com/2012/09/28/custom-wix-managed-bootstrapper-application/
The architecture is based on this:
https://msdn.microsoft.com/en-us/library/windows/desktop/bb309215(v=vs.85).aspx
and MsiSetExternalUIRecord. Basically you write code to install one or MSIs while Windows Installer calls back into your code for each event during the install. This is where you have you own startup dialogs, and Windows Installer will call back with progress messages for you to display, as well as errors, files in use situations etc. It's easier to use a tool like WiX that's provided most of the framework for that.
So it depends what you mean by custom dialogs. If you just mean "not like Visual Studio" or "I need a dialog to enter something that VS doesn't offer" and you have one MSI to install, then WiX, InstallShield, Advanced Installer and other tools will do that. If you mean "I want my own completely custom C# dialogs" then you need to write your own using C# interop with MsiSetExternalUIRecord or use WiX boostrapper, which I believe is the only one out there - correct me if that's wrong.
