I want to have a pipeline build and run it on our self-hosted build agent
The answer is yes.
To use your self-hosted build agent in the YAML, you need to use following syntax:
pool:
name: <YourAgentName>
Check the YAML schema reference for some more details. And this thread about how to create a your private agent (If you know how to create a private agent, please ignore it).
Can I have a pipeline build without having to define a pool/define
many pools as fallback?
You could have a pipeline build without having to define a pool.
Because when creating a YAML pipeline, the default agent pool always is set as Hosted Ubuntu 1604
. In the past, user could select the specific agent pool as the Default agent pool for YAML from the drop-down list on YAML tab. After the recent update Sprint 155
, user will not select the agent pool. If you want to use another agent pool, you must set the specific agent pool in the YAML script.

Then switch to YAML tab, you will find the option for default agent setting:

pool:
vmImage: string
Hope this helps.