Chris, your first assumption is correct. Customers that have an Office 365 subscription do not necessarily have an Azure subscription too. However this doesn't prohibit them from signing up for applications like yours using their Office 365 Azure AD and using single sign-on with their Office 365 accounts. During sign-up customers will consent to granting your application the required delegated permissions - this experience doesn't require an Azure subscription.
All Office365 accounts live in the Azure Active Directory to which that Office365 subscription is associated. That Azure AD can contain user accounts mastered in the cloud and/or user accounts mastered in an on-premises directory that have been syncronized to the cloud. Many of our customers are small businesses with accounts only in the cloud, whereas our enterprise customers syncronize accounts from on-premises and also create cloud only accounts.
Synchronizing on-premises directory to Office 365 Azure AD also doesn't require an Azure subscription. The Azure AD Sync tool is available outside of an Azure subscription.
Finally, Azure AD administration via the Azure management portal (requires Azure subscription) provides many identity management features that aren't available in the Office365 management portal: Azure AD premium reports including anomalous sign-ins, managing access to third party applications like the one you're building (simple users and groups assignment as well as assigning users and groups to application roles), managing security groups in the cloud and many more.
Read more about Azure AD and most importantly try out both the experiences (without and with Azure subscription).
Hope this helps.