As you mention you need to configure the areas section of the nelmio_api_doc.yaml file as listed here https://symfony.com/doc/current/bundles/NelmioApiDocBundle/areas.html
In your case you would have the package nelmio_api_doc.yaml file as something like:
nelmio_api_doc:
areas:
default:
path_patterns: [ ^/api ]
admin:
path_patterns: [ ^/admin ]
Or whatever the patterns in your routes are to these so it knows which controller function(s) to check for the swagger annotations.
And in your routing config nelmio_api_doc.yaml:
app.swagger_ui:
path: /doc/{area}
methods: GET
defaults: { _controller: nelmio_api_doc.controller.swagger_ui, area: api }
So the api documentation will be accessible via /doc or /doc/api, any other areas can be accessed by adding appending the area name for example /doc/admin etc.
Note that I have set the route as /doc/ to prevent a conflict with the firewall pattern set for ^/api picking it up first as it may conflict, you could add an extra firewall rule to catch it first, I decided to just change the route for this.
If you went with the swagger ui route as /api/doc you would need to change the default area path patterns to:
path_patterns: [ ^/api(?!/doc$) ]
As show in the symfony docs.