You're correct, the whole azp situation in OIDC is confusing. For what's it's worth, they have an open issue associated with it; see OIDC - Issue 973 (azp claim underspecified and overreaching).
From the definition of "aud" in JWT and its use in Connect's ID Token (relevant spec text is copied below), it seems that that the client id of the client/RP that made the authentication request has to be one of the values, or the only value, of the "aud" claim in the ID Token. That's logical and consistent and provides reliable and interoperable guidance to implementers about producing and consuming the ID Token. I think that the client id of the RP/client that made the authentication request should always be represented in the aud of the returned ID Token.
The text around "azp" in the ID Token section and the ID Token Validation section seems to maybe suggest something different, however. Like perhaps that the client id of the RP/client that made the authentication request could, in some totally unspecified circumstance, be the value of the azp claim and that the aud would not identify that client as an intended recipient. Am I misinterpreting things?
Personally, from the client application developer perspective, the best course of action seems to be honoring the ID Token validation rules which always imply that the value within azp will also be present as an aud. However, according to what's available online Google seemed to use it a bit differently so you could have a value in azp not listed within aud, so there may be situations where you play by Google rules and not (just) OIDC.
If you're implementing an OP a probably good option is also to entirely stay away from including azp in your issued tokens if possible or only include it when using multiple audiences with one of them being the value also in the azp.