Security rules cannot read from another service, so you won't be able to implement groups in the way you describe here. You'll instead have to encode the knowledge of what groups somebody is a member off in either your rules itself, or in the ID token of the user (as a custom claim).
Embedding group membership in the user's token
For an example of the latter, see this snippet from the Firebase documentation on making data "group private":
Another equally common use case will be to allow group permissions on an object, such as allowing several team members to collaborate on a shared document. There are several approaches to doing this:
Mint a Firebase Authentication custom token that contains additional information about a group member (such as a group ID)
Include group information (such as a group ID or list of authorized uids) in the file metadata
Once this data is stored in the token or file metadata, it can be referenced from within a rule:
// Allow reads if the group ID in your token matches the file metadata's `owner` property
// Allow writes if the group ID is in the user's custom token
match /files/{groupId}/{fileName} {
allow read: if resource.metadata.owner == request.auth.token.groupId;
allow write: if request.auth.token.groupId == groupId;
}
These days you don't need to mint a full custom token to include the group ID, but you can include information in a regular token as a custom claim through the Firebase Admin SDKs. For example in Node.js, you could add a groupId like is used above with:
admin.auth().setCustomUserClaims(uid, {groupId: "group"})
This approach works well for cases where you can clearly identify a single group (or small set of groups) that the user is a part of. If you get large group membership hierarchies, with inclusions and exclusions, it gets harder to capture them in claims, especially all claims have to be less than 1,000 bytes.
Embedding group membership in the rules
The other approach is to embed knowledge about group membership into the rules itself. In this scenario you would take the structure you now have in your database, and encode that into your rules.
You can do this manually in the case your group memberships are fairly stable. And in that case you'd end up with hard-coded UIDs in your security rules.
But since you chose to store the group membership in a database, these memberships are likely at least somewhat dynamic. In that case you may want to use a combination of the previous approach to capture part of the membership, and then generate the security rules for the other parts.
You can then deploy the generated rules, either periodically, or whenever the group membership changes.
While this is a more brute-force approach than (just) embedding the information in the user's tokens, I've seen it been used to create advanced membership tests. My main dislike for it, is that the generated security rules tend to become unreadable. You can mitigate that by generating the "group check" into a separate function in your rules, so that the hand-written rules (that you'll more frequently be reading) can just include a call to isMemberOfValidGroup(...)
or something like that.