First, validate that you do indeed have "Unique" permissions on your document library:
If you go to "Library Settings" for your document library, then click on "Permissions for this document library", this takes you to the Advanced permission management for the library. Once on this permission management page, at the top-left of the page in the "ribbon", if you see the buttons "Manage Parent" and "Stop Inheriting Permissions" (see screenshot), then this library is still accessible to users who have site level permission.
If this is the case, then you need to click on the "Stop Inheriting Permissions" button to ensure that your library has unique permissions.
Second, once you have ensured that the library has unique permissions, the buttons should now look like the following screenshot:
In that case, click the right-most button in that ribbon to "Check Permissions". In here you can type the username of a user to check. If you check the name of the user who can see the library, this permission check will tell you through which group that user has access.
Final Note: If you have already gone a step deeper and are using unique permissions on individual documents within this library to give access to users who would not have access to the Library, those users will now be able to "see" the library, although the only items they will be able to see inside the library are the ones with explicitly granted access.
