CreateFileW with exclusive access succeeds when LoadLibraryEx called with LOAD_LIBRARY_AS_DATAFILE

Question

I was under the assumption that if you use CreateFileW requesting exclusive access, that if it was already opened I should receive a sharing violation. However, I've come across an example where this is not the case. If LoadLibraryEx is called to load a DLL with the LOAD_LIBRARY_AS_DATAFILE flag set, then CreateFileW returns successfully. However there are operations that will be rejected (such as setting attributes like end of file). Is there a better way to detect if these files are open? Am I using CreateFileW incorrectly? Loading a DLL without the LOAD_LIBRARY_AS_DATAFILE flag does result in CreateFileW failing indicating it can not access the file because it is in use by another process.

HMODULE hModule = LoadLibraryEx(L"\\Pathto\\module.dll", NULL, NULL);
DWORD access = GENERIC_READ | GENERIC_WRITE | WRITE_OWNER | WRITE_DAC | ACCESS_SYSTEM_SECURITY;
DWORD creation = OPEN_EXISTING;
DWORD flags = FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT;
HANDLE file = CreateFileW(L"\\Pathto\\module.dll", access, 0, NULL, creation, flags, 0);

The above will result in CreateFileW failing with error code 32 (can't access cause it is in use by another process) which is the expected result. If I add the flag to LoadLibraryEx:

HMODULE hModule = LoadLibraryEx(name, NULL, LOAD_LIBRARY_AS_DATAFILE);    

And use the exact same call to CreateFileW then I'm told it's successful, although I will run into problems later when attempting to set end of file (as an example). Additionally, deleting the file will fail with an access denied error (without the application or any other having an open handle).

Other odd behavior involves hard links with loaded libraries. If I generate a new hard link to the loaded module, and try to delete the alternate newly created hard link, it also fails (NO open file handles, just loaded module). Why? Shouldn't it just remove the link and dereference the link count, or schedule deletion on module close since I could have obtained exclusive access previously?

Also of note, the process itself runs in a privileged user account with the following additional privileges enabled: SE_SECURITY_NAME, SE_BACKUP_NAME, SE_RESTORE_NAME, SE_TAKE_OWNERSHIP_NAME

How would one determine if you truly have exclusive access to a file when libraries are loaded?

Edit: Just to doubly check, I verified no other open handles besides loading the module via the SysInternals tool "handle".

Answer 1

your case can be more clear shown in next test

ULONG TestImageAccess(PCWSTR name, BOOL bImage, BOOL bRequestWriteAccess)
{
    SetLastError(0);

    HANDLE hFile = CreateFileW(name, GENERIC_READ, 0, 0, OPEN_EXISTING, 0, 0);

    if (hFile != INVALID_HANDLE_VALUE)
    {
        HANDLE hSection = CreateFileMappingW(hFile, 0, 
            bImage ? PAGE_READONLY|SEC_IMAGE : PAGE_READONLY|SEC_COMMIT, 0, 0, 0);

        CloseHandle(hFile);

        if (hSection)
        {
            hFile = CreateFileW(name, 
                bRequestWriteAccess ? GENERIC_WRITE : GENERIC_READ, 0, 0, OPEN_EXISTING, 0, 0);

            CloseHandle(hSection);

            if (hFile != INVALID_HANDLE_VALUE)
            {
                CloseHandle(hFile);
            }
        }
    }

    return GetLastError();
}

void TestImageAccess(PCWSTR filename)
{
    TestImageAccess(filename, TRUE, TRUE);  // ERROR_SHARING_VIOLATION
    TestImageAccess(filename, FALSE, TRUE); // NOERROR
    TestImageAccess(filename, TRUE, FALSE); // NOERROR
    TestImageAccess(filename, FALSE, FALSE);// NOERROR
}

we got ERROR_SHARING_VIOLATION when we request write access to file and file mapped as image

this case is described in Executable Images:

If the user wants write access to the file, make sure there is not a process mapping this file as an image

the call to MmFlushImageSection with MmFlushForWrite fail and you got STATUS_SHARING_VIOLATION - exactly at this line.

LoadLibraryEx with flag LOAD_LIBRARY_AS_DATAFILE create section on file with SEC_COMMIT when with 0 in flag - with SEC_IMAGE - so as image section

the privileges here absolute not related

How would one determine if you truly have exclusive access to a file when libraries are loaded?

simply open file with access which you need and do what you need. and yes, you can got error ERROR_USER_MAPPED_FILE when you try truncate mapped file. but here nothing can be done. example 2

ULONG TestImage2(PCWSTR name)
{
    HANDLE hFile = CreateFileW(name, GENERIC_READ, 0, 0, OPEN_EXISTING, 0, 0);

    if (hFile != INVALID_HANDLE_VALUE)
    {
        HANDLE hSection = CreateFileMappingW(hFile, 0, PAGE_READONLY|SEC_COMMIT, 0, 0, 0);

        CloseHandle(hFile);

        if (hSection)
        {
            PVOID pv = MapViewOfFile(hSection, FILE_MAP_READ, 0, 0, 0);

            CloseHandle(hSection);

            if (pv)
            {
                hFile = CreateFileW(name,GENERIC_WRITE|GENERIC_READ, 0, 0, OPEN_EXISTING, 0, 0);

                if (hFile != INVALID_HANDLE_VALUE)
                {
                    FILE_END_OF_FILE_INFO eof = {  };
                    // ERROR_USER_MAPPED_FILE will be here
                    SetFileInformationByHandle(hFile, FileEndOfFileInfo, &eof, sizeof(eof));
                    CloseHandle(hFile);
                }

                UnmapViewOfFile(pv);
            }
        }
    }

    return GetLastError();
}