How to call DLL functions from InstallScript called by MSI custom action

Question

I have a DLL named FSSetup.dll that I want to call from InstallScript code. And I want to call the InstallScript code from an MSI custom action. But there seems to be a problem in that I don't know where to find FSSetup.dll. The InstallScript function is starting and the whole install aborts when it gets to the line where it calls the DLL function. The MSI log reports:

MSI (c) (1C:3C) [09:39:52:261]: Invoking remote custom action. DLL: C:\Users\bmarty\AppData\Local\Temp\MSIBEAE.tmp, Entrypoint: f1
Action ended 9:39:55: BrowseFSKeyFile. Return value 3.
Info 2896. Executing action BrowseFSKeyFile failed.
Action ended 9:39:55: InstallWelcome. Return value 3.

The InstallScript code looks like this:

export prototype BrowseFSKeyFile(HWND);
prototype FSSetup.FSBrowseFile( BYVAL STRING, BYVAL STRING, BYVAL STRING, BYREF STRING );   

function BrowseFSKeyFile(hMSI)
NUMBER  nKeyFile;
STRING  svKeyFile, szKPath, szKFile, szTemp, szTemp1;
begin
    MessageBox("Hi", INFORMATION);
    svKeyFile = "";
    FSBrowseFile( "*.key", "key", "Please select a KEY File...", svKeyFile );
    MessageBox("Bye", INFORMATION);
    ...
end;

I see the "Hi" message, but the install aborts before "Bye". I suspect it's because FSSetup.dll doesn't exist in my %temp% directory where MSIBEAE.tmp resides, which presumably represents the extracted name of the ISSetup.dll that was generated by the InstallScript compile.

Edit: The MSI debugger is showing a different value for SUPPORTDIR than the InstallScript is seeing. FSSetup.dll exists in the SUPPORTDIR provided by MSI, but not in the one seen from InstallScript.

Michael Urman Michael Urman · Accepted Answer · 2015-09-11T11:46:57

You are missing a call to UseDll (and UnUseDll). In order to call UseDll, you will need to know the location of the DLL in question. If you have added it to the support files of the MSI project, you will need to retrieve the Windows Installer Property SUPPORTDIR to find this location - that's where Christopher Painter's answer comes into play - and then call e.g. UseDll(szSupportDir ^ "FSSetup.dll") before calling FSBrowseFile.

How to call DLL functions from InstallScript called by MSI custom action

2 Answers