How can I pass the 'Session' argument using InstallShield to my WiX custom action?

Question

I have simple WiX (Microsoft.Deployment.WindowsInstaller) custom action:

    [CustomAction]
    public static ActionResult TestDtf(Session session)
    {
        MessageBox.Show("Test");

        ActionResult result = ActionResult.Success;
        return result;

    }

I need to have a deferred / system context custom action I created using InstallShield call this, so how do I set up the Method Signature arguments so that it sends the Session? Is 'Session' basically the Msi handle? I have tried using the 'MsiHandle' value, however this results in error:

InstallShield: Deferred action requested property MsiHiddenProperties not provided by CustomActionData
InstallShield: Loading assembly Test.Installation.CustomActions from resource 4098
InstallShield: Loading Assembly Microsoft.Deployment.WindowsInstaller
InstallShield: Unexpected parameter type Microsoft.Deployment.WindowsInstaller.Session encountered; passing string instead
InstallShield: Calling method with parameters [(System.String)294]
InstallShield: Exception: System.ArgumentException: Object of type 'System.String' cannot be converted to type 'Microsoft.Deployment.WindowsInstaller.Session'.
   at System.RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
   at System.RuntimeType.CheckValue(Object value, Binder binder, CultureInfo culture, BindingFlags invokeAttr)
   at System.Reflection.MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
   at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at InstallShield.ClrHelper.CustomActionHelper.CallMethod(EntryPointInfo info)

Answer 1

You don't as you don't need to. You are creating the wrong kind of custom action in InstallShield. Pretend your DLL isn't .NET because as far as DTF is concerned, it's not. It has been encapsulated as a native DLL.

Answer 2

The answer for this problem for me was to change the type of DLL CA I was adding to an MSI DLL. Then it only asks for the function name and not parameters and return values. I could not find this readily explained online unfortunately and it took a bit of guessing. Here is what your method signature should look like.

[CustomAction]
  public static ActionResult VerifyCA( Session session )
  {
     //Record record = new Record( 2 );
     //record[0] = "[1]";
     //record[1] = "Testing Wix message";
     //session.Message( InstallMessage.Error, record );

     return ActionResult.Failure;
  }

After picking the right CA type, it works like a charm for me from Installshield 2009. Hope this helps someone.

Answer 3

My answer is w.r.t. Installshiled 2016 and in all likelyhood it must work the same way in earlier installshield versions as well but I've not verified the same.

Yes. The in-built MsiHandle varible is the session for the installation in progress but it requires one extra line of code to get that on C# side. I was able to achieve it in a .NET assembly which I invoke from my managed custom action. The code is as below:

//import this namespace at the top of your *.cs file
using Microsoft.Deployment.WindowsInstaller;

public static int MakeChangesInCurrentInstallSession(IntPtr hMsi)
{
    Session session = Session.FromHandle(hMsi, false);

    view = session.Database.OpenView("SELECT * FROM ComboBox");
    view.Execute();
    //do other things whatever you want to do in the installer session
    //Record record = session.Database.CreateRecord(4);
    //view.Modify(....);
    //.....
    //return success if everything went well
    return (int)ActionResult.Success;
}

If you make the signature of your method like below and tried to invoke it from your managed custom action:

public static int MakeChangesInCurrentInstallSession(Session hMsi)

Then, you face the casting error as below:

InstallShield: Unexpected parameter type Microsoft.Deployment.WindowsInstaller.Session encountered; passing string instead

Note: For the above code to work you will have add reference to Microsoft.Deployment.WindowsInstaller.dll in your C# project from extensions tab in the add reference window. Also, since this assembly is not a part of .NET framework so you must add this assembly as a dependency in your managed custom action of Installshield execute sequence as detailed here.