Deploy Service Fabric application fails - The EntryPoint Xyz.exe is not found

Question

Using Visual Studio Team Services to implement our CI/CD pipeline using build and release definitions. One of our projects builds properly but fails when deploying to a remote Service Fabric cluster with the following error:

2017-01-24T21:57:57.7395575Z ##[error]The EntryPoint mycompany.VariableLibrary.QuestionnaireDefinitionEventAdapter.exe is not found.
2017-01-24T21:57:57.7395575Z ##[error]FileName: C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\TestApplicationPackage_356048141954\10xqzk0s.1m1\applicationpackage\mycompany.VariableLibrary.QuestionnaireDefinitionEventAdapterPkg\ServiceManifest.xml

Looking at the application package, we see that mycompany.VariableLibrary.QuestionnaireDefinitionEventAdapter.exe is properly generated and not missing.

This same application package (downloaded from the Artifacts tab in VSTS) can be deployed on a local cluster or on our dev cluster using the Deploy-FabricApplication.ps1 script generated by Visual Studio Enterprise.

Looking at the task source code here, we figure that the task fails on line 146 after calling Test-ServiceFabricApplicationPackage. However, running manually this command on the downloaded application package seems to return true.

There doesn't seem to be any way to make logs more verbose. We have other projects for which the CI/CD pipeline works correctly.

Edit

Build logs are over 2MB, I can't post them here.

Release logs: http://pastebin.com/BgVJX4mw

ApplicationManifest.xml:

<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="mycompany.VariableLibrary.App" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Parameters>
    <Parameter Name="QuestionnaireDefinitionEventAdapter_InstanceCount" DefaultValue="-1" />
    <Parameter Name="FillOutEventAdapter_InstanceCount" DefaultValue="-1" />
    <Parameter Name="RestAdapter_InstanceCount" DefaultValue="-1" />
    <Parameter Name="VariableLibraryService_MinReplicaSetSize" DefaultValue="3" />
    <Parameter Name="VariableLibraryService_PartitionCount" DefaultValue="1" />
    <Parameter Name="VariableLibraryService_TargetReplicaSetSize" DefaultValue="3" />
    <Parameter Name="VariableLibraryServiceUri" DefaultValue="" />
    <Parameter Name="EnvironmentName" DefaultValue="" />
    <Parameter Name="InstrumentationKey" DefaultValue="" />
    <Parameter Name="DiagEnableErrorPage" DefaultValue="" />
    <Parameter Name="JwtDomain" DefaultValue="" />
    <Parameter Name="JwtAudience" DefaultValue="" />
    <Parameter Name="JwtClientSecret" DefaultValue="" />
    <Parameter Name="DomainEventBusServiceBusTopicPath" DefaultValue="DomainEvents" />
    <Parameter Name="DomainEventBusServiceBusConnectionString" DefaultValue="" />
    <Parameter Name="QuestionnaireDefinitionSurveyServiceUri" DefaultValue="" />
    <Parameter Name="QuestionnaireDefinitionQuestionnaireServiceUri" DefaultValue="" />
    <Parameter Name="QuestionnaireDefinitionSampleServiceUri" DefaultValue="" />
    <Parameter Name="FillOutServiceUri" DefaultValue="" />
  </Parameters>
  <ServiceManifestImport>
    <ServiceManifestRef ServiceManifestName="mycompany.VariableLibrary.QuestionnaireDefinitionEventAdapterPkg" ServiceManifestVersion="1.0.0" />
    <ConfigOverrides>
      <ConfigOverride Name="Config">
        <Settings>
          <Section Name="AppSettings">
            <Parameter Name="EnvironmentName" Value="[EnvironmentName]" />
            <Parameter Name="InstrumentationKey" Value="[InstrumentationKey]" />
            <Parameter Name="DomainEventBusServiceBusConnectionString" Value="[DomainEventBusServiceBusConnectionString]" />
            <Parameter Name="DomainEventBusServiceBusTopicPath" Value="[DomainEventBusServiceBusTopicPath]" />
            <Parameter Name="VariableLibraryServiceUri" Value="[VariableLibraryServiceUri]" />
          </Section>
        </Settings>
      </ConfigOverride>
    </ConfigOverrides>
  </ServiceManifestImport>
  <ServiceManifestImport>
    <ServiceManifestRef ServiceManifestName="mycompany.VariableLibrary.FillOutEventAdapterPkg" ServiceManifestVersion="1.0.0" />
    <ConfigOverrides>
      <ConfigOverride Name="Config">
        <Settings>
          <Section Name="AppSettings">
            <Parameter Name="EnvironmentName" Value="[EnvironmentName]" />
            <Parameter Name="InstrumentationKey" Value="[InstrumentationKey]" />
            <Parameter Name="DomainEventBusServiceBusConnectionString" Value="[DomainEventBusServiceBusConnectionString]" />
            <Parameter Name="DomainEventBusServiceBusTopicPath" Value="[DomainEventBusServiceBusTopicPath]" />
            <Parameter Name="VariableLibraryServiceUri" Value="[VariableLibraryServiceUri]" />
          </Section>
        </Settings>
      </ConfigOverride>
    </ConfigOverrides>
  </ServiceManifestImport>
  <ServiceManifestImport>
    <ServiceManifestRef ServiceManifestName="mycompany.VariableLibrary.RestAdapterPkg" ServiceManifestVersion="1.0.0" />
    <ConfigOverrides>
      <ConfigOverride Name="Config">
        <Settings>
          <Section Name="AppSettings">
            <Parameter Name="EnvironmentName" Value="[EnvironmentName]" />
            <Parameter Name="InstrumentationKey" Value="[InstrumentationKey]" />
            <Parameter Name="DiagEnableErrorPage" Value="[DiagEnableErrorPage]" />
            <Parameter Name="JwtDomain" Value="[JwtDomain]" />
            <Parameter Name="JwtAudience" Value="[JwtAudience]" />
            <Parameter Name="JwtClientSecret" Value="[JwtClientSecret]" />
            <Parameter Name="VariableLibraryServiceUri" Value="[VariableLibraryServiceUri]" />
          </Section>
        </Settings>
      </ConfigOverride>
    </ConfigOverrides>
  </ServiceManifestImport>
  <ServiceManifestImport>
    <ServiceManifestRef ServiceManifestName="mycompany.VariableLibrary.VariableLibraryServicePkg" ServiceManifestVersion="1.0.0" />
    <ConfigOverrides>
      <ConfigOverride Name="Config">
        <Settings>
          <Section Name="AppSettings">
            <Parameter Name="EnvironmentName" Value="[EnvironmentName]" />
            <Parameter Name="InstrumentationKey" Value="[InstrumentationKey]" />
            <Parameter Name="DomainEventBusServiceBusConnectionString" Value="[DomainEventBusServiceBusConnectionString]" />
            <Parameter Name="DomainEventBusServiceBusTopicPath" Value="[DomainEventBusServiceBusTopicPath]" />
            <Parameter Name="QuestionnaireDefinitionSurveyServiceUri" Value="[QuestionnaireDefinitionSurveyServiceUri]" />
            <Parameter Name="QuestionnaireDefinitionQuestionnaireServiceUri" Value="[QuestionnaireDefinitionQuestionnaireServiceUri]" />
            <Parameter Name="QuestionnaireDefinitionSampleServiceUri" Value="[QuestionnaireDefinitionSampleServiceUri]" />
            <Parameter Name="FillOutServiceUri" Value="[FillOutServiceUri]" />
          </Section>
        </Settings>
      </ConfigOverride>
    </ConfigOverrides>
  </ServiceManifestImport>
  <DefaultServices>
    <Service Name="QuestionnaireDefinitionEventAdapter">
      <StatelessService ServiceTypeName="QuestionnaireDefinitionEventAdapter" InstanceCount="[QuestionnaireDefinitionEventAdapter_InstanceCount]">
        <SingletonPartition />
      </StatelessService>
    </Service>
    <Service Name="FillOutEventAdapter">
      <StatelessService ServiceTypeName="FillOutEventAdapter" InstanceCount="[FillOutEventAdapter_InstanceCount]">
        <SingletonPartition />
      </StatelessService>
    </Service>
    <Service Name="RestAdapter">
      <StatelessService ServiceTypeName="RestAdapter" InstanceCount="[RestAdapter_InstanceCount]">
        <SingletonPartition />
      </StatelessService>
    </Service>
    <Service Name="VariableLibraryService">
      <StatefulService ServiceTypeName="VariableLibraryService" TargetReplicaSetSize="[VariableLibraryService_TargetReplicaSetSize]" MinReplicaSetSize="[VariableLibraryService_MinReplicaSetSize]">
        <UniformInt64Partition PartitionCount="[VariableLibraryService_PartitionCount]" LowKey="-9223372036854775808" HighKey="9223372036854775807" />
      </StatefulService>
    </Service>
  </DefaultServices>
</ApplicationManifest>

ServiceManifest.xml:

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="mycompany.VariableLibrary.QuestionnaireDefinitionEventAdapterPkg"
                 Version="1.0.0"
                 xmlns="http://schemas.microsoft.com/2011/01/fabric"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
    <StatelessServiceType ServiceTypeName="QuestionnaireDefinitionEventAdapter" />
  </ServiceTypes>

  <CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <ExeHost>
        <Program>mycompany.VariableLibrary.QuestionnaireDefinitionEventAdapter.exe</Program>
      </ExeHost>
    </EntryPoint>
  </CodePackage>

  <ConfigPackage Name="Config" Version="1.0.0" />

  <Resources>
    <Endpoints>
      <Endpoint Name="ServiceEndpoint" />
    </Endpoints>
  </Resources>
</ServiceManifest>

Any ideas?

Answer 1

Since you're saying that this works locally but not in a real azure cluster, I'm wondering if you're somehow hitting a path too long error somewhere which is either causing the file to not get copied to the image store correctly, or which is causing other problems. Various names within the deployment can also contribute to long paths (>260 characters), which unfortunately fail in Windows. To try to see if this is the cause, try renaming the executable as something like A.exe and try deployment again. You may also need to shorten the application and service type names.

The error normally means that one of your service packages references an executable (in this case it looks like QuestionnaireDefinitionEventAdapter.exe), but the actual package that you're producing and trying to register with the cluster is missing that executable.  You can also check the actual package that is getting produced as a part of your build and make sure that it is present in the cluster. If you want to explore what is actually getting pushed into the image store just to check, you can use Get-ServiceFabricImageStoreContent.