Xcode Server Bot integration stalls on xcrun xcodebuild -exportArchive "Exporting installable product"

Question

Xcode Server Bot integration stalls at stage six:

"Exporting installable product"

The Raw Build Log shows as final entries:

$ /usr/bin/xcrun xcodebuild -exportArchive -archivePath /Library/Developer/XcodeServer/Integrations/Integration-7f80e03e9471a040c753957c2600b0a6/BetUP.xcarchive -exportPath /Library/Developer/XcodeServer/Integrations/Integration-7f80e03e9471a040c753957c2600b0a6/ExportedProduct -exportOptionsPlist /Library/Developer/XcodeServer/Integrations/Integration-7f80e03e9471a040c753957c2600b0a6/ExportOptions.plist -IDEPostProgressNotifications=YES -DVTAllowServerCertificates=YES -DVTSigningCertificateSourceLogLevel=3 -DVTSigningCertificateManagerLogLevel=3 -DTDKProvisioningProfileExtraSearchPaths=/Library/Developer/XcodeServer/ProvisioningProfiles -configuration Debug

2016-09-05 13:50:33.659 xcodebuild[61877:3819468] [MT] IDEDistribution: -[IDEDistributionLogging _createLoggingBundleAtPath:]: Created bundle at path '/var/folders/x2/t8b7x9gs2x32b686xfzq77lw000087/T/MyApp_2016-09-05_13-50-33.658.xcdistributionlogs'.
2016-09-05 13:50:33.704 xcodebuild[61877:3819468] [MT] DeveloperPortal: Using pre-existing current store at URL (file:///var/_xcsbuildd/Library/Developer/Xcode/DeveloperPortal%207.3.1.db).
2016-09-05 13:50:33.712 xcodebuild[61877:3819482]  DVTSigningCertificateSource: Searching for signing identities in default keychain search list
2016-09-05 13:50:33.738 xcodebuild[61877:3819482]  DVTSigningCertificateSource: Searching for signing identities in default keychain search list
2016-09-05 13:50:33.757 xcodebuild[61877:3819482]  DVTSigningCertificateManager: __58-[DVTSigningCertificateManager forceFullSyncWithCallback:]_block_invoke (0x7fbc650a88a0): Signing certificates = {(

Eventually the integration hits the timeout limit and fails with an error:

Assertion: Running task was terminated because it produced no activity for more than 1200 seconds (use sudo xcrun xcscontrol --configure-integration-timeout to increase this timeout)

There's no point increasing the timeout beyond 1200 seconds - 20 minutes - as it will never completed. Earlier stages take three minutes, so seventeen minutes is plenty time to export.

Other posts such as these did not provide solutions:

Xcode Server Bot integration fails to export archive with Assertion fail: exportArchive: ipatool failed with an exception

Xcode Server Bot integration fails to export archive: Error Domain=IDEDistributionErrorDomain Code=1 "The operation couldn’t be completed."

Copying the private key - by dragging - from the login user to the system keychain also did not resolve:

Xcode Bots: Common Problems And Workarounds

Clutching at straws, I have tried pretty much everything else including --sync-portal described in the tutorial Xcode Server Hacks: 2. Taking control with xcscontrol.

Not to mention obviously restarting - not resetting! - using this command:

sudo xcrun xcscontrol --restart

Having tried almost everything I can think of, how can this problem be resolved?

Answer 1

Finally, the only thing left to try was a reset of Xcode server:

As highlighted in Xcode Server Hacks: 1. Under the Hood of Xcode Server Troubleshooting:

"When you hack too much and Xcode Server goes crazy, you need a way to reset all of its state and start over. When I was originally developing Buildasaur, I would reset Xcode Server several times a day. There is one command which stops all the running daemons of Xcode Server and deletes all the contents of /Library/Developer/XcodeServer. Be careful, all your bots and integration data will be deleted as well. During debugging it’s an indispensable tool, but be careful in production to not accidentally delete all your Bots and Integration assets."

The command blows away all the Bots and configuration:

sudo xcrun xcscontrol --reset.

Having done that, it was necessary to start afresh, configuring the Server applications's Xcode service to add the necessary developer teams.

That's where things became interesting.

Having entered the Apple ID, and selected the associated teams; an error was shown:

"You must be a team agent or an admin to add this server to a developer team"

Setting the Apple ID as admin of the teams using Apple's developer portal fixed that issue.

Having added back in a test Bot, it worked!

After going through all this pain my recommendations would be as follows:

-- Delete the teams from Server's Xcode service. Then re-add. This will expose any issues with admin privileges.

or

-- Failing that, perform a reset and reconfigure the Server's Xcode service. Then re-add the Bots again.

The second option above is a pain but it's a lot quicker than spending hours or even days trying to resolve. I was able to reset, reconfigure, and re-add about ten bots - by duplicating - in just over an hour.