OS X 3.2.1 CI "pending integration"

Question

Over the past few days, I've been working on getting CI working with an external mac mini running OS X Server. However, I have been having many problems with OS X Server 3.2.1 and XCode 6.1b3.

It looks like Apple fixed an issue in Xcode 6.1b3 which didn't put the correct provisioning profiles into Portal.keychain. However, my integrations aren't even running now.

After running a clean OS X build, XCode server won't integrate. I succesfully connected to the server and created a bot. If I visit "SERVER.local" on my development machine, I see the bot that I created.

Everything is set up properly (including the integrate immediately checkbox), however my integrations sit in the "pending" state. I checked the system.log, and nothing seems to be happening.

This could be completely unrelated, but every time I click on a pending integration, I receive this error in system.log:

NSFileCoordinator only handles URLs that use the file: scheme. This one does not:
x-code-xcsbot://XXX

I'm not sure if this is a new problem introduced in OS X server 3.2.1, or if it's just a set up issue. Apparently no one else has had this issue, couldn't find anything on Google/ SO.

Answer 1

This is still happening, but if you just want to nudge the server to wake up and run and aren't prepared to delete your x-code server configuration (provisioningProfiles, credentials and bots are deleted (as I recall)), simply run this terminal command

sudo -u _xcsbuildd /Applications/Xcode.app/Contents/Developer/usr/bin/xcsbuildd

Note that if you have multiple revs of Xcode in your Applications folder you may have named them differently, so the command may be slightly different. In my case, I've got a file named 'Xcode 6.1.1'. So the command would be

sudo -u _xcsbuildd /Applications/Xcode\ 6.1.1.app/Contents/Developer/usr/bin/xcsbuildd

Answer 2

This appears to happen when running XCode beta builds in OS X Server.

Note that this command will ❗️delete all your bots❗️
Run sudo xcrun xcscontrol --reset to reset.

https://devforums.apple.com/message/1051403#1051403

Answer 3

Here's a solution that may fix the problem without requiring to reset Xcode Server.

What's the problem?

First, check whether this answer applies by inspecting the xcsnginx.log log file:

sudo tail /Library/Developer/XcodeServer/Logs/xcsnginx.log

Search for the following line at the end of the log:

nginx: [emerg] SSL_CTX_use_PrivateKey_file("/Library/Developer/XcodeServer/Certificates/xcsnginx.key") failed (SSL: error:0906A068:PEM routines:PEM_do_header:bad password read error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib)

I you don't see that entry in the log, I'm afraid this answer won't help you. If you do see that entry, you may continue.

Why does it happen?

Xcode Server internally runs an Nginx web server (on port 20543) named xcsnginx that acts as a proxy between some services. This server uses a TLS/SSL certificate to ensure communications remain secure. The involved files are the following:

• xcsnginx.crt: contains the PEM certificate.
• xcsnginx.key: contains the private key for the certificate.
• xcsnginx.pass: contains the passphrase for the private key.

As far as I understand, the private key is stored unencrypted, which means the xcsnginx.pass should be empty (and seems to be reset every time Xcode Server starts).

However for some reason, as some point, the private key in xcsnginx.key was exported as an encrypted key. I've no idea how and why this could happen but it did happen on my server so I'll assume it may happen on your server too. The consequence is that xcsnginx cannot load the certificate and fails to launch.

You can verify that xcsnginx is not running by executing:

pgrep xcsnginx || echo "Not running"

How to fix it?

Rather than resetting Xcode Server from scratch, we can:

• export the identity again from the xcsnginx.keychain keychain or
• restore the previous certificate and key or
• create a new certificate and key for xcsnginx.

So let's have a look at each option.

Option 1

Copies of the certificate and private key are stored in the xcsnginx.keychain keychain located in /Library/Developer/XcodeServer/Keychains. This keychain is protected by a passphrase stored in a file named XCSNginxKeychainSharedSecret in the /Library/Developer/XcodeServer/SharedSecrets folder.

If you're familiar with OS X keychains, you may retrieve the certificate and the key from the keychain.

However manipulating keychains using the command-line is a real nightmare so I'll let this as an exercise for the reader (or an editor).

Option 2

The /Library/Developer/XcodeServer/Certificates folder may contains a backup of your certificate and key. Let's find out:

sudo find /Library/Developer/XcodeServer/Certificates -name "*.original"

If you're lucky, you should get the following result:

/Library/Developer/XcodeServer/Certificates/xcsnginx.crt.original
/Library/Developer/XcodeServer/Certificates/xcsnginx.key.original
/Library/Developer/XcodeServer/Certificates/xcsnginx.pass.original

Which mean you can restore the original files:

sudo cp /Library/Developer/XcodeServer/Certificates/xcsnginx.crt.original /Library/Developer/XcodeServer/Certificates/xcsnginx.crt
sudo cp /Library/Developer/XcodeServer/Certificates/xcsnginx.key.original /Library/Developer/XcodeServer/Certificates/xcsnginx.key
sudo cp /Library/Developer/XcodeServer/Certificates/xcsnginx.pass.original /Library/Developer/XcodeServer/Certificates/xcsnginx.pass

Option 3

If you cannot restore the previous certificate and keychains, you may decide to just generate new ones like this:

sudo openssl req -new -x509 -newkey rsa:2048 -nodes -out /Library/Developer/XcodeServer/Certificates/xcsnginx.crt -keyout /Library/Developer/XcodeServer/Certificates/xcsnginx.key -subj /CN=your-server.example.com -days 1000 -batch

where your-server.example.com is replaced with the DNS address of your server. Ideally the certificate should be issued by the Xcode Server Root Certificate Authority but using a single-signed certificate doesn't seem to be a problem (as far as I now / for the moment / your mileage may vary).

Finally

Now we just have to wait until the system starts xcsnginx again. That should happen automatically after a minute or less. You can verify that xcsnginx did start with:

pgrep xcsnginx || echo "Not running"

Answer 4

You can use restart instead of reset from the command line to restart the server. This will terminate the bot you are running that has frozen up.

sudo xcrun xcscontrol --restart

Answer 5

I've run into this several times. I've done the xcrun xcscontrol --reset and it does work, but I'm getting tired of that. As my bots get more complicated with pre/post integration triggers I don't want to set them up again.

What worked for me this time was just going through all mostly recent integrations and deleting any canceled ones. Cancelling a bot run can leave it in an odd state.

After I deleted a bunch of integrations I restarted the machine the server is running on and my integrations start working again!

Hope that helps someone.