Unable to provision VMs using SoftLayer API and Ansible due to 500 Server Errors

Question

We're using Ansible to provision a set of virtual machines. To do this, we're using the softlayer.py script found here, as part of ansible: https://github.com/ansible/ansible/blob/devel/contrib/inventory/softlayer.py

We are running: ansible-playbook manifest.yml -s -i softlayer.py

manifest.yml contains a bunch of ansible tasks that should provision our VMs. Here is an example with all sensitive information removed:

- gather_facts: false
hosts: localhost
name: Build Servers
sudo: false
tasks:
- local_action: 
cpus: 4
datacenter: sjc03
dedicated: false
disks:
- 25
domain: test.example.sample.net
hostname: machine-one
hourly: false
memory: 8192
module: sl_vm
nic_speed: 1000
os_code: UBUNTU_14_64
private: true
private_vlan: 1234567
ssh_keys: []
tags:
- tagone
- tagtwo
- tagthree
- tagfour
- test.example.sample.net
name: Build machine-one server

Below is the full error output, if it's of any use:

Inventory script (softlayer.py) had an execution error: Traceback (most recent call last):
File "/opt/test/job/softlayer.py", line 207, in <module>
SoftLayerInventory()
File "/opt/test/job/softlayer.py", line 84, in __init__
self.get_all_servers()
File "/opt/test/job/softlayer.py", line 204, in get_all_servers
self.get_virtual_servers()
File "/opt/test/job/softlayer.py", line 188, in get_virtual_servers
instances = vs.list_instances(mask=mask)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer-4.0.3-py2.7.egg/SoftLayer/managers/vs.py", line 141, in list_instances
return func(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer-4.0.3-py2.7.egg/SoftLayer/API.py", line 362, in call_handler
return self(name, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer-4.0.3-py2.7.egg/SoftLayer/API.py", line 330, in call
return self.client.call(self.name, name, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer-4.0.3-py2.7.egg/SoftLayer/API.py", line 226, in call
return self.transport(request)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer-4.0.3-py2.7.egg/SoftLayer/transports.py", line 162, in __call__
raise exceptions.TransportError(ex.response.status_code, str(ex))
SoftLayer.exceptions.TransportError: TransportError(500): 500 Server Error: Internal Server Error for url: https://api.softlayer.com/xmlrpc/v3.1/SoftLayer_Account

What could be causing this?

Answer 1

According to the exception, there exist a lot of 500 errors for many reasons:

• Timeouts
• Out of memory
• Invalid requests
• Bugs

I'm not sure how Ansible is handling and sending the parameters to SoftLayer API Python Client, but a quick view, Ansible is using SoftLayer-4.0.3 and SoftLayer API Python Client has released the v5.2.0, we recommend to use this, in order to get the last updates and avoid possible issues.

I did some tests using SoftLayer Python Command-line Interface sending the same parameters that you tried and it is working fine:

slcli vs create --hostname=machine-one --domain=stest.example.sample.net --cpu 4 --memory 8192 -o UBUNTU_14_64 --datacenter=sjc03 --billing=monthly --tag tagone,tagtwo,tagthree,tagfour,test.example.sample.net --public --network 1000 --private

I recommend to submit an issue in the below link, in order that Ansible can verify and fix it:

https://github.com/ansible/ansible/issues

Answer 2

This seems to be specific to calling getVirtualGuests on https://api.softlayer.com/xmlrpc/v3.1/SoftLayer_Account with a particular set of credentials / on a particular account and with a particular mask.

<?xml version='1.0'?>
<methodCall>
<methodName>getVirtualGuests</methodName>
<params>
<param>
<value><struct>
<member>
<name>headers</name>
<value><struct>
<member>
<name>SoftLayer_ObjectMask</name>
<value><struct>
<member>
<name>mask</name>
<value><string>mask[id,globalIdentifier,hostname,domain,fullyQualifiedDomainName,primaryBackendIpAddress,primaryIpAddress,datacenter,tagReferences.tag.name,lastKnownPowerState.name,powerState,maxCpu,maxMemory,activeTransaction.transactionStatus[friendlyName,name],status]</string></value>
</member>
</struct></value>
</member>
<member>
<name>SoftLayer_AccountObjectFilter</name>
<value><struct>
</struct></value>
</member>
<member>
<name>authenticate</name>
<value><struct>
<member>
<name>apiKey</name>
<value><string>xxxxxxx</string></value>
</member>
<member>
<name>username</name>
<value><string>example</string></value>
</member>
</struct></value>
</member>
</struct></value>
</member>
</struct></value>
</param>
</params>
</methodCall>

POSTing that XML to the xmlrpc endpoint above I hit the 500 error with the particular credentials mentioned in the original question. If I use a different set of (valid) credentials, I'm returned my list of servers as I'd expect. If I supply invalid credentials then I receive "Invalid API token." back as I'd expect.

The surprising thing in the 500 case is that it does seem to have run the query (as SoftLayer-Total-Items: 70 is returned in the header), but the HTTP code is 500 and the body of the response is empty.

If I inspect the http requests made by slcli -vvv vs list (which succeeds) I see that it sends a different mask to the same method, so perhaps the mask in the ansible softlayer.py is relevant.