Try with this example:
https://api.softlayer.com/rest/v3.1/SoftLayer_Account/getVirtualGuests?objectMask=mask[id,operatingSystem[id,softwareDescription[id,productItems]]]&objectFilter={"virtualGuests":{"operatingSystem":{"softwareDescription":{"productItems":{"keyName":{"operation":"OS_WINDOWS_2012_R2_FULL_STD_64_BIT"}}}}}}
OS_WINDOWS_2012_R2_FULL_STD_64_BIT is the operationSystem keyName.
For obtaining the keyName you should use:
https://api.softlayer.com/rest/v3.1/SoftLayer_Product_Package/46/getItemPrices?objectMask=mask[id,item[categories[categoryCode],keyName]]&objectFilter={"itemPrices":{"categories":{"categoryCode":{"operation":"os"}}}}
Choose the operationSystem keyName which wants to filter
I hope it helps