Automated Deployment to an F5 Load Balanced Environment

Question

We are currently using TeamCity for CI builds and we are trying to set up automated deployments as well.

The project I'm currently trying to deploy is a Windows service that sits under an F5 load balancer. In the future we would also like to automate the deployment of our IIS websites which also sit under the F5.

From TeamCity we can execute PowerShell scripts to unistall the windows service on the desired server, push our files to it, then reinstall the service.

However, I'm having trouble figuring out how to deal with the load balancer. We would want to disable 1 node at a time, watch for all the connections to drop, then deploy our code and bring the node back up.

This seems like it would be a very common issue, but I'm finding surprisingly little information about how to do it.

Thanks!

Answered

Thanks Jonathon Rossi for the iControl Powershell cmdlets!

For other users' sakes, here is a sample of shutting down, monitoring for connections to drop, pushing code, and then turning back on the F5 load balancer through a powershell script

For these scripts to work you will first have to install the F5 iControl cmdlets from the links provided in the Answer below

#PULL IN OUR F5 UTILITY FUNCTIONS
. .\F5Functions.ps1


#DEFINE LOGIC TO DEPLOY CODE TO A NODE THAT HAS ALREADY BEEN REMOVED FROM THE LOAD BALANCER
function Deploy(
    [F5Node]$Node
)
{
    Write-Host "Deploying To: "$Node.Name
    #TODO: Remotely shut down services, push code, start back up services
}


#DEFINE NODES
$nodes = @()
$nodes += New-Object F5Node -ArgumentList @("TestNode1", "1.1.1.1")
$nodes += New-Object F5Node -ArgumentList @("TestNode2", "1.1.1.2")

#DEPLOY
DeployToNodes -Nodes $nodes -F5Host $F5Host -F5UserName $F5UserName -F5Password $F5Password

And here is the reusable F5Functions script

#Load the F5 powershell iControl snapin
Add-PSSnapin iControlSnapin;

Write-Host "Imported F5 function!!!"

Add-Type @'
    public class F5Node
    {
        public F5Node(string name, string address){
            Address = address;
            Name = name;
        }
        public string Address {get;set;}
        public string Name {get;set;}
        public string QualifiedName {get{return "/Common/" + Name;}}
    }
'@

function DeployToNodes(
    [string]$F5Host = $(throw "Missing Required Parameter"),
    [string]$F5UserName = $(throw "Missing Required Parameter"),
    [string]$F5Password = $(throw "Missing Required Parameter"),
    [F5Node[]]$Nodes = $(throw "Missing Required Parameter"),    
    [int]$MaxWaitTime = 300 #seconds... defaults to 5 minutes
){
    Authenticate -F5Host $F5Host -F5UserName $F5UserName -F5Password $F5Password

    foreach($node in $Nodes){
        DisableNode -Node $node

        WaitForConnectionsToDrop -Node $node -MaxWaitTime $MaxWaitTime

        #Assume the Script that included this script defined a Deploy Method with a Node param
        Deploy -Node $node    

        EnableNode -Node $node
    }
}

function Authenticate(
    [string]$F5Host = $(throw "Missing Required Parameter"),
    [string]$F5UserName = $(throw "Missing Required Parameter"),
    [string]$F5Password = $(throw "Missing Required Parameter")
)
{
    Write-Host "Authenticating to F5..."
    Initialize-F5.iControl -HostName $F5Host -Username $F5UserName -Password $F5Password
    Write-Host "Authentication Success!!!"
}

function ParseStatistic(
        [iControl.CommonStatistic[]]$StatsCollection = $(throw "Missing Required Parameter"),
        [string]$StatName = $(throw "Missing Required Parameter")
    )
{
    for($i=0; $i -lt $StatsCollection.Count; $i++){   
        if($StatsCollection[$i].type.ToString() -eq $StatName){
            return $StatsCollection[$i].value.low  
            break
        }                      
    }
}

function GetStats(
        [F5Node]$Node = $(throw "Missing Required Parameter")
    )
{
    $arr = @($Node.QualifiedName)
    $nodeStats = (Get-F5.iControl).LocalLBNodeAddressV2.get_statistics($arr)
    return $nodeStats.statistics.statistics

    #foreach($memberStats in $poolStats.statistics){
    #    if($memberStats.member.address.ToString() -eq $Node -and $memberStats.member.port -eq $Port){
    #        return $memberStats.statistics
    #    }  
    #}
}

function GetStatistic(
        [F5Node]$Node = $(throw "Missing Required Parameter"),
        [string]$StatName = $(throw "Missing Required Parameter")
    )
{
    $stats = GetStats -Node $Node
    $stat = ParseStatistic -StatsCollection $stats -StatName $StatName

    return $stat
}

function DisableNode(
    [F5Node]$Node = $(throw "Missing Required Parameter")
)
{    
    Disable-F5.LTMNodeAddress -Node $Node.Address
    Write-Host "Disabled Node '$Node'"
}

function EnableNode(
    [F5Node]$Node = $(throw "Missing Required Parameter")
)
{
    Enable-F5.LTMNodeAddress -Node $Node.Address
    Write-Host "Enabled Node '$Node'"
}

function WaitForConnectionsToDrop(
    [F5Node]$Node = $(throw "Missing Required Parameter"),
    [int]$MaxWaitTime = 300
)
{
    $connections = GetCurrentConnections -Node $Node

    $elapsed = [System.Diagnostics.Stopwatch]::StartNew();
    while($connections -gt 0 -and $elapsed.ElapsedMilliseconds -lt ($MaxWaitTime * 1000)){        

        Start-Sleep -Seconds 10

        $connections = GetCurrentConnections -Node $Node
    }
}

function GetCurrentConnections(
    [F5Node]$Node = $(throw "Missing Required Parameter")
)
{
    $connections = GetStatistic -Node $Node -StatName "STATISTIC_SERVER_SIDE_CURRENT_CONNECTIONS"
    $name = $Node.Name + ":" + $Node.Address
    Write-Host "$connections connections remaining on '$name'"
    return $connections
}

any final solution with full source code sample working about it ? — Kiquenet
@Kiquenet - The above code snippets under the "Answered" heading are what we used to deploy code to production about 5 - 10 times daily. The second code block is the module that is loaded by the first code block, which is an example of how to use the module. The only part you need to fill in is what you actually do to deploy code while the F5 node is down (move files, decompress, stop services, restart services, etc.) Please excuse the poor naming conventions. I was new to powershell when I wrote this. — Michael
Actually as I'm re-reading my old post, I'm seeing that it's much less mature than the final product. 1. I moved the F5 funcitons script to a powershell module installed on the build server, and loaded it with Load-Module 2. The DeployToNodes function should take a ScriptBlock and and array of arguments for the ScriptBlock instead of assuming that the Deploy function is declared elsewhere 3. functions should follow the Verb-Noun naming convention 4. I put more logic around taking down nodes, so it would take down half at a time, allowing sql migrations to take place in the middle — Michael
Have you managed to automate the installation of the iControl msi package itself? — Stefano Ricciardi

Jonathon Rossi Jonathon Rossi · Accepted Answer · 2013-06-22T01:52:30

I haven't used it, but have you looked at the F5 iControl web service API and the F5 iControl PowerShell cmdlets provided by F5. The PowerShell cmdlets have been around since 2007 and can be downloaded from F5 DevCentral.

It looks like there are Enable-Member and Disable-Member cmdlets that you'll be able to use.

Automated Deployment to an F5 Load Balanced Environment

1 Answers