Draining a SMA Runbook Worker from its Jobs

There are times (say when installing Windows Updates) where it would be beneficial to reboot a SMA Worker. Within the SMA gui, there is no way for you to easily tell if the server is currently taking on new jobs, if it is currently running any jobs, or if it is taking on new jobs.

You will want to come up with some sort of process so you can keep your SMA Workers up to date. Here is what we like to do:

1. Remove the SMA Runbook Worker from the deployment.
2. Check for any running runbooks or wait the configured drain time (default 900) seconds
3. Reboot the SMA Runbook Worker
4. Add the SMA Runbook Worker to the deployment

We can accomplish all of these with Powershell….
1. This script will unload the SMA Worker from the deployment:

<#********************************************************************************************************************

Name: RemoveRBWorkerFromDeployment.ps1
Function: Removes a server from an SMA Runbook Worker Deployment
Caveats: Does not affect SCORCH jobs
Author: Matthew DeBoer
Version: 1.0
Usage: Enter the name of the server you want to remove out of the RB worker deployment

Result: 
The services are stopped on all runbook servers which pauses ALL runbooks running. 
The runbooks are given x seconds to stop.  This is controlled by the Set-SMAADminConfiguration command.
Default drain time is set at 900 seconds (15 minutes) and is configurable up to 20 minutes.
The services will stay in the stopping state until the last runbook has completed stopping. 
A new list of runbook servers is built (not containing the server to remove)
The new deployment is set and the services on the remaining servers are started.
When the services are started again, the remaining servers will continue the suspended runbooks where they left off.

*********************************************************************************************************************#>


$strRBServerToRemove = read-host -Prompt "Server to remove from Runbook deployment"
$strRBServerToRemove = $strRBServerToRemove.ToLower()

$strSMAWebService = "https://YourSMAManagementServer.fq.dn"
$arrExistingRunbookServers = Get-SmaRunbookWorkerDeployment -WebServiceEndpoint $strSMAWebService

If($arrExistingRunbookServers  | Where-Object -Property ComputerName -like $strRBServerToRemove*){
    $arrNewRunbookServers = @()
    ForEach($objRBServer in $arrExistingRunbookServers){
        $strRBServerName = $objRBServer.ComputerName
        $garbageVar = (Get-WmiObject -computer $strRBServerName Win32_Service -Filter "Name='rbsvc'").InvokeMethod("StopService",$null)
        If(!($strRBServerName.tolower().StartsWith($strRBServerToRemove))){
            $arrNewRunbookServers += $strRBServerName
        }
    }
    New-SmaRunbookWorkerDeployment -ComputerName $arrNewRunbookServers -WebServiceEndpoint $strSMAWebService -Force
    ForEach($strRBServerName in $arrNewRunbookServers){
        $garbageVar = (Get-WmiObject -computer $strRBServerName Win32_Service -Filter "Name='rbsvc'").InvokeMethod("StartService",$null)
    }
    write-host "The following runbook servers are now in the deployment: "
    Get-SmaRunbookWorkerDeployment -WebServiceEndpoint $strSMAWebService
}else{
    write-warning "$strRBServerToRemove is not in the existing runbook server deployment"
}

2. If the Runbook Service is stopped, all of the jobs have successfully stopped running on the current server. Any running jobs would now be running on one of your other servers
Is it safe to reboot?:

((Get-Service -ComputerName "ServerName" -DisplayName "Runbook Service").Status -eq "Stopped")

3. This one is easy:

Restart-Computer -force

4. Once you are ready, add the server back into your deployment:

<#********************************************************************************************************************

Name: AddRBWorkerToDeployment.ps1
Function: Adds a server from to an SMA Runbook Worker Deployment
Caveats: Does not affect SCORCH jobs
Author: Matthew DeBoer
Version: 1.0
Usage: Enter the name of the server you want to add to the RB worker deployment

Result: 
The services are stopped on all runbook servers which pauses ALL runbooks running. 
The runbooks are given x seconds to stop.  This is controlled by the Set-SMAADminConfiguration command.
Default drain time is set at 900 seconds (15 minutes) and is configurable up to 20 minutes.
The services will stay in the stopping state until the last runbook has completed stopping. 
A new list of runbook servers is built (including the new server to add)
The new deployment is set and the services on the servers are started.
When the services are started again, the remaining servers will continue the suspended runbooks from the last checkpoint

*********************************************************************************************************************#>


Try{Import-Module -Name "Microsoft.SystemCenter.ServiceManagementAutomation" -WarningAction SilentlyContinue -ErrorAction Stop
}Catch [Exception]{
    write-warning "Failed to add a required powershell module"
    break
}

    $strRBServerToAdd = read-host -Prompt "Server to add to the Runbook deployment"
    $strRBServerToAdd = $strRBServerToAdd.ToLower()
    Try{
       If(!(Get-WmiObject -computer $strRBServerToAdd Win32_Service -Filter "Name='rbsvc'")){
         write-warning "You cannot add a server to the deployment if it currently does not have the runbook worker service installed"
         break
       }
    }Catch [Exception]{
        write-warning "You cannot add a server to the deployment if it currently does not have the runbook worker service installed"
        break
    }
    
    $strSMAWebService = "https://YourSMAManagementServer.fq.dn"
    $arrExistingRunbookServers = Get-SmaRunbookWorkerDeployment -WebServiceEndpoint $strSMAWebService

    #If the new server isn't already in the list
    If(!($arrExistingRunbookServers  | Where-Object -Property ComputerName -like $strRBServerToAdd*)){
        $arrNewRunbookServers = @()

        #Stop the services on each the existing servers and add them to the new list of servers
        ForEach($objRBServer in $arrExistingRunbookServers){
            $strRBServerName = $objRBServer.ComputerName
            $garbageVar = (Get-WmiObject -computer $strRBServerName Win32_Service -Filter "Name='rbsvc'").InvokeMethod("StopService",$null)
            $arrNewRunBookServers += $strRBServerName #Add existing servers to the new server list
        }

        #Add the new server to the new server list
        $arrNewRunBookServers += $strRBServerToAdd 

        #Create the new deployment
        $return = New-SmaRunbookWorkerDeployment -ComputerName $arrNewRunbookServers -WebServiceEndpoint $strSMAWebService -Force
    
        #Start the services on each server
        ForEach($strRBServerName in $arrNewRunbookServers){
            $garbageVar = (Get-WmiObject -computer $strRBServerName Win32_Service -Filter "Name='rbsvc'").InvokeMethod("StartService",$null)
        }

        #List the servers in the deployment
        write-host "The following runbook servers are now in the deployment: "
        Get-SmaRunbookWorkerDeployment -WebServiceEndpoint $strSMAWebService
    }else{
        write-warning "$strRBServerToAdd is already in the existing runbook server deployment"
    }

These scripts are provided for demonstration purposes…you’ll need to test them in your environment….These scripts are also another good argument for checkpoints in your scripts….this is where the runbook will resume from when it switches servers…

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s