Wednesday, 9 May 2007

Automatically update and reboot Citrix Presentation Servers on a schedule

When we originally set up our Citrix PS 4.0 farm we needed to establish an efficient method of patching our servers without any maintenance windows or visible downtime. This method of patching needed to tie in with our use of WSUS and not add any administrative overhead.

The answer seemed easy, we need a system that detects if there are any outstanding patches (those approved for installation on WSUS, or available from Microsoft Update depending on environment), if there are outstanding patches it then needs to stop subsequent TS/Citrix logins, wait for everyone to log off, download and install the patches and then reboot.

I was sure that there would be plenty of people who need a similar solution but having spent some time looking all I could find were scripted methods for installing patches, tools that rebooted servers when there were no remote logons (all of which I found inconsistent) and the obvious change logon /disable command. I needed a way of pulling this together so I wrote a number of batch files and VBScript (I can't claim credit for all the VBScipt as I downloaded it from http://www.wsus.info/).

We set WSUS setting via GPOs to check for updates every night and install between 04:00 and 06:00. Therefore, if we approve a WSUS update to a group the servers in that group will update that evening. The first step was to change this GPO for the Citrix PS servers so that the servers were set to only download the updates and not install (although you could set it to do nothing, the important thing is that the servers are set to look at the WSUS server for updates). the next step was to create a scheduled task to run my scripts, which I set to run every night.

Now to the scripts, the first of which determines if updates are available: if there are it calls a batch file, if not it quits and nothing happens until the scheduled task runs again the next night.

Set updateSession = CreateObject("Microsoft.Update.Session")
Set updateSearcher = updateSession.CreateupdateSearcher()
Set searchResult = _updateSearcher.Search("IsInstalled=0 and Type='Software'")
Set File = CreateObject("Scripting.FileSystemObject")

Set LogFile = File.OpenTextFile("R:\Temp\WSUSUpdates\CheckForUpdates.log", 8, True)
LogFile.WriteLine("***************************************************************")LogFile.WriteLine( "START TIME : " & now)LogFile.WriteLine( "Searching for updates..." & vbCRLF)

LogFile.WriteLine( "List of applicable items on the machine:")

For I = 0 To searchResult.Updates.Count-1
Set update = searchResult.Updates.Item(I)
LogFile.WriteLine( I + 1 & "> " & update.Title)
Next

If searchResult.Updates.Count = 0 Then
LogFile.WriteLine( "There are no updates waiting to be installed, quiting.")
WScript.Quit
Else
LogFile.WriteLine( "There are updates waiting to be installed.")
LogFile.WriteLine( "Stopping the IMA service to prevent logons.")

dim shell set shell=createobject("wscript.shell")
LogFile.WriteLine( "Checking for logged in users....")

shell.run "R:\Temp\WSUSUpdates\CheckLoggedInUsers.bat 350"
set shell=nothing
WScript.Quit

End If

You will notice that if patches are available the batch file CheckLoggedInUsers.bat is called. This batch file disables new logins and checks for users logged in remotely using the ICA protocol and if there remote users it loops round until they are not! It takes the parameter from the first VBScript "350" as the number of times to retry before giving up and quiting. You will need PS Tools from: http://www.microsoft.com/technet/sysinternals/Utilities/PsTools.mspx installed for this to work and sleep.exe from the Windows 2003 server resource kit tools: http://www.microsoft.com/downloads/details.aspx?FamilyID=9D467A69-57FF-4AE7-96EE-B18C4790CFFD&displaylang=en.

@echo off
If "%1"=="" GOTO Parameter
set E2=0
set E1=0
set E0=0
%systemdrive%
cd %systemdrive%\temp\wsusupdates\
date /t >>CheckLoggedInUsers.log
echo Stopping logons >>CheckLoggedInUsers.log
echo Stopping logons
change logon /disable
goto firstrun
:loop
echo Users logged on, checking retry limit
echo Users logged on, checking retry limit >>CheckLoggedInUsers.log
echo Incrementing retry count >>CheckLoggedInUsers.log
echo Incrementing retry count
:E0
if %E0%==9 goto E1
if %E0%==8 set E0=9
if %E0%==7 set E0=8
if %E0%==6 set E0=7
if %E0%==5 set E0=6
if %E0%==4 set E0=5
if %E0%==3 set E0=4
if %E0%==2 set E0=3
if %E0%==1 set E0=2
if %E0%==0 set E0=1
goto DONE
:E1
set E0=0
if %E1%==9 goto E2
if %E1%==8 set E1=9
if %E1%==7 set E1=8
if %E1%==6 set E1=7
if %E1%==5 set E1=6
if %E1%==4 set E1=5
if %E1%==3 set E1=4
if %E1%==2 set E1=3
if %E1%==1 set E1=2
if %E1%==0 set E1=1
goto DONE
:E2
set E1=0
if %E2%==9 set E2=0
if %E2%==8 set E2=9
if %E2%==7 set E2=8
if %E2%==6 set E2=7
if %E2%==5 set E2=6
if %E2%==4 set E2=5
if %E2%==3 set E2=4
if %E2%==2 set E2=3
if %E2%==1 set E2=2
if %E2%==0 set E2=1
goto DONE
:DONE
If "%E2%%E1%%E0%"=="%1" GOTO RetryLimit
echo Sleeping for 60 seconds >>CheckLoggedInUsers.log
echo Sleeping for 60 seconds
sleep.exe 600
:firstrun
echo Checking for ICA Sessions >>CheckLoggedInUsers.log
echo Checking for ICA Sessions
query session FIND "ica-tcp#" >nul
IF NOT errorlevel 1 GOTO loop
:update
Echo Performing an update >>CheckLoggedInUsers.log
Echo Performing an update
InstallWSUSUpdates.vbs
psshutdown /r /f
:RetryLimit
echo Retry limit exceeded, script exiting >>CheckLoggedInUsers.log
echo Retry limit exceeded, script exiting
GOTO end
:Parameter
echo No retry limit entered (enter three characters i.e. 003), script exiting >>CheckLoggedInUsers.log
echo No retry limit entered (enter three characters i.e. 003), script exiting
GOTO end
:end


We now have no logged in users, new users cannot login and there are patches waiting to be installed. The third of the scripts installs the patches:

Set updateSession = CreateObject("Microsoft.Update.Session")
Set updateSearcher = updateSession.CreateupdateSearcher()
Set searchResult = _
updateSearcher.Search("IsInstalled=0 and Type='Software'")
Set File = CreateObject("Scripting.FileSystemObject")
Set LogFile = File.OpenTextFile("R:\Temp\WSUSUpdates\InstallWSUSUpdates.log", 8, True)
LogFile.WriteLine("***************************************************************")
LogFile.WriteLine( "START TIME : " & now)
LogFile.WriteLine( "Searching for updates..." & vbCRLF)
LogFile.WriteLine( "List of applicable items on the machine:")
For I = 0 To searchResult.Updates.Count-1
Set update = searchResult.Updates.Item(I)
LogFile.WriteLine( I + 1 & "> " & update.Title)
Next
If searchResult.Updates.Count = 0 Then
LogFile.WriteLine( "There are no applicable updates.")
WScript.Quit
End If
LogFile.WriteLine( vbCRLF & "Creating collection of updates to download:")
Set updatesToDownload = CreateObject("Microsoft.Update.UpdateColl")
For I = 0 to searchResult.Updates.Count-1
Set update = searchResult.Updates.Item(I)
LogFile.WriteLine( I + 1 & "> adding: " & update.Title )
updatesToDownload.Add(update)
Next
LogFile.WriteLine( vbCRLF & "Downloading updates...")
Set downloader = updateSession.CreateUpdateDownloader()
downloader.Updates = updatesToDownload
downloader.Download()
LogFile.WriteLine( vbCRLF & "List of downloaded updates:")
For I = 0 To searchResult.Updates.Count-1
Set update = searchResult.Updates.Item(I)
If update.IsDownloaded Then
LogFile.WriteLine( I + 1 & "> " & update.Title )
End If
Next
Set updatesToInstall = CreateObject("Microsoft.Update.UpdateColl")
LogFile.WriteLine( vbCRLF & _
"Creating collection of downloaded updates to install:" )
For I = 0 To searchResult.Updates.Count-1
set update = searchResult.Updates.Item(I)
If update.IsDownloaded = true Then
LogFile.WriteLine( I + 1 & "> adding: " & update.Title )
updatesToInstall.Add(update)
End If
Next
logFile.WriteLine( "Installing updates...")
Set installer = updateSession.CreateUpdateInstaller()
installer.Updates = updatesToInstall
Set installationResult = installer.Install()
'Output results of install
LogFile.WriteLine( "Installation Result: " & installationResult.ResultCode )
LogFile.WriteLine( "Reboot Required: " & installationResult.RebootRequired & vbCRLF )
LogFile.WriteLine( "Listing of updates installed " & "and individual installation results:" )
For I = 0 to updatesToInstall.Count - 1
LogFile.WriteLine( I + 1 & "> " & updatesToInstall.Item(i).Title & ": " & installationResult.GetUpdateResult(i).ResultCode )
Next
'if installationResult.RebootRequired = -1 then
'LogFile.WriteLine( "reboot the server")
'strComputer = "."
'Dim refWMIService
'Set objWMIService = GetObject("winmgmts:" _
'& "{impersonationLevel=impersonate,(Shutdown)}!\\" & strComputer & "\root\cimv2")
'Set colOperatingSystems = objWMIService.ExecQuery ("Select * from Win32_OperatingSystem")
'For Each objOperatingSystem in colOperatingSystems
'ObjOperatingSystem.Reboot()
'Next
'end if
LogFile.WriteLine( "STOP TIME : " & now)
LogFile.WriteLine("***************************************************************")
LogFile.Close


You can comment back in the VBScript reboot command but I found using psshutdown /r /f within the calling batch file (script 2) much more dependable.

I also have a scheduled task to re-enable logins at bootup: change logon /enable

That is it. This has been running problem free for about six months and does not require any administrative overhead (any more than WSUS does anyway).

Suggestions and comments always welcome!

5 comments:

Anonymous said...

How is the third script, Install Windows Update, toggling the terminal server to install mode?

Rob Head said...

Simple answer, it doesn't. When installing updates from Windows Update, the Terminal Server
doesn't have to be in install mode, since those updates only affect
the HK_Local_Machine portion of the registry, not the
HK_Current_User. Rob

AlainAssaf said...

You haven't run into a situation where all the servers are offline at the same time? I use WSUS and Res Wisdom to accomplish basically the same thing, but we do half the servers in the farm one night and the other half the next night.

Alain
http://wagthereal.wordpress.com

Rob Head said...

Hi Alain,
We use WSUS to release the updates to half of the servers one night and the other half another night, pretty much the same as you.
This has been working faultlessly for about 2 years now, so we are very happy with it, but res Wisdom looks interesting.
Cheers,
Rob

Martin Zugec said...

Hi Rob,

we have very advanced solution called S4Matic to manage XenApp farm reboot.

We have multiple enterprise customers, currently biggest implementation is 350 servers, however within 3 months we should have 700 at one site. S4Matic maintenance is designed to support huge and small environments (from 2 servers)


Major features:
1.) Reboot are done per silo, not per farm. What this means is that we guarantee that every resource (publish application) is available as long as there are at least 2 servers assigned to it
2.) No downtime needed - solution is designed for 24\7 environments
3.) Intelligent reboots - servers are rebooted dynamically, when possible, not at predefined schedules. S4Matic will also detect if server was not rebooted successfully.
4.) Seamless reboots - by design, your business shouldn't notice that servers are in maintenance
5.) 50% guarantee - if anything goes wrong, S4Matic should keep at least 50% of all servers assigned to all published applications - for example if one of patches breaks your windows installation, S4Matic will stop execution of maintenance on all servers from that silo
6.) No configuration needed - S4M is designed the way to automatically adapt to your farms. Servers are loaded from XenApp itself and assigned to silos based on their published applications
7.) Maintenance evaluation - S4M will evaluate constantly every silo if there are enough servers to finished maintenance
8.) Emergency maintenance supported - in the middle of the week, you can specify that you need emergency maintenance on group of servers. Two modes are supported - automated maintenance (let's say every weekend) and on-demand maintenance
9.) Every operation supports timeouts, so servers should never get stuck in the middle of the process
10.) tons more (event logging, trapping of unexpected events, retrying failed steps...)

Send me email to m.zugec(at)loginconsultants.com if you are interested, I'll be more than happy to share additional details with you. We got even customers that decided to use this maitenance mode to rebuild completely their farm once per month.

Martin