Wednesday 5 December 2012

Windows file server migration to NetApp CIFS

Whilst migrating several windows file servers (various OSes from Windows 2000 to Windows 2008 R2) to a new NetApp CIFS environment I encountered a small but show stopping issue which caused a lot of head scratching for a while until the solution became clear.

The migration approach taken was to create a hidden root share on the NetApp CIFS volume and then on each file server run a sync job between them and a subfolder beneath the root share on the NetApp.  The tool used for the copying was called Super Flexible File Synchronizer (now called Syncovery I believe) which not only copies all of the data with NTFS permissions but also copies shares and share permissions across too and sets them up on the NetApp...nice and simple!

Once all of the data had been copied and a few subsequent incremental syncs completed we were ready to cut over from the windows servers to the NetApp appliance for all of our file shares.  The process for this is quite simple really.

Firstly, we had to shutdown the file servers being migrated and then change the DNS entries for each server to point to the CIFS IP address of the NetApp.  As the NetApp would be hosting several different file servers we also needed to set up netbios aliases too so that the controller would respond to the client requests as the original file servers (this saves re-mapping drives and even dfs links which was a real time saver!).
The commands for this were:

  1. options cifs.netbios_aliases (name of server)
  2. cifs nbalias load
  3. options cifs.gpo.enable on
  4. cifs gpupdate


In our case the list of server names was quite long and thus needed to be inputted like this as the line is not cumulative and subsequent additions would otherwise overwrite the previous entries:

          options cifs.netbios_aliases server1,server2,server3,server4,....etc

Next the gotcha.
After doing the above I then went to access a file share which had been migrated and instead of seeing all of the shares I had a blank windows explorer. no error, but no shares either.
It turns out that the last step to do is to actually remove the old file servers AD computer object and allow replication to occur.
Once the computer object was deleted and the client machine rebooted I was then able to connect to the file shares fine and also the dfs links too.

I'm not exactly sure why the computer object needs to be removed but I guess it has something to do with the NetApp being joined to the AD domain (for CIFS).  If I get a more specific reason for this I'll update this blog in the future...

Tuesday 20 November 2012

Unable to start vApp - Error: Invalid network in property vami.ip0.VM_1.

There was a requirement to restart our vCenter Operations Manager vApp recently which normally would have been a fairly straightforward process (Log into each VM and initiate a shutdown of the OS and then shut the vApp down). This time though there was an issue when I came to power the vApp back on again as I was greeted by the following error:

(Invalid network 255.255.0.0 in property vami.netmask0.VM_1.)

...and this one too:
(Invalid network in property vami.ip0.VM_1.)

Now this was not something which I had seen before and threw me for a while before finally figuring it out.

The vApp is assigned it's IP settings from an IP Pool associated with the datacenter, in this case both vms receive IP, Netmask, Gateway and DNS settings from this pool. When checking this in more detail I found that the network which was associated with this IP Pool was incorrect.
What had happened was that we had migrated the vApps network from a standard vSwitch for a Distributed vSwitch a few months ago.  The port groups used in the old standard vSwitch was named slightly differently than the new port group on the VDS even though they were the same VLAN ID. Was this meant was that when the vApp tried to power on again it was still looking for the port group from the old vSwitch and as such could not find it and could therefore not power on again.

To resolve this there was two simple settings to change:

First the IP Pool needed to be associated with the correct network again.
To do this, simply go to the datacenter in the vSphere client and then select the IP Pools tab. Then right click the Pool and select properties and then go to the 'Associations' tab and place a tick in the associated network for the IP Pool.


Second, the vApp itself needed to be updated to use the correct network for each of its IP settings.
Select the vApp from the Hosts and Clusters view and then in the summary tab, select 'Edit Settings'. Select 'Advanced' from the left menu and then select the 'Properties' box to reveal the 'Advanced Property Configuration' window as below.
Next just select each entry in turn and select 'Edit' to change the network to the correct value.


Once these settings were applied, then vApp could be started and it's IPs were once again allocated to each vm and all was well!

Simple error and all entirely self inflicted!  Just be aware of vApps and their associated networks as these settings are not changed when you change the individual vms network settings.



Thursday 15 November 2012

Storage vMotion Error: The method is disabled by 'SYMC-FULL dd-mm-yyyy...'

I had this error come up the other day whilst trying to SvMotion one of our vms over to a new storage array:


Now this is actually one of those obvious and helpful error messages that you get every now and then and just by looking at the error message I could see what had caused this issue.

We use Backup Exec with the avvi agent to perform backups of some of our production vms.
The avvi agent allows us to perform SAN to Tape backups off host which means we don't need to do anything special with regards to backup configuration on any of our ESXi hosts.  The configuration process is a simple of case enable the option within the Backup Exec media servers and present the ESX Datastores to them (with the same LUN IDs etc) and that's pretty much it. Most of our vms are also running the Backup Exec Remote Agent for it's OS (Windows or Linux) which then allows us to have granular file recovery from our image based backups which is a nice feature...although not as useful when doing your backups to tape and not disk as the recovery process still needs to extract the full vmdk off of the tape before recovering the individual files to be restored to the vm or elsewhere.

A good guide for setting this configuration up can be found on Symantec's website here:

Now what usually happens when a backup job is run on a vm using this method is this:

·         The BE job starts on the media server and talks to vCenter to take a snapshot of the vms vmdk
·         Once completed the vm is now running from the snapshot and the original vmdk is static and only read by the vm  
·         BE then gets the ESXi host and guest virtual machine information from vCenter it needs to backup
·         BE then opens a connection with the ESXi server to ask for the virtual machine metadata
·         BE then informs vCenter to disable Storage vMotion for that VM to ensure that the backups can complete successfully.
·         Using vStorage APIs, Backup Exec then opens a direct data connection to the ‘unknown’ SAN volumes which have been presented to it and the virtual machine data is offloaded directly to the media server for backup
·         Once the backup process has completed the snapshot is deleted and BE disconnects from the ESXi host and informs vCenter to enable Storage vMotion again for the vm
·         Backup job then completes.


The error above is caused by the Storage vMotion being disabled by Backup Exec to run the backups.  After the backup job completes the call to vCenter does not get made or fails and so the vm is stuck with it's Storage vMotion disabled.

The trouble with this is that you often don't know this is an issue until you go to perform a Storage vMotion or unless you have vms inside an SDRS cluster and they fail to migrate to other datastores.

You can however identify these vms though by performing a lookup within the vCenter database as described in this VMware KB article:

Luckily this is a known issue and there are two very easy ways to address this if you have this issue.  
The first, and often easiest way, is to shutdown the vm and remove it from the inventory.  Then browse thedatastore where it resides, locate the vmx file and add it to the inventory again.
This approach basically gives the vm a new id within vcenter and thus gets any customised settings removed allowing it to SvMotion again.
This does pose an issue however in that you will need downtime on your vm, although very short, in order to resolve this.

The other approach, as detailed by VMware in the KB above, is to manually edit the settings within the vCenter DB for the vm affected.  Whilst this does not require a vm outage to work, it does require vCenter to be stopped whilst you access the DB and in some instances (Environments with vCloud Director, SRM, LabManager etc) this is more impacting than 1 vm being shutdown for a couple of mins and so finding a quiet evening or weekend to shut the vm down is my preferred approach and this can be easily scripted anyway to save those long hours from building up!

This is not restricted to Symantec btw.  I have seen this issue with VEEAM backup software also and as yet I'm not aware of any definitive solution to prevent this from happening from time to time. It pays to keep an eye on this if you are running a similar backup technology in your environment.


Tuesday 23 October 2012

Hot Add CPU to a Windows 2008 R2 Enterprise VM

I'm still often caught out by just how cool modern IT really is.

Today for example we had an issue with one of our businesses systems which pretty much took it out of the water.  It seems that the business had deployed a large number of additional users to this system the previous day and by now they were all starting to access the server and place significant demand on it's single vCPU.

For about an hour or more the server had been sitting there at 100% CPU (with the odd blip here and there as it churned its way through all of the work) and for new users the system was so unresponsive that it was effectively dead to them.  The few users who were already on the system and working before this CPU increase were still OK, although they were feeling the effects of the slow performance, and were still inputting data and working on their models etc.

This posed a problem.  On the one hand it was clear that this system needed to be running on more than 1 CPU now given the increased workload but on the other we didn't want to take the server out of the water (even for just a few minutes that it would take) to do this as it would mean cutting access to the users who were in and potentially causing them issues with unfinished modelling and part entered data.

Luckily enough this VM had been deployed as Windows Server 2008 R2 Enterprise.  This one decision, taken by some possibly overzealous engineer who originally built the server, saved the day as it allowed us to simple edit the VMs properties and increase the number of vCPUs allocated from 1 to 2.

Now the really cool part was watching the server via the console with task manager opened at the performance tab.  The changes were completed within vCenter and then within around 5 seconds, the OS popped up a message to state that the data was incorrect and it needed to restart task manager. Clicking OK restarted it and up popped two CPU graphs and almost instantly the CPU levels started to drop to around 55-75% utilization.
I had done this in the lab many times and new this was easily technically possible if ever we needed to do it but we had never had this occur within the few years or so that this has been possible to do. To actually use this feature in 'live' and against a running production system (of some significant importance to the business too) gave you a real sense of satisfaction.
It's great when IT just works.

Kudos to VMware and Microsoft for some pretty neat tech!

Thursday 11 October 2012

p2v'd Windows 2000 vm with high cpu

We recently acquired some old Windows 2000 physical servers which needed to be brought online in our virtual infrastructure.  It's been a while since I p2v'd any servers as we had pretty much sucked up all of the old physical servers we had since we deploying our first virtual environment and so it was time to dust down my old copied of VMware converter, a trusted utility which had been used almost flawlessly back in those early virtual days.
In fact I chose to download a newer version of this tool from the VMware website and then burn it to a CD to make a bootable disk from which to perform a cold clone of these old servers.

The cloning process was pretty standard as before with a few nice extras now afforded to me in the post migration steps making things even easier.  The newly cloned vm came up and then it was a simple process of going through all of the old hardware applications (HP management agents and the like) and removing them, along with uninstalling the old 'hidden' hardware from the servers previous physical state.
The server then came online and with a few tweeks here and there was ready to go into production again.

It was then that we noticed some issues around the performance of the vm which seemed familiar.  Within the vSphere client the CPU performance of the windows 2000 server was showing as practically 100% almost all of the time, yet within the OS the CPU was showing as idle.
I immediately thought that the HAL was not set correctly as this is a well documented issue, especially with windows 2000 vms, however when I went into device manager, and under computer, the HAL was indeed set to what it should be; 'ACPI Multiprocessor PC'.  This was the same as on the other Win2k vms which had been migrated at the same time and they were not displaying this same CPU issue.

After looking into the issue a little more it seems that the idle thread (which  normally lets the computer save power when not in use) gets stuck in a busy loop and so although the OS believes it is not actually utilizing the CPU, the physical CPU is constantly receiving commands and therefore the CPU demand for the vm is actually 100%. 

This issue was resolved in this case by changing the HAL from 'ACPI Multiprocessor PC' to 'Advanced Configuration and Power Interface (ACPI) PC'.
Select 'ACPI Multiprocessor PC' and right clicking to Properties. Select Update Driver and then in the window which appears, select 'Show all hardware of this device class' to list available drivers as below:
Select the Advanced Configuration and Power Interface (ACPI) PC model and click Next etc to install. 

This process required a reboot of course to complete and I honestly can't stress enough how important it is to backup the vm before making these changes. As a belt and braces approach we cloned the vm first and applied the changes to the clone and then monitored it for a couple of days before applying the change to the live server.
Once the system rebooted the performance was what we expected again as per the other vms migrated at the same time.

I'm not sure how this driver differs from the previous one but swapping the driver made all the difference to this servers performance and as its not going to be around too long anyway I'm happy to leave it this way until the application is moved to a new environment entirely.

Update: The above issue can also sometimes be resolved by re-installing the ACPI Multiprocessor PC driver.  Simply follow the steps above but instead of selecting the Advanced Configuration and Power Interface (ACPI) PC drive, just re-select the same driver, complete the install and then reboot the system.
This is less risky than changing the driver but if this does not work then you should look to change to the advanced driver next.

Thursday 4 October 2012

Setting Storage Path alerts

Since vSphere 4.0 there has been a large increase in the available alarms which not only come pre-configured but are also available to be created and you can pretty much now create an alarm for almost anything within vCenter.

It still surprises me though why some quite essential monitoring areas are not included within the default set of pre-configured alarms.  One such alarm is the Storage Path Redundancy alarm will let you know when you have lost paths to your SAN storage and what datastores this will be affecting etc.  This is a very simple alarm to setup but also pretty essential to virtually all vSphere implementations these days I'd imagine.

To set up the alarm select the vCenter server in the vSphere client and then go to the 'Alarms' tab.
Select 'Definitions' to see a list of all currently configured alarms and then right click in the section to create a new alarm.
Give the alarm a name ('Degraded Storage Paths' for example) and change the Monitor to 'Hosts' and then choose 'Monitor for specific events occurring on this object, for example, VM powered On'.
On the 'Triggers' tab click 'Add' and then change the Event type to 'Degraded Storage Path Redundancy'.
Next select the 'Actions' tab and Add an action to be performed when this event occurs.  This can either be an email alert perhaps to the storage team or even a task for the ESXi host to perform.
Once set, click 'OK' and the alarm is set.

It's also worth creating another alarm to go along with this once which alerts when one of the ports goes offline too.  That way you get notifications of path redundancy lost or a full port connectivity loss which will help in troubleshooting the issue being experienced.

To set this up, simply create another rule as above but this time set the trigger to be 'Lost Storage Path Redundancy' and set whatever actions you would like.

There are many other good alarms to set depending on what monitoring solutions you may or may not have in place for your virtual environment so its always good to have a look through the list of available alarms and just check that you have everything you need configured before you need it....they're not going to do that much if you've created them after the event!

Tuesday 2 October 2012

vSphere vmotion network outages

During heavy vmotion operations I was experiencing intermittent network outages of ESXi hosts and some vms running within the cluster.
This seemed to get progressively worse over a period of several weeks until it became almost every time a host was placed into maintenance mode there would be some network outage of some vms and even other ESXi hosts within the same cluster.

After initially looking at the network infrastructure we noticed that there was a large flood of unicast traffic on the vlan which was being shared by vmotion, and some windows based vms, around the time of the vmotions (to be expected in vmotion operations).

Now VMware best practice is to have vmotion and ESXi Management on their own separate vlans or networks but this had never been an issue previously with this cluster which was about 4 years old and had been upgraded over that time from ESX 3.5 to ESXi 5.0 u1 (it's current state). There had been no significant network changes during this period also which could have had a waggling finger pointed at them so it was not obvious how we had come to this issue.
It seemed obvious to start thinking that the gradual changes and growth of the cluster had started to cause this issue for us.  Over the various versions which these hosts have been running the vmotion feature has been greatly enhanced and improved and the amount of simultaneous vmotions a host can support has also increased from 2 to 4 (or 8 with 10Gbe) as can be seen here:

(taken from the vSphere 5.1 Documentation center here)
Network Limits for Migration with vMotion
Operation
ESX/ESXi Version
Network Type
Maximum Cost
vMotion
3.x
1GigE and 10GigE
2
vMotion
4.0
1GigE and 10GigE
2
vMotion
4.1, 5.0
1GigE
4
vMotion
4.1, 5.0, 5.1
10GigE
8

There had also been a sizable growth in the number of hosts and virtual machines in this cluster and the hosts had been increased in capacity etc along the way too.  This all resulted in a much heavier demand for vmotion during the process of placing a host into maintenance mode as often I would be looking at somewhere between 20-50 virtual machines being migrated across the cluster.

It turned out that this was in fact our issue and so as we had spare capacity within our hosts, due to the recent removal of some iSCSI connections to this cluster, we were able to hook up a couple of dedicated vmotion nics per host and placed them into their own vlan away from the management and any other systems.
vSphere 5.0 gives us the ability to utilize more than 1 vmotion nic per host. All that was needed was to create 2 new vmkernel ports on a new vSwitch and have vmkernel port 1 bound to vmnicX as active and vmnicY as standby, then just reverse the configuration for the second vmkernel port.
Once the two new vmotion ports were created and assigned IPs on the new vlan, I just removed the old vmotion port which was in the shared vlan and then that was all of the configuration which was needed.

I performed a few test migrations after that and the performance improvement was easily visible even without measuring it.  We used to have windows vms with 4GB ram move between hosts within 1-2 mins and now we are getting them within 30 seconds.

Best of all, when now entering maintenance mode on a host, even one which is running many vms, we are no longer getting any network outages and the process is a lot speedier too. Happy times again!



Monday 24 September 2012

Free VMware SRM training videos

VMware have just released a set of free (yup, totally free!) training videos for Site Recovery Manager (SRM) on their website:

http://blogs.vmware.com/education/2012/09/free-site-recovery-manager-training.html

This is a great resource for those wishing to deploy SRM and I would urge all to take a look through the videos before starting your deployments.


Wednesday 19 September 2012

Migrating VMs running on VSS to VDS in a Production ESX cluster (Migrate Virtual Machine Networking…)


With one of our ESXi 5.0 clusters growing to 12 hosts and our Networking team constantly wanting to deploy new vlans like they are going out of fashion it was time to implement a distributed vSwitch (vDS) to the cluster in order to reduce the administrative overhead of adding all of the port groups to each vSwitch on each host (not really the case but it’s always good to keep up with the professional dogging of those poor network guys eh?). 

The process to deploy a new vDS to a cluster is pretty straight forward and you can follow the process from within vCenter here: VMware KB

Once created the next step was to create each of the vlan port groups onto the VDS.  Here I simply setup the vlan with the same name as used currently on each of the VSS (you can do this as the name has ‘(dvswitch)’ appended to it anyway so it keeps these separate from the existing port groups when selecting network connectivity when editing VMs) and the same vlan ID entry etc.

Next I moved 2 of the 4 x 1Gb adaptors from each VSS into the dvuplink ports on the VDS.  This then allowed the existing VSS port groups to continue to service network requests for all of the running VMs and also allowed me to start moving VMs from the VSS to VDS .

To migrate the VMs from their current port groups on each VSS to the newly created port groups on the VDS you can use the excellent Migrate Virtual Machine Networking utility which manages the bulk modifications to VMs.

To do this simply so to the networking screen in the VI client and right click on the new VDS and choose ‘Migrate Virtual Machine Networking…’

Next select the source network from the drop down list (this is the current port group that you want to move VMs off of) and then select the destination network (the corresponding port group on the dvSwitch)

Click Next and you can now select all or some of the VMs to be migrated.  If you select all of the VMs you’ll be able to sit back and watch as each VM is modified in turn and moved over.  It really is simple and best of all results in no network outage to the running VMs.
I migrated several hundred VMs across our various vlans to the distributed switch without one little blip! 
Then it was a just a matter of going through each of the hosts and cleaning up the old port groups and vSwitches which were no longer being used.

Sunday 16 September 2012

vSphere VM deployment customizations

A small but annoying thing had started to happen to your deployments of Windows 2008 R2 vms in our production environment recently. Whenever we deployed a new vm and used our pre-saved customization specification the vm would be deployed as expected except that it did not join the new vm to our production domain.
The image would be customised, the server name changed, IP settings applied, administrator password set etc but it would no longer join the vm to our windows domain.

Alarmingly, although the option was set within the specification there were no errors recorded for this in the logs on the newly deployed vm (these can be found at c:\windows\temp\vmware-imc\guestcust.log) which I would have expected.

The answer it turned out was very simple.  The customization had been modified to have domain\username in the username field of the domain customization properties.  Although this looks perfectly reasonable to have in a windows environment this actually needs to be just the username of the domain account which will be joining the vm to the domain.

After changing the pre-saved customization to just the account name and re-entering the password I fired off a test deployment and voilà, 1 windows vm deployed and sitting on our production domain as before!

Wednesday 12 September 2012

Virtual Machine disk consolidation fails with I/O error on change tracking file

A vm was displaying the warning that 'Virtual machine disks consolidation is needed' which is a nice feature of vSphere 5 which now actively tells you about this issue (It's always been there in previous releases but never highlighted in this way until 5.0).

We often get this issue as we use a snapshot backup technology to backup our vms each day and for some reason or other sometimes the remove snapshot process does not complete properly and we get this situation where the snapshots are removed but the snapshot files are still present and referenced in the vm. See the following VMware kb article for details.Consolidating snapshots in vSphere 5

Usually this is a simple process of right clicking the vm, selecting 'snapshot > consolidate' to have the snapshot child disk files consolidated back to the parent disk file but in this case the consolidation failed with the error message: 'A general system error occurred: I/O error accessing change tracking file'.

After some investigation I found that our backup system had a lock on one of the files and so I was able to release the file from the backup software and then re-run the consolidation which completed and all was good again!
The troubleshooting steps to identify the locked file can be found here: Investigating virtual machine file locks on ESX/ESXi

Previously I've also been able to resolve the issue of not being able to consolidate vm disks by creating a clone of the troubled vm and bringing it up as the active vm and then deleting the old one. Not always possible though in a production environment!

Tuesday 11 September 2012

vCenter Operations Manager not displaying Risk or Efficiency data

So finally made the upgrade from CapacityIQ and deployed VMwares new vCenter Ops Manager in it's place.  The upgrade process of deploying the new Appliance was straight forward and error free.

During the installation process you have the option to import your old data and settings from the CapacityIQ appliance into the new vCOM database so you don't lose any of the existing trending information etc.

This process worked like a charm but after a few days or so I noticed that the Risk and Efficiency data never populated on the dsahboard screen and I was not able to get any Capacity or Trending information.

After looking at a few blogs and the excellent VMware Communities I was still not able to find why this was not working and so logged a support call.  The answer was simple and when thinking about it, obvious.
The below was the summary provided to me from support: 

To calculate time remaining and capacity remaining metrics, there are overall 5 resources that we consider
 
* cpu
* memory
* disk IO
* disk space
* network IO
 
However, these 5 resources do not apply to all object types. So under the hood, we actually consider a subset of the applicable resources for each object type. For example, for datastore object we consider only selected resources out of disk space and disk IO resources ; for vm object we consider only selected resources out of cpu, memory, disk space resources; for host and up, we consider selected resources out of all resources. For a given object type, if all the applicable resources are unchecked (i.e., none are selected), the metric calculation module is unable to figure of the metric dependency and unable to calculate the time remaining or vm remaining values.


Now in CapacityIQ we never cared for disk Capacity as a factor in our host capacity reports as we run several SANs which are attached to our ESXi cluster and this space is only carved up and added to the environment on a per-need basis. We were mainly only concerned about CPU and Memory primarily and so these settings were not selected in CapacityIQ and so did not come accross to the new vCOM deployment when we imported the settings and data from CapacityIQ.

In our case, simply adding 'Disk Space capacity and usage' and/or 'Disk I/O capacity and usage' in the "Capacity & Time Remaining" configuration panel solved the problem!
When the Analytics process next run on the system (1am by default) the Risk and Efficiency areas populated and all was well.

The support guy did mention that this is being fixed in version 5.6 so that at least one of the applicable resources for each object type is checked, but for now it's a manual process.


Thursday 6 September 2012

VMware SRM 5 recovery plan environment scripts

How to create a recovery plan script in SRM5 that will perform different tasks depending if the recovery plan is in test mode or recovery mode.



It's pretty easy to add scripts to recovery plans in SRM5 to perform all sorts of tasks in recovered environments or VMs but what if you need to have the script do something different when it is run in a test scenario like add some test environment specific routes or add some host file entries to allow recovered VMs to talk to one another in a non-production LAN (no DNS or Gateways exist for example)?  Well thanks to SRM5 you can make use of some environment variables which are injected into the recovered VMs by the SRM service in order to do just that!

The main variable to look at here would be VMware_RecoveryMode. This variable has a setting of either test or recovery depending on how the recovery plan is being run at the time and so can be referenced in your script to act differently according the the value of this variable.

A basic example of this can be found in the below script which is a simple batch file...

IF %VMware_RecoveryMode% EQU test (Goto TestRun) Else (Goto OtherRun)
 
:TestRun
for /f "delims=: tokens=2" %%a in ('ipconfig ^| findstr /R /C:"IPv4 Address"') do (set tempip=%%a)
 set tempip=%tempip: =%

route add 10.10.1.0 mask 255.255.255.0 %tempip% -p
route add 10.10.2.0 mask 255.255.255.0 %tempip% -p

Echo Routes Applied to Test environment on %date% at %time% >> c:\srm\srmlog.txt

Echo 10.99.53.13 server1.company.com >> %windir%\system32\drivers\etc\hosts
Echo Host file entries Applied on %date% at %time%>> c:\srm\srmlog.txt
EXIT

:OtherRun
IF %VMware_RecoveryMode% EQU recovery (Goto RecoveryRun) Else (Echo an unexpected result occurred on %date% at %time% >> c:\srm\srmlog.txt)
EXIT

:RecoveryRun
Echo Recovery started on %date% at %time% >> c:\srm\srmlog.txt
EXIT

This script checks to see if the recovery mode is 'test' and if it is then proceeds to run some things under the :TestRun section.
If the mode is not test then it checks to see if it is in recovery mode and again if it is it then runs some things under the :RecoveryRun section.
If it for some reasons doesn't see either test or recovery in the variable it will just write a simple log file to C:\SRMfolder of the VM running the script.

There are other environment variables available to play with too which can be found in the SRM administrators guide so hop over to VMware and check it out