Azure backup service - new retention policy options

Since the Azure Backup service was launched way back in October 2013 there was a desire to move some of our backup jobs over to this service. In part, this was to alleviate the strain on our existing on-premise solution, but also cloud storage as a backup destination makes a lot of sense, especially when you think about long term data retention.  Managing backups on tape with 3rd party services is OK, but can you really be sure (without considerable expense) that your data will be recoverable if/when you ever need to go back to your tape in a year or mores time?

Since the launch of this service though I have not been able to, as yet, have this service running in any production environment.  This has been due to a number of issues but the main one being that the data retention policy was just too short.  This though, has now changed!

Previously you only had 120 recovery points available to be used.  not a problem you would think, until you then discover that you can only have one policy per server...daily or weekly.  If you set a daily backup, say Monday to Friday, you could save your backups for up to 169 days (a little over 5 months) before you would then hit your retention limit.

If you changed to a weekly backup to get a longer retention period you would have backups going as far back as just over 2 years...better, but then these would only be weekly backups, with no daily jobs running.  Not a good place to be!

However, as of 10th February 2015 Microsoft have now addressed this...and in quite a big way.
They now support up to 366 recovery points and also multiple policies per server.  using these new options now gives you the ability to have backups as you would expect within most enterprises with retention periods of many years whilst still retaining at least daily backups of your datasets!
The below screenshot shows the out-of-the-box default policy configured when you install the latest Azure Backup client onto your windows server and register it with your vault in Azure.

It's quite a change as you can see and features not only daily, weekly and monthly policies but also yearly too, with a default retention period of 10 years for your yearly backups!

I'm not saying that this is now perfect.  There are still many areas in need of improvement (such as alerting and centralised views of all of your backup jobs at the very least) but this at least covers the number 1 reason why we would never have used this service for any production jobs previously and is certainly going to be re-evaluated again for some of our services.

It would be great to hear if anyone else has much experience of this service and any other pros/cons you have encountered?

Restoring long file or file paths from volume shadow copy

We have had a couple of instances recently where we have tried to restore files for users using the previous version feature on a Windows client/server as well as from a CIFS file share and because of the very long folder paths and file names used the restore/copy process fails to restore all of the data back to the live environment (it doesn't actually tell you this in some cases!).

On a windows machine you need to expose the volume shadow copy via command line in order to create a persistent link which we could map to so that we reduce the path length ( more indepth article on that process can be found here) but on a NetApp CIFS share what’s the process?

Well its actually much easier than doing it the Windows method J

You need to first expose the snapshot directory of the CIFS volume.  This is done by running the command options cifs.show_snapshot on on the vfiler and then connecting to the root volume share containing the data to map a network drive. Within this folder you will now see the ~snapshot folder containing all of the previous snapshots kept on disk.

Simply browse the snapshot folder for the date/time you require (you can get the correct time info from the snapshot view within the OnCommand tool for the volume) and when you are suitably along the file path, simply map a drive to the folder and another drive to the same directory on the live share and copy/restore away!

If the CIFS share you are trying to restore from is on a vfiler and not the root filer then you will need to enable the snapshot view on the vfiler and not the root filer. In that case, log into the root filer and then type: vfiler context vfilername (or whatever vfiler name yours is called) and then run the options cifs.show_snapshot on on the vfiler.

·        

One last point - Once you have finished the restore don't forget to turn off the snapshot view on the filer/vfiler (unless you want to have this enabled)

Hope this helps anyone else who has to do this in the future…

A NetApp Flash Accel 1.3 deployment

***Disclaimer*** This is a bit of a long post.  I wanted to get it all down in one go, so apologies if you are still reading this in another 30 mins time :-)

So I have just finished a deployment of the newly released NetApp Flash Accel 1.3 for VMware (as of 30th Jan 2014) and I have to say I am impressed.
Not only was the entire install process straightforward and clean, but the performance benefits of the deployment were being seen within the first hour of VMs being migrated - always a benefit when you have management looking over your shoulder as you deploy a new solution!

For this deployment I was installing 6 new HP BL460c G8 blades, each with 2 x Intel DC3700 400GB SSD drives in a RAID 0 configuration (I am using the MicroSD card for the ESXi OS as these blades only have 2 x 2.5" disk slots).

A couple of pre-requisites exist for this solution, in that you must be running vSphere 5.1 (although you do not need Enterprise Plus license as any 64-bit 5.1 version is supported) and also you need to allow for additional memory requirements of the host depending on how mush SSD you have configured for Flash Accel in each host.  The second point is almost mute though for all but the very heavily used vSphere deployments as the requirements for version 1.3 have dropped from 0.006GB to 0.0035GB GB of RAM per 1 GB of SSD. As described by the manual you therefore only need 3.5GB of additional host RAM for 1TB of SSD...not a lot really considering most modern ESXi hosts are likely to sport 128GB, 192GB or even 256GB+ of RAM these days.

Other considerations for deployment of Flash Accel is that currently it only supports Windows 2008 R2 and higher VMs.  This is because there is an OS agent that needs to be installed on each VM which is being enabled for Flash Accel and currently NetApp have only written a windows driver, although they do state on their webpage that a Linux driver will follow at some point.
Now this is going to be a pain point for some people as Linux VMs are popular (and in many cases, getting more popular) and even our deployment would have benefited further from being able to accelerate Linux VMs (of which we have many) as well.

NB: If you are running Enterprise Plus licences and have upgraded to vSphere 5.5 then you always have vFlash as an option.  This solution from VMware is kernel based and so does not require any drivers and therefore supports Windows and Linux VMs the same. Do bear in mind though that one of the main reasons that you are likely to be deploying Flash Accel is that you are also running a NetApp filer as your back end storage and if you are also utilising the filer for snapshot backups and replication of your VMs then Flash Accel is going to be a much safer bet for this given that they are all about the data coherency (See the NetApp Geek blog here for more on that: LINK)

Once I had all the hardware built and SSDs in placeI was then ready to deploy the Flash Accel Management Console (FAMC).  This is an ova template which is downloaded from the NetApp site and deployed onto your vSphere cluster.  It's a Linux VM which basically manages the deployment of agents to the Hosts and VMs as well as assigning cache disks to the VMs and even showing analytics of the VMs performance.

This is the usual ova kind of deployment with nothing to worry about.  The only pointer I'd give you is to only enter a server name and not an FQDN in the deployment wizard otherwise it fails and you have to re-enter all the info again within the console of the VM. It doesn't break anything but was annoying.

Once the FAMC is up and running I simply hit the IP address in a browser and logged in with default username and password of; u: admin p: netapp

Here I setup the appliance to connect to the vCenter server, entered a username and password for the hosts and then a generic local admin account for the guests VMs; This is used to deploy the OS agent for VMs being cache enabled.  Again all straightforward and simple stuff.

Next I needed to allow the FAMC to perform an inventory of the environment (this took a while as I have a large estate connected to the vCenter) and once complete I could see it list all of the hosts and VMs.

Now in my deployment I had created a dedicated cluster which would be the only one with SSD drives for caching (at present anyway), so I only had to deploy the host agent to the 6 hosts in this cluster.
Again, this is pretty straight forward.  Place the host into maintenance mode and then select the host and push the agent from the FAMC (you upload the latest host and vm agents as part of the setup of the FAMC btw).  The host will be rebooted once installed and then I assign the local SSD on the host to be a caching disk for Flash Accel and finally enable the disk.
Quick hint:  Don't forget to take the host out of maintenance mode once the agent has been installed otherwise you will not be able to assign the SSD disk or enable it.
Perhaps future releases of this console will automatically place the host in and out of maintenance mode but for now you will just get an error message if you forget to do this yourself.

With all of the hosts installed the next step was to install OS agents onto VMs that were to be enabled for caching.
This is something that needs to be given some thought before going ahead and just enabling caching on all Windows VMs in the cluster.  This solution is a read IO caching solution which means that it is only servicing read IO of the VM from the local SSD cache.  VM writes are still going to be placed back to your NetApp controller, ensuring that your data is central and secure as usual but will not be (directly) accelerated.

To identify VMs which are going to be good candidates for Flash Accel I used a SolarWinds product called Virtualisation Manager.  This shows VM IOPs and breaks down read and write IO values easily to see which VMs are going to benefit from the cache.  It can also break this down further into which vdisk is generating the IO so you can tune the cache better and only cache the drives which are generating the read IO.  Other solutions that can do this would be vCenter Operations Manager (vCOPS) or NetApp DFM (now called Operations Manager I believe).

Once the VMs were identified the biggest culprits were the usual suspects; Domain Controllers, SQL servers and web content servers were all seen to be producing read IO higher than average across the environment.

The OS agent installation process is yet another easy process but it does require at least 1 reboot to complete so scheduled downtime is needed to roll this out in production environments.
Quick hint: You need to have a UAC turned off for the installation to work on both 2008R2 and 2012/2012R2.  If UAC is enabled then turn this off by running msconfig and using the UAC tool to turn UAC off, then reboot the VM to apply the changes.
NB: Windows 2012 and 2012R2 also need a registry change made to actually turn UAC off. See the link here for more on that: LINK

With UAC off and the agent installed and the VM rebooted I could then allocate the cache on the VMs. I was careful to only set a size that fit the dataset of the VM being enabled.  It's not always required to allocate a whole disk worth of cache to a VM to get benefit.  For me, I allocated 10GB of cache to our DCs as each of these VMs have around 60GB of disk allocated and approximately 30-40GB on use.  As you can see from the screenshots of the analytics below, this yielded an excellent cache hit rate and meant that I had more cache left over to allocate to other larger VMs too.

The above screenshot is a DC which I knew was going to give good cache hit rates given the nature of what a DC does.

The screenshot below shows a 6 hour period of another DC from enabling the cache:

You can see that as the cache population of the cache rises, the hit rate goes up meaning that more of the read IO is being served from the cache instead of the NetApp controller.

It's early days for the deployment but already I have seen a reduction in read IO on the controllers.  So far so good and once this cluster is populated fully I plan to back-fit our other clusters with cache and do the same again with VMs on them too.

I hope to share some more experiences of this solution soon and will give updates on the success of other systems as they are enabled.

Thanks for sticking around until the end of the post :-)

Getting around vCenter 512bit certificates

Can't get vCenter WebClient  to work on your new Windows 8.0/8.1 machine as the certificate is not trusted?

The issue arises when you have deployed a vCenter server from version 4.0 and by default the self-signed certificate generated by vCenter was 512bits. If you have never replaced your certificate then it will still be 512bits. If you have upgraded your vCenter to 4.1, 5.0, 5.1 or even 5.5 then, unless you have replaced the certificate along the way, it will still be the same 512bit certificate it was when you first started out.

Nothing very wrong with this setup, until MS released KB2661254 which changed the default minimum accepted key length from 512bits to 1024bits to nudge people up the security ladder a little. This resulted in vCenter certificates no longer being supported by clients and hence stopping access to vCenter web portal.

Now the correct way to deal with this is to generate a new vCenter certificate which has at least a 1024bit key (preferably 2048bits) and this will then not only allow the updated clients to function again, but will give you a warm fussy feeling that only running your environment at a higher security level can achieve. This is, however, easier said than done. 

The process of replacing certificates within vCenter is not straightforward. VMware have significantly improved the process by way of the SSL Automation Tool for vCenter 5.0 and above but this is still a fairly lengthy process which is fraught with possible danger (of breaking your vCenter deployment). This needs to be planned and tested and adequate backup and recovery processes put in place before you proceed with doing this on a mature production environment.

A short term fix to get around this is to once again trust the 512bit key and proceed as you were.

The below command can be run from a command prompt on a Windows client to revert the KBs effects:

certutil -setreg chain\minRSAPubKeyBitLength 512

Obviously, doing this will also result in the client trusting ALL 512bit keys that are out there so this should only be viewed as a short term fix whilst you plan the certificate upgrades for vCenter as recommended by VMware.

Once you have resolved the certificate issues and are now sporting a shiny new 1024bit (or 2048bit) certificate, don’t forget to revert the changes above to secure your client(s) again. This can be easily done by removing the registry entry that the above command creates here:

HKEY_LOCAL_MACHINE\Software\Microsoft\Cryptography\OID\EncodingType0\CertDLLCreateCertificateChainEngine\Config\minRSAPubKeyBitLength

vCloud Director sysprep files

Had some fun running up a vCD server this past week so thought I'd post a quick memo to advise of the following changes in vCD between vCD 5.1 and vCD 5.5 regarding sysprep files.

I had been following some excellent blogs on the vCD 5.1 install process from Kendrick Coleman (Install vCD 5.1 & vCD Networking) and applying this to my vCD 5.5 installation.  When I tried to follow the process copy the sysprep files over to the vCD cell I hit a snag as there was no script to run to generate the sysprep files required. This, it turns out, is because in 5.5 they have improved this process and now you simply need to create the directories and place the sysprep files into the directory and away you go.  Not even a service restart is required to start customizing older OSes through vCD.

The folder locations in vCD 5.5 should be (extract taken from the VMware install document for vCD 5.5 - which I should have read more keenly it seems!):

Procedure:

1. Log in to the target server as root.
2. Change directory to $VCLOUD_HOME/guestcustomization/default/windows.
   [root@cell1 /]# cd /opt/vmware/vcloud-director/guestcustomization/default/windows
3. Create a directory named sysprep.
   [root@cell1 /opt/vmware/vcloud-director/guestcustomization/default/windows]# mkdir sysprep
4. For each guest operating system that requires Sysprep binary files, create a subdirectory of
   $VCLOUD_HOME/guestcustomization/default/windows/sysprep.
   Subdirectory names are specific to a guest operating system and are case sensitive.
• Windows 2003 (32-bit) should be called svr2003
• Windows 2003 (64-bit) should be called svr2003-64
• Windows XP (32-bit) should be called xp
• Windows XP (64-bit) should be called xp-64
5. Copy the Sysprep binary files to the appropriate location on each vCloud Director server in the server group.
6. Ensure that the Sysprep files are readable by the user vcloud.vcloud.
   Use the Linux chown command to do this.
   [root@cell1 /]# chown -R vcloud.vcloud $VCLOUD_HOME/guestcustomization

When the Sysprep files are copied to all members of the server group, you can perform guest customization

on virtual machines in your cloud. You do not need to restart vCloud Director after the Sysprep files are copied.

So there you go...simple if you read the manuals properly in the first place :)

VMworld 2013 - some thoughts...

I had the very good fortune of attending the VMworld 2013 conference in Barcelona in October [for free too, courtesy of one of out IT suppliers :-)] and so thought I'd post a few thoughts and impressions gathered from the conference whilst I still remember them fresh(ish).

I had previously been to one other VMworld, Cannes in 2009, and had been very impressed with the conference and the general quality of the break out sessions and so was looking forward to this conference immensely especially given some of the new technologies which had been revealed during the US event a couple of months prior such as, vSphere 5.5, vFRC and the awesome looking VSAN.

The venue, having now moved to Barcelona, was new but the quality of the event was still top notch!
The break-out sessions are the real reason to go to these conferences and they did not disappoint one bit. Close to the start of the event it seemed that many of the sessions I wanted to attend were fully booked up. At first I was annoyed with this but soon realised that just going to the session and waiting outside before it started pretty much guaranteed you a place in the room anyway (although probably at the back) and I ended up not missing a single session all week.
My favourite sessions were on VSAN, flash caching and some of the new cloud automation suites that VMware are now doing. Flash, btw, was everywhere at this event.  If you were in any doubt about how things are progressing with flash technology, you were left in no doubt at this event that flash is going to be EVERYWHERE pretty soon (if it's not made it into your datacentre already).

VMworld had released a mobile app for your smartphone where you could register for sessions and plan your days activities and this was really useful to have, especially when trying to navigate around the enormous conference suite. they had provided maps, social feeds and even an interactive game in the app. This was a really good improvement from the last VMworld I'd been to and even though there were large screens displaying all of the session info almost everywhere you looked, it was so handy to have when you were sitting in a quiet spot in the 'hang-space' and trying to plan where to go to later that day.

I remember being impressed by the Labs at the 2009 conference and again I really liked the accessibility and ease of which you can get first hand experience of so many of the new tech coming out from VMware.  This was a popular part of the conference, especially on the first day, but later in the event it was fairly easy to get a desk and get onto any lab that you wanted.
They had even provided BYOD lab areas where you would use your own laptop to connect to the lab environment which I thought was a great idea (except that I'd only brought my old Android tablet out with me which wasn't really up for the challenge).

The solutions exchange was where all of the vendors pitched up to show off their wares and this had all of the usual suspects that you would expect.  One very noticeable exception though was Symantec.  I had hoped they would be attending (like they had in 2009) as we use Symantec backup products I had a few things I wanted to discuss around vSphere backups and virtual machine AV protection.  From what I gathered this was probably a political withdrawal due to some support issues with their backup products being a little late to the vSphere 5.1 support party (by nearly a year) and probably didn't want to be on the end of too much public bashing where the people who really felt these issues would likely be.
Having said that,  I read recently of how Symantec are offering support for vSphere 5.5 and future releases within 90 days of GA.  This is a great response to the problem and if they keep it up, they will surely keep vSphere backup customers and gain new ones too! 90 days is a very acceptable time frame by which you would start to think about deploying an upgrade to the GA of a new mission critical infrastructure platform such as vSphere.

Some of the solutions exchange highlights I saw this year were these (in no particular order):

• Tintri - VM aware storage promising great performance at a price point that makes a lot of sense to seriously question your next SAN upgrade.
• NetApp Flash Accel integration with VSC 5.0 - This is something which I am currently looking to deploy into production and probably the subjet of my next blog post too!  A great product (which is free to existing NetApp customers) and now fully integrated into the vSphere web client.  Looked very slick and adds to the already excellent VSC product too.
• Flashsoft - Flash caching for physical and virtual environments.  Reasonable price and even though the vendor is SANdisk, it works with any SSD or PCIe flash device too
• Infinio - VM caching solution which uses ESXi host RAM instead of SSD devices.  Very nice concept and again another sweet price point too (albeit with the requirement to have significant memory free in each ESXi host which is not that typical in my experience)

There were many great products and demos and I've certainly missed out loads of good ones.  These were just some that I was particularly impressed with and liked what they were doing. 

As I said earlier, flash and storage caching solutions were everywhere in the solutions exchange and this is a space where there will be a huge change to how we are mostly all doing our virtual deployments at present.  It's getting cheaper and the solutions are getting smarter too.  Always a good combination!

Failed to open (The parent virtual disk has been modified since the child was created)

Error:

• Failed to open (The parent virtual disk has been modified since the child was created).

This error came up the other day one a couple of our virtual machines when we tried to power them on after they dies over a weekend.

This issue is in fact covered extremely well by the following KB article here,

and I would highly recommend that you read through the article and get to grips with how the various files fit together which make up the virtual server and it's disks and snapshots etc. as it will help no ends when trying to fix this or similar issues.

Now it turns out that this issue was being caused by our backup software trying to take a weekly tape copy of some virtual machines whilst at the same time a NetApp SnapMirror for Virtual Infrastructure (SMVI) backup and replication job was trying to run.

The two snapshot commands seem to have overlapped and whilst one was being deleted the other was trying to create a new snapshot and so the disk descriptor files were pointing to different snapshot delta files and referencing the wrong parent ID (This all makes more sense when you read the KB article, trust me!).

I'm not to sure why this is allowed to occur but this has now happened around 5 times in our environment over weekends to different vms and as such we have had to be more selective about when we schedule the tape backups to avoid the regular NetApp snapshots (We only do both as we do not hold long disk retention policies offsite and so require tape backups to supplement our disk backup strategy for long term backup retention...a pain, but just the way it is at present. 

To fix this issue the article recommends connecting to the host and manually opening, reading and possibly editing these files using VI but that is not too easy when you are trying to compare multiple files and cross-reference IDs and parent IDs on potentially 3, 4, 5 or more disk descriptor files depending on the number of snapshots and disks the vm has.

My approach is to follow the steps below and use free 3rd party tools to make things easier on yourself too.

Process:

1. Enable SSH on the ESXi host and open the hosts file wall port for SSH server if not already allowed (do this through vCenter for ease!)
2. Connect to the ESXi host using WinSCP – This is much easier than going through the command line or vMA service as detailed in the KB
3. Copy the following files to your local machine to identify the issue:
1. Virtualserver.log – use this to identify which disk and which snapshot file is reporting the issue
2. Virtualserver.vmx – use this to identify which snapshots are currently identified as in use
3. Virtualserver.vmdk – this is the base disk descriptor file containing the first parent CID
4. Virtualserver-00001.vmdk – this will be the first snapshot delta disk descriptor file and should have the base disks CID as its parent (there may be more than one snapshot file per disk such as 00002.vmdk and/or 00003.vmdk etc. which should all reference the preceding snapshot as their parent until they eventually lead back to the base disks CID)
4. Use NotePad++ or similar to view all of the files (This utility is excellent for formatting these files into a more readable state and also maintains the files formatting when modifying which you are likely to have to do!)
5. Make a copy of the files unedited on your machine in case the resolution doesn't work (IMPORTANT!!!)
6. Make the required changes to the disk descriptor files or the vmx file as required in order to resolve using the information in the KB article. For reference if the snapshot delta file does not contain any data (16mb or less for example) then it may be best to just edit this out of the vmx file and point to an earlier snapshot or the base disk itself in order to bring the vm back online again.
7. Copy the edited file(s) back to the original location and overwrite as needed using WinSCP
8. Power on the VM and cross those fingers! :)
9. If all is good then be sure to delete any unused snapshot descriptor, delta and check point files from the virtual servers directory so as not to affect any future snapshots and to keep things clean.

This is a good and fairly straightforward resolution to the issue. Key to getting this right though is understanding how the descriptor files work and mapping out (often on a piece of paper if needs be) the relationship between each base disk and the snapshot(s) before making any changes.  As mentioned, keep a copy of these files as you may be able to revert any changes made in error just by replacing these files.  Ideally though if you are not certain, always ensure that you have a full backup of all of the files (especially the flat files) before making any changes as per best practices!

Good luck.

RemoteApp international keyboard layouts

Whilst publishing some RemoteApps to a disperse group of users internationally we needed to be able to support more than the default (well, default to most of the English speaking world anyway) QWERTY keyboard layout.

Initially I was thinking that this could be a big deal and my first thoughts were that this would involve creating custom profile setups for different groups with the correct language and keyboard layout defined but, as it turned out, this was not the case and this was simply a case of installing the additional keyboard layouts using good old 'Region and Language' option within control panel and then going to the 'Keyboards and Languages' tab and adding the additional layout required.

After that, in the 'Text Services and Input Languages' window, go to 'Advanced Key Settings' and verify or change the option to toggle between the two keyboard layouts.  The default is to use Left Alt+Shift but this can be changed to something else if you wish.

Once set, when you launch the RemoteApp you can now toggle the keyboard layout by simply pressing the toggle command you set and away you go!  Simple....as IT should be! :)

Missing 'Unknown Media Changer' from device manager

We recently performed an expansion to one of our tape libraries which was a Quantum Scalar i500 unit.  These are very flexible units which range from 2 to 18 tape drives depending on configuration.

We we expanding this unit from 4 to 10 LTO4 drives which meant that we had to add two new drives to the existing enclosure and then bolt on a 9u expansion unit to house the other 4 drives.

All of this went well and after running all of the library tests and coming up good we were ready to re-present this library back to the backup servers to be re-incorporated to our backup policies.

It was here that we ran into a problem which I suspect is quite common in this situation.

When we brought our backup server online again the server could now see 10 LTO4 drives, but could no longer see the 'unknown media changer' which appears within device manager representing the library configuration.

After being sent on a bit of a goose chase by a Quantum article stating that this is caused by the wrong HBA driver installed on our fibre HBA (The article states that this is the case when a storport driver is used instead of a scsiport or fcport driver and to check with the HBA vendor for the correct driver to resolve the issue) it turns out that the issue was a lot simpler to fix (isn't it always the case?).

On the library configuration portal there is an option within the 'Setup' menu called 'Control Path'. This had nothing configured when we checked our library after the upgrade and as this sets up one of the tape drives to present all of the library configuration through to the connected server this was obviously our issue.

To resolve this we simply selected one of the drives to be the control path and applied the setting.  A quick re-scan of the devices within the media server and up popped our library and all of the correct slots etc.

The reason why this setting was no longer there was worrying to me though so I did some digging and it turns out that the way that the drive numbers are allocated can change when you add additional drives to the library, even if the original drives are not moved from their current slots.

Also, keep in mind that if you have more than one partition on your library that you will need to set the control path for each partition within the library.

PS: once you get the media changer presented to your server be sure to install the correct driver for the changer as this will then name the media changer with the make/model of the library etc.

Windows file server migration to NetApp CIFS

Whilst migrating several windows file servers (various OSes from Windows 2000 to Windows 2008 R2) to a new NetApp CIFS environment I encountered a small but show stopping issue which caused a lot of head scratching for a while until the solution became clear.

The migration approach taken was to create a hidden root share on the NetApp CIFS volume and then on each file server run a sync job between them and a subfolder beneath the root share on the NetApp.  The tool used for the copying was called Super Flexible File Synchronizer (now called Syncovery I believe) which not only copies all of the data with NTFS permissions but also copies shares and share permissions across too and sets them up on the NetApp...nice and simple!

Once all of the data had been copied and a few subsequent incremental syncs completed we were ready to cut over from the windows servers to the NetApp appliance for all of our file shares.  The process for this is quite simple really.

Firstly, we had to shutdown the file servers being migrated and then change the DNS entries for each server to point to the CIFS IP address of the NetApp.  As the NetApp would be hosting several different file servers we also needed to set up netbios aliases too so that the controller would respond to the client requests as the original file servers (this saves re-mapping drives and even dfs links which was a real time saver!).
The commands for this were:

1. options cifs.netbios_aliases (name of server)
2. cifs nbalias load
3. options cifs.gpo.enable on
4. cifs gpupdate

In our case the list of server names was quite long and thus needed to be inputted like this as the line is not cumulative and subsequent additions would otherwise overwrite the previous entries:

          options cifs.netbios_aliases server1,server2,server3,server4,....etc

Next the gotcha.
After doing the above I then went to access a file share which had been migrated and instead of seeing all of the shares I had a blank windows explorer. no error, but no shares either.
It turns out that the last step to do is to actually remove the old file servers AD computer object and allow replication to occur.
Once the computer object was deleted and the client machine rebooted I was then able to connect to the file shares fine and also the dfs links too.

I'm not exactly sure why the computer object needs to be removed but I guess it has something to do with the NetApp being joined to the AD domain (for CIFS).  If I get a more specific reason for this I'll update this blog in the future...

Unable to start vApp - Error: Invalid network in property vami.ip0.VM_1.

There was a requirement to restart our vCenter Operations Manager vApp recently which normally would have been a fairly straightforward process (Log into each VM and initiate a shutdown of the OS and then shut the vApp down). This time though there was an issue when I came to power the vApp back on again as I was greeted by the following error:

(Invalid network 255.255.0.0 in property vami.netmask0.VM_1.)

...and this one too:

(Invalid network in property vami.ip0.VM_1.)

Now this was not something which I had seen before and threw me for a while before finally figuring it out.

The vApp is assigned it's IP settings from an IP Pool associated with the datacenter, in this case both vms receive IP, Netmask, Gateway and DNS settings from this pool. When checking this in more detail I found that the network which was associated with this IP Pool was incorrect.
What had happened was that we had migrated the vApps network from a standard vSwitch for a Distributed vSwitch a few months ago.  The port groups used in the old standard vSwitch was named slightly differently than the new port group on the VDS even though they were the same VLAN ID. Was this meant was that when the vApp tried to power on again it was still looking for the port group from the old vSwitch and as such could not find it and could therefore not power on again.

To resolve this there was two simple settings to change:

First the IP Pool needed to be associated with the correct network again.
To do this, simply go to the datacenter in the vSphere client and then select the IP Pools tab. Then right click the Pool and select properties and then go to the 'Associations' tab and place a tick in the associated network for the IP Pool.

Second, the vApp itself needed to be updated to use the correct network for each of its IP settings.

Select the vApp from the Hosts and Clusters view and then in the summary tab, select 'Edit Settings'. Select 'Advanced' from the left menu and then select the 'Properties' box to reveal the 'Advanced Property Configuration' window as below.

Next just select each entry in turn and select 'Edit' to change the network to the correct value.

Once these settings were applied, then vApp could be started and it's IPs were once again allocated to each vm and all was well!

Simple error and all entirely self inflicted!  Just be aware of vApps and their associated networks as these settings are not changed when you change the individual vms network settings.

Storage vMotion Error: The method is disabled by 'SYMC-FULL dd-mm-yyyy...'

I had this error come up the other day whilst trying to SvMotion one of our vms over to a new storage array:

Now this is actually one of those obvious and helpful error messages that you get every now and then and just by looking at the error message I could see what had caused this issue.

We use Backup Exec with the avvi agent to perform backups of some of our production vms.

The avvi agent allows us to perform SAN to Tape backups off host which means we don't need to do anything special with regards to backup configuration on any of our ESXi hosts.  The configuration process is a simple of case enable the option within the Backup Exec media servers and present the ESX Datastores to them (with the same LUN IDs etc) and that's pretty much it. Most of our vms are also running the Backup Exec Remote Agent for it's OS (Windows or Linux) which then allows us to have granular file recovery from our image based backups which is a nice feature...although not as useful when doing your backups to tape and not disk as the recovery process still needs to extract the full vmdk off of the tape before recovering the individual files to be restored to the vm or elsewhere.

A good guide for setting this configuration up can be found on Symantec's website here:

How to backup ESX 4.0-5.0 using SAN Transport Mode with Backup Exec

Now what usually happens when a backup job is run on a vm using this method is this:

·         The BE job starts on the media server and talks to vCenter to take a snapshot of the vms vmdk

·         Once completed the vm is now running from the snapshot and the original vmdk is static and only read by the vm  

·         BE then gets the ESXi host and guest virtual machine information from vCenter it needs to backup

·         BE then opens a connection with the ESXi server to ask for the virtual machine metadata

·         BE then informs vCenter to disable Storage vMotion for that VM to ensure that the backups can complete successfully.

·         Using vStorage APIs, Backup Exec then opens a direct data connection to the ‘unknown’ SAN volumes which have been presented to it and the virtual machine data is offloaded directly to the media server for backup

·         Once the backup process has completed the snapshot is deleted and BE disconnects from the ESXi host and informs vCenter to enable Storage vMotion again for the vm

·         Backup job then completes.

The error above is caused by the Storage vMotion being disabled by Backup Exec to run the backups.  After the backup job completes the call to vCenter does not get made or fails and so the vm is stuck with it's Storage vMotion disabled.

The trouble with this is that you often don't know this is an issue until you go to perform a Storage vMotion or unless you have vms inside an SDRS cluster and they fail to migrate to other datastores.

You can however identify these vms though by performing a lookup within the vCenter database as described in this VMware KB article:

Storage vMotion fails with the error: The method is disabled by 'SYMC-INCR dd-mm-yyyy hh:mm'

Luckily this is a known issue and there are two very easy ways to address this if you have this issue.  

The first, and often easiest way, is to shutdown the vm and remove it from the inventory.  Then browse thedatastore where it resides, locate the vmx file and add it to the inventory again.

This approach basically gives the vm a new id within vcenter and thus gets any customised settings removed allowing it to SvMotion again.

This does pose an issue however in that you will need downtime on your vm, although very short, in order to resolve this.

The other approach, as detailed by VMware in the KB above, is to manually edit the settings within the vCenter DB for the vm affected.  Whilst this does not require a vm outage to work, it does require vCenter to be stopped whilst you access the DB and in some instances (Environments with vCloud Director, SRM, LabManager etc) this is more impacting than 1 vm being shutdown for a couple of mins and so finding a quiet evening or weekend to shut the vm down is my preferred approach and this can be easily scripted anyway to save those long hours from building up!

This is not restricted to Symantec btw.  I have seen this issue with VEEAM backup software also and as yet I'm not aware of any definitive solution to prevent this from happening from time to time. It pays to keep an eye on this if you are running a similar backup technology in your environment.

Hot Add CPU to a Windows 2008 R2 Enterprise VM

I'm still often caught out by just how cool modern IT really is.

Today for example we had an issue with one of our businesses systems which pretty much took it out of the water.  It seems that the business had deployed a large number of additional users to this system the previous day and by now they were all starting to access the server and place significant demand on it's single vCPU.

For about an hour or more the server had been sitting there at 100% CPU (with the odd blip here and there as it churned its way through all of the work) and for new users the system was so unresponsive that it was effectively dead to them.  The few users who were already on the system and working before this CPU increase were still OK, although they were feeling the effects of the slow performance, and were still inputting data and working on their models etc.

This posed a problem.  On the one hand it was clear that this system needed to be running on more than 1 CPU now given the increased workload but on the other we didn't want to take the server out of the water (even for just a few minutes that it would take) to do this as it would mean cutting access to the users who were in and potentially causing them issues with unfinished modelling and part entered data.

Luckily enough this VM had been deployed as Windows Server 2008 R2 Enterprise.  This one decision, taken by some possibly overzealous engineer who originally built the server, saved the day as it allowed us to simple edit the VMs properties and increase the number of vCPUs allocated from 1 to 2.

Now the really cool part was watching the server via the console with task manager opened at the performance tab.  The changes were completed within vCenter and then within around 5 seconds, the OS popped up a message to state that the data was incorrect and it needed to restart task manager. Clicking OK restarted it and up popped two CPU graphs and almost instantly the CPU levels started to drop to around 55-75% utilization.

I had done this in the lab many times and new this was easily technically possible if ever we needed to do it but we had never had this occur within the few years or so that this has been possible to do. To actually use this feature in 'live' and against a running production system (of some significant importance to the business too) gave you a real sense of satisfaction.
It's great when IT just works.

Kudos to VMware and Microsoft for some pretty neat tech!

p2v'd Windows 2000 vm with high cpu

We recently acquired some old Windows 2000 physical servers which needed to be brought online in our virtual infrastructure.  It's been a while since I p2v'd any servers as we had pretty much sucked up all of the old physical servers we had since we deploying our first virtual environment and so it was time to dust down my old copied of VMware converter, a trusted utility which had been used almost flawlessly back in those early virtual days.

In fact I chose to download a newer version of this tool from the VMware website and then burn it to a CD to make a bootable disk from which to perform a cold clone of these old servers.

The cloning process was pretty standard as before with a few nice extras now afforded to me in the post migration steps making things even easier.  The newly cloned vm came up and then it was a simple process of going through all of the old hardware applications (HP management agents and the like) and removing them, along with uninstalling the old 'hidden' hardware from the servers previous physical state.

The server then came online and with a few tweeks here and there was ready to go into production again.

It was then that we noticed some issues around the performance of the vm which seemed familiar.  Within the vSphere client the CPU performance of the windows 2000 server was showing as practically 100% almost all of the time, yet within the OS the CPU was showing as idle.

I immediately thought that the HAL was not set correctly as this is a well documented issue, especially with windows 2000 vms, however when I went into device manager, and under computer, the HAL was indeed set to what it should be; 'ACPI Multiprocessor PC'.  This was the same as on the other Win2k vms which had been migrated at the same time and they were not displaying this same CPU issue.

After looking into the issue a little more it seems that the idle thread (which  normally lets the computer save power when not in use) gets stuck in a busy loop and so although the OS believes it is not actually utilizing the CPU, the physical CPU is constantly receiving commands and therefore the CPU demand for the vm is actually 100%. 

This issue was resolved in this case by changing the HAL from 'ACPI Multiprocessor PC' to 'Advanced Configuration and Power Interface (ACPI) PC'.

Select 'ACPI Multiprocessor PC' and right clicking to Properties. Select Update Driver and then in the window which appears, select 'Show all hardware of this device class' to list available drivers as below:

Select the Advanced Configuration and Power Interface (ACPI) PC model and click Next etc to install. 

This process required a reboot of course to complete and I honestly can't stress enough how important it is to backup the vm before making these changes. As a belt and braces approach we cloned the vm first and applied the changes to the clone and then monitored it for a couple of days before applying the change to the live server.

Once the system rebooted the performance was what we expected again as per the other vms migrated at the same time.

I'm not sure how this driver differs from the previous one but swapping the driver made all the difference to this servers performance and as its not going to be around too long anyway I'm happy to leave it this way until the application is moved to a new environment entirely.

Update: The above issue can also sometimes be resolved by re-installing the ACPI Multiprocessor PC driver.  Simply follow the steps above but instead of selecting the Advanced Configuration and Power Interface (ACPI) PC drive, just re-select the same driver, complete the install and then reboot the system.

This is less risky than changing the driver but if this does not work then you should look to change to the advanced driver next.

Setting Storage Path alerts

Since vSphere 4.0 there has been a large increase in the available alarms which not only come pre-configured but are also available to be created and you can pretty much now create an alarm for almost anything within vCenter.

It still surprises me though why some quite essential monitoring areas are not included within the default set of pre-configured alarms.  One such alarm is the Storage Path Redundancy alarm will let you know when you have lost paths to your SAN storage and what datastores this will be affecting etc.  This is a very simple alarm to setup but also pretty essential to virtually all vSphere implementations these days I'd imagine.

To set up the alarm select the vCenter server in the vSphere client and then go to the 'Alarms' tab.
Select 'Definitions' to see a list of all currently configured alarms and then right click in the section to create a new alarm.
Give the alarm a name ('Degraded Storage Paths' for example) and change the Monitor to 'Hosts' and then choose 'Monitor for specific events occurring on this object, for example, VM powered On'.
On the 'Triggers' tab click 'Add' and then change the Event type to 'Degraded Storage Path Redundancy'.
Next select the 'Actions' tab and Add an action to be performed when this event occurs.  This can either be an email alert perhaps to the storage team or even a task for the ESXi host to perform.
Once set, click 'OK' and the alarm is set.

It's also worth creating another alarm to go along with this once which alerts when one of the ports goes offline too.  That way you get notifications of path redundancy lost or a full port connectivity loss which will help in troubleshooting the issue being experienced.

To set this up, simply create another rule as above but this time set the trigger to be 'Lost Storage Path Redundancy' and set whatever actions you would like.

There are many other good alarms to set depending on what monitoring solutions you may or may not have in place for your virtual environment so its always good to have a look through the list of available alarms and just check that you have everything you need configured before you need it....they're not going to do that much if you've created them after the event!

vSphere vmotion network outages

During heavy vmotion operations I was experiencing intermittent network outages of ESXi hosts and some vms running within the cluster.
This seemed to get progressively worse over a period of several weeks until it became almost every time a host was placed into maintenance mode there would be some network outage of some vms and even other ESXi hosts within the same cluster.

After initially looking at the network infrastructure we noticed that there was a large flood of unicast traffic on the vlan which was being shared by vmotion, and some windows based vms, around the time of the vmotions (to be expected in vmotion operations).

Now VMware best practice is to have vmotion and ESXi Management on their own separate vlans or networks but this had never been an issue previously with this cluster which was about 4 years old and had been upgraded over that time from ESX 3.5 to ESXi 5.0 u1 (it's current state). There had been no significant network changes during this period also which could have had a waggling finger pointed at them so it was not obvious how we had come to this issue.
It seemed obvious to start thinking that the gradual changes and growth of the cluster had started to cause this issue for us.  Over the various versions which these hosts have been running the vmotion feature has been greatly enhanced and improved and the amount of simultaneous vmotions a host can support has also increased from 2 to 4 (or 8 with 10Gbe) as can be seen here:

(taken from the vSphere 5.1 Documentation center here)

Network Limits for Migration with vMotion
Operation	ESX/ESXi Version	Network Type	Maximum Cost
vMotion	3.x	1GigE and 10GigE	2
vMotion	4.0	1GigE and 10GigE	2
vMotion	4.1, 5.0	1GigE	4
vMotion	4.1, 5.0, 5.1	10GigE	8

There had also been a sizable growth in the number of hosts and virtual machines in this cluster and the hosts had been increased in capacity etc along the way too.  This all resulted in a much heavier demand for vmotion during the process of placing a host into maintenance mode as often I would be looking at somewhere between 20-50 virtual machines being migrated across the cluster.

It turned out that this was in fact our issue and so as we had spare capacity within our hosts, due to the recent removal of some iSCSI connections to this cluster, we were able to hook up a couple of dedicated vmotion nics per host and placed them into their own vlan away from the management and any other systems.
vSphere 5.0 gives us the ability to utilize more than 1 vmotion nic per host. All that was needed was to create 2 new vmkernel ports on a new vSwitch and have vmkernel port 1 bound to vmnicX as active and vmnicY as standby, then just reverse the configuration for the second vmkernel port.
Once the two new vmotion ports were created and assigned IPs on the new vlan, I just removed the old vmotion port which was in the shared vlan and then that was all of the configuration which was needed.

I performed a few test migrations after that and the performance improvement was easily visible even without measuring it.  We used to have windows vms with 4GB ram move between hosts within 1-2 mins and now we are getting them within 30 seconds.

Best of all, when now entering maintenance mode on a host, even one which is running many vms, we are no longer getting any network outages and the process is a lot speedier too. Happy times again!

Free VMware SRM training videos

VMware have just released a set of free (yup, totally free!) training videos for Site Recovery Manager (SRM) on their website:

http://blogs.vmware.com/education/2012/09/free-site-recovery-manager-training.html

This is a great resource for those wishing to deploy SRM and I would urge all to take a look through the videos before starting your deployments.

Migrating VMs running on VSS to VDS in a Production ESX cluster (Migrate Virtual Machine Networking…)

With one of our ESXi 5.0 clusters growing to 12 hosts and our Networking team constantly wanting to deploy new vlans like they are going out of fashion it was time to implement a distributed vSwitch (vDS) to the cluster in order to reduce the administrative overhead of adding all of the port groups to each vSwitch on each host (not really the case but it’s always good to keep up with the professional dogging of those poor network guys eh?). 

The process to deploy a new vDS to a cluster is pretty straight forward and you can follow the process from within vCenter here: VMware KB

Once created the next step was to create each of the vlan port groups onto the VDS.  Here I simply setup the vlan with the same name as used currently on each of the VSS (you can do this as the name has ‘(dvswitch)’ appended to it anyway so it keeps these separate from the existing port groups when selecting network connectivity when editing VMs) and the same vlan ID entry etc.

Next I moved 2 of the 4 x 1Gb adaptors from each VSS into the dvuplink ports on the VDS.  This then allowed the existing VSS port groups to continue to service network requests for all of the running VMs and also allowed me to start moving VMs from the VSS to VDS .

To migrate the VMs from their current port groups on each VSS to the newly created port groups on the VDS you can use the excellent Migrate Virtual Machine Networking utility which manages the bulk modifications to VMs.

To do this simply so to the networking screen in the VI client and right click on the new VDS and choose ‘Migrate Virtual Machine Networking…’

Next select the source network from the drop down list (this is the current port group that you want to move VMs off of) and then select the destination network (the corresponding port group on the dvSwitch)

Click Next and you can now select all or some of the VMs to be migrated.  If you select all of the VMs you’ll be able to sit back and watch as each VM is modified in turn and moved over.  It really is simple and best of all results in no network outage to the running VMs.

I migrated several hundred VMs across our various vlans to the distributed switch without one little blip! 

Then it was a just a matter of going through each of the hosts and cleaning up the old port groups and vSwitches which were no longer being used.

vSphere VM deployment customizations

A small but annoying thing had started to happen to your deployments of Windows 2008 R2 vms in our production environment recently. Whenever we deployed a new vm and used our pre-saved customization specification the vm would be deployed as expected except that it did not join the new vm to our production domain.
The image would be customised, the server name changed, IP settings applied, administrator password set etc but it would no longer join the vm to our windows domain.

Alarmingly, although the option was set within the specification there were no errors recorded for this in the logs on the newly deployed vm (these can be found at c:\windows\temp\vmware-imc\guestcust.log) which I would have expected.

The answer it turned out was very simple.  The customization had been modified to have domain\username in the username field of the domain customization properties.  Although this looks perfectly reasonable to have in a windows environment this actually needs to be just the username of the domain account which will be joining the vm to the domain.

After changing the pre-saved customization to just the account name and re-entering the password I fired off a test deployment and voilà, 1 windows vm deployed and sitting on our production domain as before!

Virtual Machine disk consolidation fails with I/O error on change tracking file

A vm was displaying the warning that 'Virtual machine disks consolidation is needed' which is a nice feature of vSphere 5 which now actively tells you about this issue (It's always been there in previous releases but never highlighted in this way until 5.0).

We often get this issue as we use a snapshot backup technology to backup our vms each day and for some reason or other sometimes the remove snapshot process does not complete properly and we get this situation where the snapshots are removed but the snapshot files are still present and referenced in the vm. See the following VMware kb article for details.Consolidating snapshots in vSphere 5

Usually this is a simple process of right clicking the vm, selecting 'snapshot > consolidate' to have the snapshot child disk files consolidated back to the parent disk file but in this case the consolidation failed with the error message: 'A general system error occurred: I/O error accessing change tracking file'.

After some investigation I found that our backup system had a lock on one of the files and so I was able to release the file from the backup software and then re-run the consolidation which completed and all was good again!
The troubleshooting steps to identify the locked file can be found here: Investigating virtual machine file locks on ESX/ESXi

Previously I've also been able to resolve the issue of not being able to consolidate vm disks by creating a clone of the troubled vm and bringing it up as the active vm and then deleting the old one. Not always possible though in a production environment!