RemoteApp international keyboard layouts

Whilst publishing some RemoteApps to a disperse group of users internationally we needed to be able to support more than the default (well, default to most of the English speaking world anyway) QWERTY keyboard layout.

Initially I was thinking that this could be a big deal and my first thoughts were that this would involve creating custom profile setups for different groups with the correct language and keyboard layout defined but, as it turned out, this was not the case and this was simply a case of installing the additional keyboard layouts using good old 'Region and Language' option within control panel and then going to the 'Keyboards and Languages' tab and adding the additional layout required.

After that, in the 'Text Services and Input Languages' window, go to 'Advanced Key Settings' and verify or change the option to toggle between the two keyboard layouts.  The default is to use Left Alt+Shift but this can be changed to something else if you wish.

Once set, when you launch the RemoteApp you can now toggle the keyboard layout by simply pressing the toggle command you set and away you go!  Simple....as IT should be! :)

Missing 'Unknown Media Changer' from device manager

We recently performed an expansion to one of our tape libraries which was a Quantum Scalar i500 unit.  These are very flexible units which range from 2 to 18 tape drives depending on configuration.

We we expanding this unit from 4 to 10 LTO4 drives which meant that we had to add two new drives to the existing enclosure and then bolt on a 9u expansion unit to house the other 4 drives.

All of this went well and after running all of the library tests and coming up good we were ready to re-present this library back to the backup servers to be re-incorporated to our backup policies.

It was here that we ran into a problem which I suspect is quite common in this situation.

When we brought our backup server online again the server could now see 10 LTO4 drives, but could no longer see the 'unknown media changer' which appears within device manager representing the library configuration.

After being sent on a bit of a goose chase by a Quantum article stating that this is caused by the wrong HBA driver installed on our fibre HBA (The article states that this is the case when a storport driver is used instead of a scsiport or fcport driver and to check with the HBA vendor for the correct driver to resolve the issue) it turns out that the issue was a lot simpler to fix (isn't it always the case?).

On the library configuration portal there is an option within the 'Setup' menu called 'Control Path'. This had nothing configured when we checked our library after the upgrade and as this sets up one of the tape drives to present all of the library configuration through to the connected server this was obviously our issue.

To resolve this we simply selected one of the drives to be the control path and applied the setting.  A quick re-scan of the devices within the media server and up popped our library and all of the correct slots etc.

The reason why this setting was no longer there was worrying to me though so I did some digging and it turns out that the way that the drive numbers are allocated can change when you add additional drives to the library, even if the original drives are not moved from their current slots.

Also, keep in mind that if you have more than one partition on your library that you will need to set the control path for each partition within the library.

PS: once you get the media changer presented to your server be sure to install the correct driver for the changer as this will then name the media changer with the make/model of the library etc.

Windows file server migration to NetApp CIFS

Whilst migrating several windows file servers (various OSes from Windows 2000 to Windows 2008 R2) to a new NetApp CIFS environment I encountered a small but show stopping issue which caused a lot of head scratching for a while until the solution became clear.

The migration approach taken was to create a hidden root share on the NetApp CIFS volume and then on each file server run a sync job between them and a subfolder beneath the root share on the NetApp.  The tool used for the copying was called Super Flexible File Synchronizer (now called Syncovery I believe) which not only copies all of the data with NTFS permissions but also copies shares and share permissions across too and sets them up on the NetApp...nice and simple!

Once all of the data had been copied and a few subsequent incremental syncs completed we were ready to cut over from the windows servers to the NetApp appliance for all of our file shares.  The process for this is quite simple really.

Firstly, we had to shutdown the file servers being migrated and then change the DNS entries for each server to point to the CIFS IP address of the NetApp.  As the NetApp would be hosting several different file servers we also needed to set up netbios aliases too so that the controller would respond to the client requests as the original file servers (this saves re-mapping drives and even dfs links which was a real time saver!).
The commands for this were:

1. options cifs.netbios_aliases (name of server)
2. cifs nbalias load
3. options cifs.gpo.enable on
4. cifs gpupdate

In our case the list of server names was quite long and thus needed to be inputted like this as the line is not cumulative and subsequent additions would otherwise overwrite the previous entries:

          options cifs.netbios_aliases server1,server2,server3,server4,....etc

Next the gotcha.
After doing the above I then went to access a file share which had been migrated and instead of seeing all of the shares I had a blank windows explorer. no error, but no shares either.
It turns out that the last step to do is to actually remove the old file servers AD computer object and allow replication to occur.
Once the computer object was deleted and the client machine rebooted I was then able to connect to the file shares fine and also the dfs links too.

I'm not exactly sure why the computer object needs to be removed but I guess it has something to do with the NetApp being joined to the AD domain (for CIFS).  If I get a more specific reason for this I'll update this blog in the future...

Unable to start vApp - Error: Invalid network in property vami.ip0.VM_1.

There was a requirement to restart our vCenter Operations Manager vApp recently which normally would have been a fairly straightforward process (Log into each VM and initiate a shutdown of the OS and then shut the vApp down). This time though there was an issue when I came to power the vApp back on again as I was greeted by the following error:

(Invalid network 255.255.0.0 in property vami.netmask0.VM_1.)

...and this one too:

(Invalid network in property vami.ip0.VM_1.)

Now this was not something which I had seen before and threw me for a while before finally figuring it out.

The vApp is assigned it's IP settings from an IP Pool associated with the datacenter, in this case both vms receive IP, Netmask, Gateway and DNS settings from this pool. When checking this in more detail I found that the network which was associated with this IP Pool was incorrect.
What had happened was that we had migrated the vApps network from a standard vSwitch for a Distributed vSwitch a few months ago.  The port groups used in the old standard vSwitch was named slightly differently than the new port group on the VDS even though they were the same VLAN ID. Was this meant was that when the vApp tried to power on again it was still looking for the port group from the old vSwitch and as such could not find it and could therefore not power on again.

To resolve this there was two simple settings to change:

First the IP Pool needed to be associated with the correct network again.
To do this, simply go to the datacenter in the vSphere client and then select the IP Pools tab. Then right click the Pool and select properties and then go to the 'Associations' tab and place a tick in the associated network for the IP Pool.

Second, the vApp itself needed to be updated to use the correct network for each of its IP settings.

Select the vApp from the Hosts and Clusters view and then in the summary tab, select 'Edit Settings'. Select 'Advanced' from the left menu and then select the 'Properties' box to reveal the 'Advanced Property Configuration' window as below.

Next just select each entry in turn and select 'Edit' to change the network to the correct value.

Once these settings were applied, then vApp could be started and it's IPs were once again allocated to each vm and all was well!

Simple error and all entirely self inflicted!  Just be aware of vApps and their associated networks as these settings are not changed when you change the individual vms network settings.

Storage vMotion Error: The method is disabled by 'SYMC-FULL dd-mm-yyyy...'

I had this error come up the other day whilst trying to SvMotion one of our vms over to a new storage array:

Now this is actually one of those obvious and helpful error messages that you get every now and then and just by looking at the error message I could see what had caused this issue.

We use Backup Exec with the avvi agent to perform backups of some of our production vms.

The avvi agent allows us to perform SAN to Tape backups off host which means we don't need to do anything special with regards to backup configuration on any of our ESXi hosts.  The configuration process is a simple of case enable the option within the Backup Exec media servers and present the ESX Datastores to them (with the same LUN IDs etc) and that's pretty much it. Most of our vms are also running the Backup Exec Remote Agent for it's OS (Windows or Linux) which then allows us to have granular file recovery from our image based backups which is a nice feature...although not as useful when doing your backups to tape and not disk as the recovery process still needs to extract the full vmdk off of the tape before recovering the individual files to be restored to the vm or elsewhere.

A good guide for setting this configuration up can be found on Symantec's website here:

How to backup ESX 4.0-5.0 using SAN Transport Mode with Backup Exec

Now what usually happens when a backup job is run on a vm using this method is this:

·         The BE job starts on the media server and talks to vCenter to take a snapshot of the vms vmdk

·         Once completed the vm is now running from the snapshot and the original vmdk is static and only read by the vm  

·         BE then gets the ESXi host and guest virtual machine information from vCenter it needs to backup

·         BE then opens a connection with the ESXi server to ask for the virtual machine metadata

·         BE then informs vCenter to disable Storage vMotion for that VM to ensure that the backups can complete successfully.

·         Using vStorage APIs, Backup Exec then opens a direct data connection to the ‘unknown’ SAN volumes which have been presented to it and the virtual machine data is offloaded directly to the media server for backup

·         Once the backup process has completed the snapshot is deleted and BE disconnects from the ESXi host and informs vCenter to enable Storage vMotion again for the vm

·         Backup job then completes.

The error above is caused by the Storage vMotion being disabled by Backup Exec to run the backups.  After the backup job completes the call to vCenter does not get made or fails and so the vm is stuck with it's Storage vMotion disabled.

The trouble with this is that you often don't know this is an issue until you go to perform a Storage vMotion or unless you have vms inside an SDRS cluster and they fail to migrate to other datastores.

You can however identify these vms though by performing a lookup within the vCenter database as described in this VMware KB article:

Storage vMotion fails with the error: The method is disabled by 'SYMC-INCR dd-mm-yyyy hh:mm'

Luckily this is a known issue and there are two very easy ways to address this if you have this issue.  

The first, and often easiest way, is to shutdown the vm and remove it from the inventory.  Then browse thedatastore where it resides, locate the vmx file and add it to the inventory again.

This approach basically gives the vm a new id within vcenter and thus gets any customised settings removed allowing it to SvMotion again.

This does pose an issue however in that you will need downtime on your vm, although very short, in order to resolve this.

The other approach, as detailed by VMware in the KB above, is to manually edit the settings within the vCenter DB for the vm affected.  Whilst this does not require a vm outage to work, it does require vCenter to be stopped whilst you access the DB and in some instances (Environments with vCloud Director, SRM, LabManager etc) this is more impacting than 1 vm being shutdown for a couple of mins and so finding a quiet evening or weekend to shut the vm down is my preferred approach and this can be easily scripted anyway to save those long hours from building up!

This is not restricted to Symantec btw.  I have seen this issue with VEEAM backup software also and as yet I'm not aware of any definitive solution to prevent this from happening from time to time. It pays to keep an eye on this if you are running a similar backup technology in your environment.

Hot Add CPU to a Windows 2008 R2 Enterprise VM

I'm still often caught out by just how cool modern IT really is.

Today for example we had an issue with one of our businesses systems which pretty much took it out of the water.  It seems that the business had deployed a large number of additional users to this system the previous day and by now they were all starting to access the server and place significant demand on it's single vCPU.

For about an hour or more the server had been sitting there at 100% CPU (with the odd blip here and there as it churned its way through all of the work) and for new users the system was so unresponsive that it was effectively dead to them.  The few users who were already on the system and working before this CPU increase were still OK, although they were feeling the effects of the slow performance, and were still inputting data and working on their models etc.

This posed a problem.  On the one hand it was clear that this system needed to be running on more than 1 CPU now given the increased workload but on the other we didn't want to take the server out of the water (even for just a few minutes that it would take) to do this as it would mean cutting access to the users who were in and potentially causing them issues with unfinished modelling and part entered data.

Luckily enough this VM had been deployed as Windows Server 2008 R2 Enterprise.  This one decision, taken by some possibly overzealous engineer who originally built the server, saved the day as it allowed us to simple edit the VMs properties and increase the number of vCPUs allocated from 1 to 2.

Now the really cool part was watching the server via the console with task manager opened at the performance tab.  The changes were completed within vCenter and then within around 5 seconds, the OS popped up a message to state that the data was incorrect and it needed to restart task manager. Clicking OK restarted it and up popped two CPU graphs and almost instantly the CPU levels started to drop to around 55-75% utilization.

I had done this in the lab many times and new this was easily technically possible if ever we needed to do it but we had never had this occur within the few years or so that this has been possible to do. To actually use this feature in 'live' and against a running production system (of some significant importance to the business too) gave you a real sense of satisfaction.
It's great when IT just works.

Kudos to VMware and Microsoft for some pretty neat tech!

p2v'd Windows 2000 vm with high cpu

We recently acquired some old Windows 2000 physical servers which needed to be brought online in our virtual infrastructure.  It's been a while since I p2v'd any servers as we had pretty much sucked up all of the old physical servers we had since we deploying our first virtual environment and so it was time to dust down my old copied of VMware converter, a trusted utility which had been used almost flawlessly back in those early virtual days.

In fact I chose to download a newer version of this tool from the VMware website and then burn it to a CD to make a bootable disk from which to perform a cold clone of these old servers.

The cloning process was pretty standard as before with a few nice extras now afforded to me in the post migration steps making things even easier.  The newly cloned vm came up and then it was a simple process of going through all of the old hardware applications (HP management agents and the like) and removing them, along with uninstalling the old 'hidden' hardware from the servers previous physical state.

The server then came online and with a few tweeks here and there was ready to go into production again.

It was then that we noticed some issues around the performance of the vm which seemed familiar.  Within the vSphere client the CPU performance of the windows 2000 server was showing as practically 100% almost all of the time, yet within the OS the CPU was showing as idle.

I immediately thought that the HAL was not set correctly as this is a well documented issue, especially with windows 2000 vms, however when I went into device manager, and under computer, the HAL was indeed set to what it should be; 'ACPI Multiprocessor PC'.  This was the same as on the other Win2k vms which had been migrated at the same time and they were not displaying this same CPU issue.

After looking into the issue a little more it seems that the idle thread (which  normally lets the computer save power when not in use) gets stuck in a busy loop and so although the OS believes it is not actually utilizing the CPU, the physical CPU is constantly receiving commands and therefore the CPU demand for the vm is actually 100%. 

This issue was resolved in this case by changing the HAL from 'ACPI Multiprocessor PC' to 'Advanced Configuration and Power Interface (ACPI) PC'.

Select 'ACPI Multiprocessor PC' and right clicking to Properties. Select Update Driver and then in the window which appears, select 'Show all hardware of this device class' to list available drivers as below:

Select the Advanced Configuration and Power Interface (ACPI) PC model and click Next etc to install. 

This process required a reboot of course to complete and I honestly can't stress enough how important it is to backup the vm before making these changes. As a belt and braces approach we cloned the vm first and applied the changes to the clone and then monitored it for a couple of days before applying the change to the live server.

Once the system rebooted the performance was what we expected again as per the other vms migrated at the same time.

I'm not sure how this driver differs from the previous one but swapping the driver made all the difference to this servers performance and as its not going to be around too long anyway I'm happy to leave it this way until the application is moved to a new environment entirely.

Update: The above issue can also sometimes be resolved by re-installing the ACPI Multiprocessor PC driver.  Simply follow the steps above but instead of selecting the Advanced Configuration and Power Interface (ACPI) PC drive, just re-select the same driver, complete the install and then reboot the system.

This is less risky than changing the driver but if this does not work then you should look to change to the advanced driver next.