Hi all 
A few days ago, I finished a project at one of my customers, a medium-size one. This project was about migrating his Virtual Infrastructure in his Manufacturing Plant site from vSphere 4.0 (BN 721907) to vSphere 5.x.
Fortunately, I came up with a plan to migrate all of his Infrastructure online -although he was planning for a downtime- and it went successfully. So, why not to share it with you all ..?!
Infrastructure:
Infrastructure consists of 3 HP hosts with aggregate memory of 32GB of RAM and aggregate CPU of 6 sockets with 4 cores each. In addition, FC SAN was used as back-end storage of 2 TBs. Single cluster with HA/DRS enabled and Tier 2 applications hosted as well as single critical Document Server.
Requirements:
- Migrate all of Virtual Infrastructure from vSphere 4.0 (BN 721907) to vSphere 5.x.
- Migration to be in weekends (he thought that migration will be offline).
- Removing old vCenter Server and Build new fresh vCenter 5.x Server Windows VM.
- Document Server must be online ASAP even at the cost of all other VMs to be offline.
My Initial Plan and Decisions:
Although the environment seemed to be really small, this customer was really worried. That Document Server was really critical for him and he had a bad experience with similar migration process done in HQ.
I decided to plan for making it online during working hours for a couple of reasons:
- Environment was really small, no need for shutting down all services without need.
- Customer has Enterprise License which gave me many features to use and why not using it..??
- I worried about shutting down that Document Server and powering it up (It was legacy document server).
- No network advanced configuration used, i.e. simple Standard vSwitches without any special configuration.
- No worries about upgrading VMFS datastores used from VMFS 3 to VMFS 5 online. Any drawback in upgraded VMFS datastore -regarding preserving block size- will not affect future business.
I began to plan for migration and make conceptual migration on papers. One thing made things little easier was that he'll burn down his old vCenter which was physical machine, so no need for P2V phase. Another thing was that he used simple network configuration as mentioned before. I decided that the plan will be something like:
- Upgrading certain host and build new vCenter on it to build new DRS cluster.
- Leveraging vMotion and DRS capabilities for migrating all VMs from old host to new one and then upgrade all old hosts one by one.
- Upgrading datastores used to VMFS 5 online.
- VMs HW Level and VMware tools to be upgraded according regular Maintenance Windows.
- After total migration is finished, decompose old vCenter Server.
I chose to upgrade to vSphere 5.1 U2, as I'm really familiar to it and customer doesn't need any features of vSphere 5.5.
Plan Phases:
Phase 1: Preparation Phase:
In this phase, all software needed is downloaded and moved to the far Plant. All VMs are backed up for purposes of safety of any unknown sudden failures or circumstances.
Phase 2: vSphere 5.1 Host Deployment:
For simplicity, let's name the hosts: Host 1,2 and 3.
1-) on old vSphere 4 Cluster, I reviewed and recorded any network or storage configuration needed before beginning and there was nothing special.
2-) I started by choosing Host 1 and making it into Maintenance Mode, so all VMs are surely migrated by DRS to the other hosts.
3-) After evacuating the host, I removed Host 1 from vCenter 4 Cluster and rebooted it into vSphere 5.1 U2 installation.
4-) After installing vSphere 5.1 U2 on it, I re-configured Host 1 for networking and storage. Now, Host 1 has similar portgroups as old Host 2 and 3 and attached to the same datastores as well as vMotion is enabled.
5-) I created a new VM with hardware level 9 for the new vCenter Sever. On this VM, I installed Windows 2k8 r2, updated it and antivirus agent is installed following Customer's Policy.
6-) On that VM, I installed vCenter Server 5.1 U2 with embedded DB as there're no need for further expansions beyond its limit.
7-) After successful installation, I created new cluster with HA/DRS enabled.
8-) I re-added online all hosts (1, 2, 3) to the new Cluster (It through a false warning about removing hosts from old vCenter, but nothing to worry about ).
9-) I re-balanced the cluster using DRS to make sure that every VM was working perfectly till now and migrations of VM were smooth between these different hosts. Fortunately everything was just fine !!
Phase 3: Upgrading the Remaining Hosts:
1-) I make Host 2 it into Maintenance Mode, so all VMs are surely migrated by DRS to the other hosts.
2-) After evacuating the host, I removed Host 2 from vCenter 5.1 Cluster and rebooted it into vSphere 5.1 U2 installation.
3-) After installing vSphere 5.1 U2 on it, I re-configured Host 2 for networking and storage. Now, Host 1 has similar portgroups as old Host 2 and 3 and attached to the same datastores as well as vMotion is enabled.
4-) I re-added Host 2 to vCenter 5.1 and the new cluster.
5-) I re-balanced the cluster using DRS to make sure that every VM was working perfectly till now and migrations of VM were smooth.
6-) I repeated steps from 1-5 again with Host 3.
7-) I made sure that all Virtual infrastructure were properly licensed with the new license.
Phase 4: Storage Upgrade:
I upgraded the two datastores online from VMFS 3 to VMFS 5 by just one click. I made sure that everything till now was working fine.
Now the only thing remaining was to upgrade VMware tools and VM HW Level of VMs. Customer states that he'd do it regularly during his Maintenance Windows.
It took me only 2 days to finish that project and customer was above the clouds .
Subsidiary Notes:
1-) In case of using Distributed Switches version 4, I managed to test that in my home lab and I found that the best way is to create temp Standard vSwitches, add VMs to them to make vMotion really easy and then migrate all VMs netowkring to a new Distributed Switch 5.1. The reason is that, I discovered that vMotion operation can be done if only VMKernel ports of source and destination hosts on the same LAN segment. VMKernel portgroups don't affect vMotion, i.e. if they're differently named, it doesn't prevent vMotion. Unfortunately, vMotion requires VM portgroups to be identical on the source and destination hosts, hence Distributed switch can't be used in that migration plan, as you can't create a host to two Distributed Switches, then create two portgroups on these distributed switches with the same name on the same host. For more information read the following article by Chris Wahl:
This may introduce some limited downtime.
2-) In case you need to know the difference between upgraded VMFS 5 datastores and newly-created ones, refer to the following article by Jason Boche:
http://www.boche.net/blog/index.php/2011/07/21/vmfs-5-vmfs-3-whats-the-deal/
If you want to re-create your VMFS datastores on VMFS 5, you should have at least two shared datastores and Enterprise License. Use Storage vMotion to move VMs from one datatsore to the other one, re-format the empty datastore then re-create it using vCenter 5.1 or ESXi Hosts 5.1. Repeate that for the other datastores till you finish. Also, keep in mind SCSI Reservation Issue -in case no VAAI available- that VMs on a single datastore should not be high to cause SCSI Reservation Conflicts (Usually 10-15 VMs per datatsore).
Waiting for your feedback .
Share the Knowledge .