vCenter Appliance upgrade failed

RePost from my blog: http://www.virtualizationexpress.com/2013/12/vcenter-appliance-upgrade-failed/

I was trying to upgrade vCenter Server Appliance 5.0 to vCSA 5.1, there is no inplace upgrade since

vCSA 5.0 uses IBM DB2 as a DB and

vCSA 5.1 use VMware vPostgres DB ,

so the only available option is migrate the DB

Steps for Upgrade

Keep the vCSA 5.0 Powered on –> Deploy new vCSA 5.1 –> Enable the Trust between both Appliance –> Start Importing.

Detailed steps / videos can be found on below links.

• http://kb.vmware.com/kb/2033990
• http://kb.vmware.com/kb/2031331
• http://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.upgrade.doc%2FGUID-6A5C596D-103E-4024-9353-5569263EB427.html
• http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.upgrade.doc%2FGUID-6A5C596D-103E-4024-9353-5569263EB427.html

Error that I faced while upgrading was with the iNodes on the new appliance, there were no free iNodes left so that Upgrade failed with error “General system error occurred”

While reviewing the upgrade log /var/log/vmware/vami/upgrade.log there are more than 1 million (10 Lakhs) records on VPX_EVENT_ARG, all the vC event descriptions are stored in this table and it was the cause to fill-up the iNodes .

~ # df -ih  Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda3 632K 632K 0 100% / devtmpfs 0 0 0 - /dev

iNodes are nothing but Index-node to represent the filesystem. for each file it takes each iNode for its ease of accessibility. since it has 1 million records it failed to Index all the files. I was thinking of increasing the iNodes but to increase iNodes it is a requirement to re-create the filesystem, so I approached alternative plan to delete the contents from VPX_EVENT_ARG table.

It was difficult for me to find the commands to truncate / delete the DB. I used below commands to delete the contents from VPX_EVENT_ARG table.

1. Backup and Snapshot the vCenter Server appliance 5.0

1. Start a SSH session to the vCenter Server Appliance. When you are prompted to log in, use the user root and the password:
   1. User: root
   2. Password: vmware (default)

1. To query the DB2 database(s), perform one of these options:

Use isql to query the DB2 database.Note: Make sure you are at the root prompt/user.

• Retrieve the password for the vCenter Server database user with the command:
  • grep EMB_DB_PASSWORD /etc/vmware-vpx/embedded_db.cfg
• Using the retrieved password string, run this command to connect via isql:
  • VCSA-50:~ # isql -v 'VMware VirtualCenter' vc password
• When connected, you see a SQL prompt that appears similar to SQL>. You can now query the DB2 database from the prompt.
• Run the below command to count the number of rows from VPX_EVENT_ARG table
  • VCSA-50:~#select count(*) from VPX_EVENT_ARG
  • VCSA-50:~# Output similar to 1000054
• Run the below command to delete the contents from VPX_EVENT_ARG table
  • VCSA-50:~#delete from VPX_EVENT_ARG

Note: Sometimes the Transaction log will be full and the delete commands gets terminated, in these scenario you can delete 100000 rows at a time to avoid transactional logs fill-up

• VCSA-50:~#delete from (select * from VPX_EVENT_ARG fetch first 100000 rows only)

Note: Use all CAPS for the SQL quries if it don’t work