vCloud Director Troubleshooting.

Repost : http://www.virtualizationexpress.com/2013/12/vcloud-director-troubleshooting-html/

I started working with vCloud director from its initial release (vCloud 1.0) and I was thinking to write article about vCloud director troubleshooting for very long time but finally vCloud director(vCD) is going to die oops..I.e it is getting better and better , have a look at below articles

http://longwhiteclouds.com/2013/12/07/vmware-vcloud-director-is-dead-long-live-vcloud-director/

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2057589

still it’s not late to write, as per the VMware KB2057589 support for vCD 5.5 is available till Q3 2017 and vCD 5.1 till Sep 2014.

What is in this post ?

Troubleshooting , resolving vCloud director issues are complex and tricky, there are no straight forward method to resolve those issues (For Ex:Go ahead just “Restart the service”, “Reboot the server” won’t work with vCloud director) . It needs more  patience and calm approach. To understand the behavior, it is important to understand all the components / services in the vCloud. I tried explaining my way of  approach, below are the topics I covered which I feel it is essential to read.

• vCD components
• vCD Cell Architecture
• Log Components

• vCD Troubleshooting Checklist

• vCD Issues

vCD & its components

You must understand what is vCloud director, what it can do, how it can be used to implement IaaS (Infrastructure as a Service) , there are numerous blog-post and references explaining what is vCloud director (vCD), some of them which I think it is important to read  are listed below.

http://www.yellow-bricks.com/2010/08/31/vmware-vcloud-director-vcd/

http://www.vmware.com/products/vcloud-director/

vCloud Training :http://vmwarelearning.com/y79/vcloud-director-concepts-and-architecture/

http://www.virtualizationexpress.com/2013/04/vcloud-director-vshield-training-vmware.html/

I hope the above links will make clear about vCD, I would like to cover troubleshooting part of vCD, simply I listed the software components that makes vCD which is required for troubleshooting.

Components:

These are the collective software components which makes vCloud

• vCloud director
• vCloud Networking and Security (Formerly vShield Manager)
• vCenter Server
• ESX(i)
• vCenter Chargeback manager (optional)

vCD Cell Architecture:

To begin with vCD troubleshooting, one must understand its Architecture and components. I have summarized the component architecture and its connection.Readers should keep this architecture diagram in their mind when they approach an issue, for example if we are not able to view the vCloud director UI, it should strike that JAVA OSGI should be completely UP to access the UI, likewise the below diagram explains the internal connectivity and its relationship.I have listed only the required components for troubleshooting, to know more about the vCD architechture review the below VMware KB.http://kb.vmware.com/kb/1030954

Internal Components of vCD:

• vCloud Cell is a Supported Linux Server with vCloud director software installed.
• Transfer Service is involved when a user in vCloud director uploads a OVF / Media to their allocated resources
• Console Proxy is proxy service for VMRC- Virtual Machine Remote Console, this component re-directs the VMRC from the ESXi host to vCloud director UI.
• VC Proxy / Cell Listener service is agent which sends / receives updates from the vCenter server, like power on/off VM, Create VM…Etc…
• Active MQ– Message Bus is communication bus between multiple vCloud cells, it is for synchronization.–

vCD log components:

If you encounter any issues in vCD you must review the logs  to find the cause and resolve the issue.Login to your vCD cell via SSH (Often Putty) and navigate to vCD installation directory i.e /opt/vmware/vcloud-director/Note: $VCLOUD_HOME is the environment variable for the vCloud director installation folder, use this to go to the location.Example cd $VCLOUD_HOME/logsThese are the logs that will be useful in reviewing the vCD issues.

Required services for vCloud director

• vCenter services
• VMware SSO services (Including all dependent services) (Applies on vCenter 5.1 and above)
• VMware vSphere Profile-Driven storage Service  (Applies on vCenter 5.1 and above)
• vShield Manager
• vCloud services (vmware-vcd)

vCD Troubleshooting Checklist:

These are the common items must take into consideration before we begin troubleshooting

vCD Components:

• vCenter and its DB
• vCloud and its DB
• vShield Manager

Check List:

• Time Sync– Check if the time is synchronized of all vCD components
• Name Resolution– Check if the vCD components are resolved by its FQDN
• Valid SSL certificate on all vCD components
• Connectivity between all vCD components
  • Perform Ping, telnet, netcat test to validate
• Synchronization– Check if the synchronization of all vCD components
  • In vCloud director –> System tab –> vSphere resources –> vCenter, see if there is tick mark in the Status bar
  • In vCloud director –> System tab –> Cloud Resources –> Cells, check if the vCenter proxy is running
  • In vShield manager –> Configuration — >vCenter, Check the last inventory sync, it should contain the latest time stamp

vCD Issues:

Below listed issues are most common issues I have seen in vCloud Director.

• Service Start-up Issue
• VMRC Issues
• Ovf Upload issue
• Networking issues ..ahem
• Resource allocation issues
• Multi Cell and Load balancer
• Guest Customization
• Connectivity issues
  • vCenter
  • vShield

PS: I have published only the vCD startup issue, I am working on others and will publish shortly.