Had a couple of VMware vSphere ESXi 5.5 hosts stop responding in Virtual Center today. In the VC they were showing as not responding, if you tried to go to the host on https then you would get the error :
503 Service Unavailable
To troubleshoot this I enabled SSH from the DCI and then opened a putty session to the host. Once SSH’d to the server I restarted the service management agents.
/sbin/services.sh restartAfter restarting them I checked the status of the hostd service
/etc/init.d/hostd statusThis came back with a status of not running. I started the service up using the command
/etc/init.d/hostd startHowever shortly after it started it stopped again. I then looked at the VMkernel log file.
tail /var/log/vmkernel.log
From the log file I could see that it was complaining the ramdisk tmp location was full.
2014-05-30T17:01:01.536Z cpu29:8332408)WARNING: VisorFSRam: 305: Cannot extend visorfs file /tmp/auto-backup.8332400/etc/shadow because its ramdisk (tmp) is full.To see if the tmp file location is full the command below can be run:
vdf -h
If we go into this directory we can then see some log files have filled up the drive.
After the logs were removed and the service management agents restarted again the server came back to life.
It would seem that the problem is caused by the scratch path not being specified on the host. To configure this location use the VMware KB below :-
VMware KB 1033696 : Creating a persistent scratch location for ESXi 4.x and 5.x