How to Fix Log Disk Exhaustion on vCenter (VCSA) | 2 Methods

Alert: Log Disk Exhaustion on vCenter

Have you ever seen a alert like “Log Disk Exhaustion on [vCenter name]”, or received “Log Disk Exhaustion on vcsa” after logging into the vSphere Client?

These prompts all seem to indicate a disk space error. And the thing is, the default disk for storing logs in a VCSA deployment is 10GB, which usually triggers an alert that the log disk exhaustion when it reaches or exceeds 78%.

Especially in older vCenter Servers, the log disks are more likely to fill up because the rotation of vCenter logs does not work properly sometimes.

Knowing the general cause, how to investigate the disk usage further and resolve vCenter log disk exhaustion issue?

How to fix log disk exhaustion on vCenter 7 (VCSA)

The indication of this error message is obvious, and there are 2 methods we can use to fix vCenter /storage/log full:

💬1. Delete obsolete logs that take up space

💬2. Extend the virtual disk of log partition

You can choose your method according to the actual situation, but before that, let's investigate the disk usage specifically (requires SSH access to VCSA). 

*The following tutorial is based on vCenter 7. You can also fix log disk exhaustion on vCenter 6.7 with similar steps.

How to inspect the disk exhaustion issue

To learn where the disk space is being used, log into the vCenter Server Appliance as root via SSH or the vCenter virtual machine Console.

Run this command to enable the Bash shell:

Type this and press Enter:

Run this command and it will return the space usage of each storage volume or disk.  

*Here's an alternative command which only shows you the partitions which are getting full:

As the disk exhaustion message shows, the /storage/log partition is the one exceeding the default threshold. Next, you can use one of the following solutions to fix the vCenter /storage/log full error.

Solution 1. Delete obsolete logs to fix disk exhaustion on vCenter

When disk expansion is not an option, you can resolve this log disk exhaustion on VCSA error by deleting obsolete logs. To do this, you need to find where the storage is being used specifically under the /storage/log partition, run this command to identify top 20 files with high disk usage:

Then go into any suspiciously large directory with this command (replace the directory name):

And look for large files using the commands:

📢Warning: Before deleting any files from the vCenter Appliance, ensure you have a good backup of vCenter.

Here are some practical examples of removing log files:

🔹Directories for Apache Tomcat service log files:

• /storage/log/vmware/sso/tomcat/
• /storage/log/vmware/eam/web/
• /storage/log/vmware/lookupsvc/tomcat/ (vCenter 7.0 only)

If there are a lot of these, you may check and delete them to free up space:

🔹Directories for PostgreSQL service health alarm log files:

• /storage/log/vmware/sso/tomcat/
• /storage/log/vmware/eam/web/
• /storage/log/vmware/lookupsvc/tomcat/ (vCenter 7.0 only)

Run these commands to check and remove large log files:

🔹Check and remove excessive Storage Profile Service access logs

After removing some large log files, you should be able to reclaim some space back. Then, run df -h to check whether the disk usage has been significantly reduced. If the log disk exhaustion on vCenter issue persists, you may need to increase the size of the /storage/log virtual disk.

Solution 2. Extend the log disk to fix the exhaustion issue on vCenter

It’s possible to fix this problem by increasing the size of the relevant disk. To do this, first you need to identify the exact virtual disk to extend.

In vCenter Server Appliance Management Interface (VAMI)​​​​​​, please go to Monitor > Disks.

As you can see, the counterpart to log partition is Hard disk 5. This is the virtual disk you should enlarge.

📢Note that you cannot change disk size on the vCenter VM if there is a snapshot of the VM.

Right click vCenter Appliance, choose Edit Settings to increase the relevant disk.

Then, go back to shell and run this command to extend the disk:

It checks all partitions of the device and extends them to 100% of the disk size. Then you can run df -h to check the system again, and see whether the log disk extension issue is resolved now.

Addition: Backup vCenter VMs in case of any accidents

vCenter Server Appliance contains a file-based backup and restore function to help you recover your environment after a failure. This is an effective protection measure before you perform some major operations on the VCSA.

However, this backup method does not protect the virtual machines managed by vCenter. If you want to back up your VMs on a regular basis to avoid data-loss and restore the VMs to a usable state (or even to another place), you can use a dedicated VMware backup solution.

Here I will use a reliable VMware backup software - AOMEI Cyber Backup as an example. It allows unattended automated backup with email notification, and the following benefits:

Backup in batch: Easily add a standalone ESXi host or vCenter Server to back up virtual machines managed by it.
Easy-to-use: Centralized and streamlined web console to help you achieve any operation with simple clicks.
Agentless: Create complete and independent image-level backup for VMs.
Cloud Archive: Save backup versions to an AWS S3 storage for long-time storage.
Automation: Set schedules to automate VM backup with email notification, and enable retention policy to auto delete old backup files.
Out-of-place recovery: Besides in-place recovery, you can restore a VM to new location in the same or another datastore, host, or vCenter for VM cloning/migration

AOMEI Cyber Backup works with VMware ESXi 6.0 and above versions (free ESXi is also supported). Click the button below to enjoy a fully-functional free trial:

*You can choose to install this VM backup software on either Windows or Linux system.

Conclusion

This article discusses two measures you can take when the Log Disk Exhaustion on vCenter 7 or VCSA, i.e., deleting large log files to free up space, or extending virtual disk of log partition.

Note that before you proceed (especially before deleting any files, it’s recommended to make a backup of vCenter. This way, even if something goes wrong, you can quickly restore the environment for use. In addition to this, it’s also a wise decision to use a reliable VMware backup solution for virtual machine protection.