Applies To
- Zenoss Resource Manager 5.x
- CentOS 7.2
Summary
Upgrading to CentOS 7.2 from CentOS 7.1 results in storage issues for Zenoss Resource Manager, including running out of storage space. When fstrim is run on an LVM thin volume it fails with the following log message:
fstrim: /mnt: the discard operation is not supported
This means that instead of performing the expected fstrim procedure and reducing the data usage reported by the lvs command, the storage use increases and can fill up the disk.
Prior to upgrading the kernel, fstrim seems to run very quickly with a message that it trimmed a specific amount of data, for example:
31.6 GiB (33955295232 bytes) trimmed
The message is erroneous however because the LVS command returns the same amount of storage consumed both before and after running fstrim. After successful completion of the following procedure, the fstrim process runs correctly.
Additional information is available at the CentOS bug site.
Procedure:
Perform the following to update the kernel and address the fstrim issue:
- Log into the Zenoss host as root, or a user with root authority.
- Stop Zenoss Resource Manager:
serviced service stop zenoss.resmgr
- Issue the following:
serviced service status | grep -iv stopped
Watch the output from the command and wait until all are stopped.
- Issue the following command on all hosts except the master:
systemctl stop serviced
- Issue the following command on all hosts:
mount | grep -i volumes
- Issue the following command on all hosts it shows being mounted on:
umount /opt/serviced/var/volumes/<tenant id>
- On the master, issue the following:
systemctl stop serviced yum -y update kernel
- Reboot master and serviced should come back up.
- On all other hosts, issue:
systemctl start serviced
- Start Resource Manager:
serviced service start zenoss.resgmr
Access the Control Center UI. Wait for all Resource Manager services to come up (green).
- Confirm there are no snapshots:
serviced snapshot list -t
Delete any snapshots:
serviced snapshot rm
- Run the lvs command:
lvs
Consult the output for errors and storage use.
- Run the fstrim command:
fstrim -b /opt/serviced/var/volumes/<tenant id>
- Run the lvs command:
lvs
Consult the output for errors and verify the storage is correct.
- Confirm that Resource Manager is still all green in the Control Center UI.
Comments