Problem:
We have had reports of customers experiencing the following issues when running with RHEL 7.4 or CentOS 7.4, or earlier 7.x versions that have updated NFS packages:
-
Zeneventserver times out and/or crashes
-
Deletion of files takes a very long time from the /opt/serviced/var/volumes/<instanced id>/zeneventserver/index directory
- Upgrade of serviced fails
Remediation:
Edit to nfsmount.conf (Defaultvers=4.0)
While we are still investigating the root cause, there may be a defect with file locking when using NFS 4.1 with RHEL or CentOS 7.x. Customers using RHEL/CentOS 7.x should force the use of NFS to 4.0, as we've seen positive results doing this to affected users. To do this, follow these steps:
- Edit /etc/nfsmount.conf on each host
- Find the following line:
# Defaultvers=4 - Uncomment it and change it so that it looks like:
Defaultvers=4.0 - Save that file
- Stop Resource Manager and serviced on all of their hosts where NFS is mounted
- On each host, issue: systemctl restart nfs-server
- Bring serviced back up and start resource manager
- Once the nfs shares are remounted, verify that NFS mounts are using NFS version 4.0 by running: nfsstat -m on the delegates in the RM pool
[root@delegate1 ~]# nfsstat -m
/opt/serviced/var/volumes/afwdlj8lpc6n6fk3rojbfq7lt from 10.10.123.123:/serviced_volumes_v2/afwdlj8lpc6n6fk3rojbfq7lt
Flags:
rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.10.123.124,local_lock=none,addr=10.10.123.123
Edit to lvm.conf (units="h")
RHEL/CentOS 7.4 changes the LVM configuration (in /etc/lvm/lvm.conf) to use units="r", which causes LVM commands to output sizes as human-(r)eadable with '<' rounding indicator. This rounding indicator causes a parsing error in the serviced pre-upgrade script.
The fix for this is straightforward:
- Edit /etc/lvm/lvm.conf on each host
- Find the following line:
units="r" - Change it to read:
units="h" - Save that file
- Find the following line:
- (NOTE: you do not have to restart LVM to make these changes take effect)
- To confirm that LVM is now using the correct units
- run the command lvmconfig | grep units on each host:
[root@delegate1 ~]# lvmconfig | grep units
units="h" - run the pvs, vgs, and lvs commands on each host, and confirm you do not see any rounding symbols (such as <) in the sizes shown
- run the command lvmconfig | grep units on each host:
How about specifying the --units argument for lvm commands in the pre-upgrade script instead of requiring everyone to change their lvm config?