Resource Manager HBase and ZooKeeper services fail due to incorrect permissions

Applies To

• Zenoss Resource Manager 5.1.2
• Zenoss Resource Manager 5.1.3

Summary

Changes to file permissions cause Resource Manager manager issues. Resulting errors include HBase and Zookeeper services failing to start and DFS running out of disk space.

If the file ownership and permissions for the directory
/var/lib/zookeeper/version-2 are incorrect, the services fail. The file permissions for the file must be 755.

The file permission error appears in the Zookeeper log, for example:

...
2016-06-13 11:17:50,501 [myid:1] ERROR [main:QuorumPeer@453] Unable to load database on disk
java.io.FileNotFoundException: /var/lib/zookeeper/version2/ currentEpoch.tmp (Permission denied)
  ...

Procedure

To reset the file permissions for /var/lib/zookeeper/version-2, perform the following:

1. Log into the Resource Manager host as root or a user with root authority
2. Issue the following commands, depending on whether this is done on each Zookeeper instance or from the Master host:
• For each Zookeeper instance:
  1. Attach to the container as root:

     serviced service attach zookeeper

     NOTE: If there is more than one instance of Zookeeper, the instance number must be specified, for example:

     serviced service attach zookeeper/0

  2. Change the permissions to 755:

     sudo chmod 755 /var/lib/zookeeper/version2

  3. Change the ownership to zookeeper:

     sudo chown zookeeper:zookeeper /var/lib/zookeeper/version2

  4. Exit the container:

     exit

• Performed from the master host:
  1. Change the permissions to 755:

     sudo chmod 755 /opt/serviced/var/volumes/*/hbasezookeeper*

  2. Change the ownership:

     sudo chown 102:105 /opt/serviced/var/volumes/*/hbasezookeeper*