How To Use Zenoss Enterprise Backup (ZEB) to Back Up the Resource Manager Databases

Applies to

Zenoss Resource Manager version 4.2.4 with SP 336 or later, or Resource Manager version 4.2.5 or later
ZenDS version 5.5.36 or later

Summary

By default, the zenbackup command creates a logical backup (sometimes called a database dump) of your Resource Manager databases. If Resource Manager is running during a logical backup, there is a risk that changes made to the database during the backup may create inconsistencies in the backup file that won't be discovered until the backup is restored.

Unlike a logical backup, Zenoss Enterprise Backup (ZEB) creates a physical backup of the underlying database files. ZEB is able to take a “hot” backup of InnoDB tables and a “warm” backup of MyISAM and other non-InnoDB tables. This allows Zenoss to remain online while a high-integrity backup is performed. See http://dev.mysql.com/doc/mysql-enterprise-backup/3.10/en/meb-backup-overview.html for a detailed explanation of the different types of backups.

Zenoss Resource Manager maintains three separate databases served by ZenDS: the events database (zenoss_zep), the object database (zodb), and the UI session database (zodb_session). In most deployments all three are contained in a single ZenDS instance. All of the example commands in this article back up the entire set of databases served by the local ZenDS instance. If the event and object databases are served by separate instances of ZenDS, you will need to back up each instance individually.

The zenbackup --enterprise-tools option instructs zenbackup to back up the Resource Manager databases using ZEB rather than the default logical database dump. This article introduces invoking ZEB explicitly for administrators who want finer control over how their Resource Manager databases are backed up. If you would prefer to invoke ZEB through zenbackup, please refer to How do I Use zenbackup with Zenoss Enterprise Backup (ZEB) to Back Up the Resource Manager Databases?

Notes:

Zenoss recommends that you schedule a daily backup of the Resource Manager databases. These can all be full backups; or, for example, a full backup on one day of the week, and incremental backups on the others. Schedule your backups on the database server using cron for a time when Resource Manager is usually not busy.

In order to run the zeb command from cron, you must wrap it in a shell script as shown in the example below, and invoke the shell script from the appropriate crontab file:

#!/bin/sh

export USE_ZENDS=1
source /etc/profile.d/zends_env.sh

# Run full or incremental backup
zeb …

# Back up non-database data
zenbackup --no-eventsdb --no-zodb

Zenoss strongly recommends that you test your backup and recovery on a test platform before you face an actual crisis. Your test platform should replicate the scale of your production systems to the greatest degree practical and any backup procedure only be adopted for production after you have validated it through full scale testing.
The Enterprise Backup User’s Guide is available at http://dev.mysql.com/doc/mysql-enterprise-backup/3.10/en/index.html
By default, every invocation of zeb writes a log file under the backup directory to a meta/MEB_DateTimeStamp_operation.log file. For example:
```
/mnt/backups/zenoss/full/2014-01-02_03-04-05/meta/MEB_2014-01-02.03-04-05_backup_apply_log.log
```

Procedures

In the following command examples, substitute path names that are appropriate for your environment for the example path names (shown in green, bold, and italic type: /a/b/c).

Determine the Host Name of Your Database Server

To determine the host name of your ZenDS server, examine the value of the zodb-host option in $ZENHOME/etc/global.conf on the Zenoss master server. One way to do this is:
$ awk '/^zodb-host/ { print $2 }' $ZENHOME/etc/global.conf

Determine Your Resource Manager and ZenDS Version Levels

Run the following command as zenoss on the Resource Manager host (sometimes referred to as the Zenoss master server):
$ zenup status
Product: zenoss-resmgr-4.2.5 (id = zenoss-resmgr-4.2.5)
Home: /opt/zenoss
Revision: 80
Updated On: Mon Apr 28 15:33:51 2014
Consult the output and verify the following:
- The product (zenoss-resmgr) version is 4.2.5 or greater
- OR, the product (zenoss-resmgr) version is 4.2.4, and the revision number is 336 or greater
Run the following command as zenoss on the ZenDS database host:
$ zends --version

/opt/zends/bin/.mysql Ver 14.14 Distrib 5.5.36, for linux2.6 (x86_64) using EditLine wrapper
Verify that the version number following “Distrib” is 5.5.36 or greater

If your Zenoss instance does not meet both of these requirements, or if the zenup command was not found, contact Zenoss support for assistance.

Prepare for the First Backup

Log into the database server as root.
Set your user mask to an appropriate value. For example:
# umask 007
Note: A user mask of 007 prevents “global” (non-owner, non-group users) read, write, and execute permissions from being set. See the chmod(1) man page for more information.
Create the directories that will hold the backup files. In the examples in this article, we save the backup files to a remote file system mounted under /mnt. (The procedures to create, export, and mount remote file systems are beyond the scope of this article.) Substitute your path names for the example path names.
# mkdir -p /mnt/backups/zenoss/{full,incremental}
Enable the /mnt/backups directory to be traversed by any user. For example:
# chmod go+x /mnt/backups
Set the owner and group of the new backup directories to zenoss. For example:
# chown -R zenoss:zenoss /mnt/backups/zenoss

Full Backup Example

On the database server as the zenoss user:

 $ zeb \
--user=root \
--no-locking \
--backup-dir=/mnt/backups/zenoss/full \
--with-timestamp \
backup-and-apply-log

Note: Because we specified that ZEB should not lock the database during the backup, changes can be made to InnoDB tables while the backup is in progress. The backup-and-apply-log command instructs ZEB to apply those changes, if any, to the backup files at the conclusion of the backup.

Incremental Backup Example

On the database server as the zenoss user:

$ zeb \
--user=root \
--no-locking \
--incremental \
--incremental-backup-dir=/mnt/backups/zenoss/incremental \
--incremental-base=history:last_backup \
--with-timestamp \
backup

Back Up Other (non-database) Resources

When using Enterprise Backup to back up your Resource Manager databases, you must continue to back up the important data that is stored outside of ZenDS. Use “zenbackup --no-eventsdb --no-zodb” for this purpose. Zenoss recommends that you run this command via cron on the master server in close coordination with your zeb database backups. Note: This step is critical when Service Impact is installed.

Restore Example

Before restoring the database, you must apply any incremental backups made since the most recent full backup.
On the database server as the zenoss user: For each incremental backup performed since the full backup and in chronological order (oldest first), apply the incremental backup to your full backup. This process can be performed while ZenDS is running and does not affect the active database. For example:
```
$ zeb \
--user=root \
--backup-dir=/mnt/backups/zenoss/full/TIMESTAMP \
--incremental-backup-dir=/mnt/backups/zenoss/incremental/TIMESTAMP \
apply-incremental-backup
```

On the Zenoss master server as the zenoss user, shut down all local daemons and remote hub daemons:

$ zenoss stop
$ for h in $(dc-admin --collector-pattern=NONE list | awk -F= '!/localhost/{print $2}'); do
dc-admin --no-master --hub-pattern=$h --collector-pattern=NONE stop
done

On the database server as the zenoss user, determine the number of log files (this value is required for the --innodb_log_files_in_group option in step #6):
```
$ zends -u root -e 'select @@innodb_log_files_in_group;'
```
Shut down ZenDS:
$ zendsctl stop
Note: The error message “rm: cannot remove `/var/lock/subsys/zends': Permission denied” can safely be ignored.
Set your user mask to an appropriate value. For example:
# umask 007
Note: A user mask of 007 prevents “global” (non-owner, non-group users) read, write, and execute permissions from being set. See the chmod(1) man page for more information.
Restore the full backup. For example:
```
$ zeb \
--datadir=$ZENDSHOME/data \
--backup-dir=/mnt/backups/zenoss/full/TIMESTAMP \
--innodb_log_files_in_group=2 \
copy-back
```
Note: Use the value determined in step #3 for the innodb_log_files_in_group option, for example: --innodb_log_files_in_group=2
Restart the database:
$ zendsctl start
Drop and recreate the zodb_session database:
$ zends -u root -e 'drop database zodb_session; create database zodb_session;'
On the Zenoss master server as the zenoss user, restore the data from zenbackup:
```
zenrestore --no-eventsdb --no-zodb --file $ZENHOME/backups/DATESTAMP.tgz
```

Restart the local daemons and remote hub daemons:

$ zenoss start
$ for h in $(dc-admin --collector-pattern=NONE list | awk -F= '!/localhost/{print $2}'); do
dc-admin --no-master --hub-pattern=$h --collector-pattern=NONE start
done

Performance Recommendations

Because compression can add significant wall clock and CPU time to the process, Zenoss recommends that you create the backup without compression. The backup files can be compressed later if required.
Refer to Chapter 7. Performance Considerations for MySQL Enterprise Backup of the Enterprise Backup User’s Guide for additional tuning options.
Ensure the destination file system has enough free space to hold the backup files.