How To Use Zenoss Enterprise Backup (ZEB) to Back Up the Zenoss Analytics Databases

Applies to

Zenoss Analytics, version 4.3 or later
ZenDS, version 5.6.16 or later

Summary

Administrators who want to minimize interruptions to Zenoss Analytics’ reporting availability, or those with a very large data warehouse can use Zenoss Enterprise Backup (ZEB) as an alternative to performing a logical backup (sometimes called a database dump) of the Analytics databases. If Analytics is running during a logical backup, there is a risk that changes made to the database during the backup may create inconsistencies in the backup file that won't be discovered until the backup is restored.

Unlike a logical backup, Zenoss Enterprise Backup (ZEB) creates a physical backup of the underlying database files. ZEB is able to take a “hot” backup of InnoDB tables and a “warm” backup of MyISAM and other non-InnoDB tables. This allows Analytics to remain online while a high-integrity backup is performed. See http://dev.mysql.com/doc/mysql-enterprise-backup/3.10/en/meb-backup-overview.html for a detailed explanation of the different types of backups.

Zenoss Analytics maintains two distinct databases: the reporting database, which contains the data warehouse; and the zenoss_analytics database, which contains report resource definitions and other data required by the Analytics application. Both databases can be served by a single instance of ZenDS, or they can each have their own dedicated ZenDS instance. In the later case, you will need to back up each instance individually.

Notes:

Zenoss recommends performing a regular incremental backup of both the Analytics application database and the Analytics data warehouse. Schedule these using cron for a time when Analytics is usually not busy. In addition, we recommend that you schedule a weekly full (non-incremental) backup of both databases.
In order to run the zeb command from cron, you must wrap it in a shell script like the example below, and invoke the shell script from the appropriate crontab file:
```
#!/bin/sh
export USE_ZENDS=1
source /etc/profile.d/zends_env.sh
zeb …
```
Zenoss strongly recommends that you test your backup and recovery on a test platform before you face an actual crisis. Your test platform should replicate the scale of your production systems to the greatest degree practical and any backup procedure only be adopted for production after it is validated through full scale testing.
The Enterprise Backup User’s Guide is available from http://dev.mysql.com/doc/mysql-enterprise-backup/3.10/en/index.html
By default, every invocation of zeb writes a log file under the backup directory to a meta/MEB_DateTimeStamp_operation.log file. For example:
```
/mnt/backups/analytics/full/2014-01-02_03-04-05/meta/MEB_2014-01-02.03-04-05_backup_apply_log.log
```

Procedures

In the following command examples, substitute path names that are appropriate for your environment for the example path names (shown in green, bold, and italic type: /a/b/c).

Prepare for the First Backup

Log into the Analytics application server as root.
Set your user mask to an appropriate value. For example:
# umask 007
Note: A user mask of 007 prevents “global” (non-owner, non-group users) read, write, and execute permissions from being set. See the chmod(1) man page for more information.
Create the directories that will hold the backup files. In the examples in this article, we save the backup files to a remote file system mounted under /mnt. (The procedures to create, export, and mount remote file systems are beyond the scope of this article.) Substitute your path names for the example path names.
```
# mkdir -p /mnt/backups/analytics/{full,incremental}
```
Enable the /mnt/backups directory to be traversed by any user. For example:
```
# chmod go+x /mnt/backups
```
Set the owner and group of the new backup directories to zenoss. For example:
```
# chown -R zenoss:zenoss /mnt/backups/
```

Full Backup Example

On the Analytics server as the zenoss user:

$ zeb \
--user=root \
--no-locking \
--backup-dir=/mnt/backups/analytics/full \
--with-timestamp \
backup-and-apply-log

Notes:

Because we specified that ZEB should not lock the database during the backup, changes can be made to InnoDB tables while the backup is in progress. The backup-and-apply-log command instructs ZEB to apply those changes, if any, to the backup files at the conclusion of the backup.
If your Zenoss Analytics installation employs a separate server for the reporting data warehouse, run this command on the data warehouse server.

Incremental Backup Example

On the database server as the zenoss user:

$ zeb \
--user=root \
--no-locking \
--incremental \
--incremental-backup-dir=/mnt/backups/analytics/incremental \
--incremental-base=history:last_backup \
--with-timestamp \
backup

Note: If your Zenoss Analytics installation uses a separate database server for the reporting data warehouse, repeat these steps on the data warehouse server.

Restore from Backup

Log into the database server as root.
Switch to the zenoss user:
# su - zenoss
Before restoring the database, you must apply any incremental backups made since the full backup.
For each incremental backup performed since the full backup, in chronological order (oldest first), apply the incremental backup to your full backup. For example:
```
zeb \
--user=root \
--backup-dir=/mnt/backups/analytics/full/TIMESTAMP \
--incremental-backup-dir=/mnt/backups/analytics/incremental/TIMESTAMP \
apply-incremental-backup
```
Determine the number of InnoDB log files (required for the zeb copy-back command):
```
$ zends -u root -e 'select @@innodb_log_files_in_group;'
```
As root, shut down the Zenoss Analytics and ZenDS services:

# service zenoss_analytics stop Note: applies to application server only

# service zends stop Note: applies to both application and data warehouse servers
Switch to the zenoss user:
# su - zenoss

Restore the backup image. For example:

$ zeb \
--datadir=/opt/zends/data \
--backup-dir=/mnt/backups/analytics/full/TIMESTAMP \
--innodb_log_files_in_group=2 \
copy-back

Note: Use the values determined in step #4 for the innodb_log_files_in_group options.

As root, restart the ZenDS and Zenoss Analytics services:

# service zends start Note: applies to both application and data warehouse servers

# service zenoss_analytics start Note: applies to application server only

Back Up Other (non-database) Resources

Back up the following non-database file system paths on the Analytics application server as part of your Analytics backup strategy. In general it is sufficient to create an archive of each listed directory with tar and use scp to move it to another server:

/opt/zenoss_analytics: This directory contains the analytics software, configuration information, and temporary working files. In particular, the work subdirectory contains temporary files that have completed extract and are in a "staging" state that have not yet been loaded by the analytics server to the data warehouse. Make sure to exclude the directory containing the database backups if you have already copied those to another server as part of the the database backup automation.
/etc/zenoss_analytics: This directory contains the Analytics configuration file.

Performance Recommendations

Because compression can add significant wall clock and CPU time to the process, Zenoss recommends that you create the backup without compression. The backup files can be compressed later if required.
Refer to Chapter 7. Performance Considerations for MySQL Enterprise Backup of the Enterprise Backup User’s Guide for additional tuning options.
Ensure the destination partition has enough free space.

Database Backup Automation Recommendations

Automate backups using cron.
Perform, at a minimum, weekly full backups of both the Analytics application and data warehouse databases.
Copy completed backups of a particular server off that server as soon as possible.
If your Analytics application and data warehouse are on separate servers, plan to make only full backups of the Analytics application server because the application databases are small enough to make this practical nightly.
If the size of your Analytics data warehouse is very large, nightly full backups might be impractical. Consider weekly full backups with daily incremental backups in between.
It is not necessary to retain old backups as long as you keep a full backup until the next one is created because you can restore to a consistent state. Automate purging of backups from your backup directories as necessary, based on available disk space.