Follow

How to Backup Zenoss (5.1 and above) With serviced backup

Applies To

  • Zenoss 5.1 and greater

Summary

It is important to have accurate and tested system backups because they can mitigate problems caused by software or hardware issues. One method to backup Control Center and its installed applications is to use serviced backup. Backups can be performed via the command line or through automated cron jobs (crontab).

The serviced backup command saves a backup of the current system, including the state of all services, and data to a compressed archive file (.tgz). Because serviced backup leverages functionality provided by serviced snapshot, backups can be performed without the need to shutdown zenoss or the docker containers because they are only momentarily suspended to enable reading the data.

NOTE: It is important that there is enough free space to receive and store backups because running low on available disk space will result in errors and impact system performance.

About serviced backup

The default directory that receives backups from serviced backup is /opt/serviced/var/backups.

The serviced backup command includes various options to tailor the command for specific use:

USAGE
serviced backup [command options] [arguments...]
DESCRIPTION
serviced backup DIRPATH
OPTIONS
--exclude '--exclude option --exclude option' 

Subdirectory of the tenant volume to exclude from backup
 
--help, -h
Shows the help for an option

Procedures

Backups

Backups can be initiated from the UI or the command line. Zenoss stores its backups by default in the directory /opt/serviced/var/backups.

Best practices for backup include:

  • Weekly backups for production environments.
  • Increasing backup frequency to daily during initial deployment or for development environments. 

Backup Using Control Center

To backup the system through the Control Center UI, navigate to Control Center > Backup/Restore and click the Create Backup button.

If a backup is run through the UI, the path is controlled by an entry in the /etc/default/serviced file:

# SERVICED_BACKUPS_PATH=/opt/serviced/var/backups

Change this entry to specify a new target directory for the UI initiated backups.

Backup Using the Command Line

To backup your system through the command line:

  • As root or a valid Control Center user, backup the system to the default directory /opt/serviced/var/backups with the following command:
    serviced backup /opt/serviced/var/backups

    A successful backup is indicated by the system displaying the new backup file name, for example:

    backup-2016-09-29-163201.tgz

Automating Backup with CentOS/RHEL 7

For automating system backup on CentOS/RHEL 7, crontab alone is not used, instead, bash scripts are used to automate cron, fstrim, zenossdbpack etc. Zenoss is configured with various bash scripts for this purpose.

For examples of how to create appropriate automation scripts, see the files in  /etc/cron.daily/, /etc/ cron.hourly/ and /etc/cron.weekly/ in the current installation.

Note: If automation is used to perform backups, care must be exercised to ensure accumulation of old backups does not consume all disk space. Old backups must be curated to maintain available storage space.

Example serviced-backup File

As an example of a script using serviced backup, add a file, called serviced-backup in this example to the appropriate directory with the following contents:

#!/bin/sh
# Execute a Control Center backup, writing the results to the backups directory
${SERVICED:=/opt/serviced/bin/serviced} backup /opt/serviced/var/backups
 
Was this article helpful?
4 out of 4 found this helpful

Comments

  • Avatar
    Cisco TAC

    What would make this article more useful is presentation of how to "rotate" the backups. E.g., if you are taking weekly backups, your backup storage area will eventually fill up, which will prevent new backups from occurring. If the customer is not manually managing their backup storage on a weekly basis, they could end up getting a bad surprise when it is time to do a restore (i.e., they could find that all their backups are old). Can you provide a script (that could be part of the serviced-backup script) to cull the oldest backup if disk space is low? Thanks!

  • Avatar
    Adam McCurdy

    Hey there,

    I've submitted a request to our KB team to get this article updated to reflect that. Thanks for the feedback!

  • Avatar
    Tim Caiazza

    I'm assuming to restore the backup we would want to do

    serviced restore backupfile.tgz

    is that correct?

  • Avatar
    Cisco TAC

    You will want to provide the directory path, not the file name, like this:

    serviced backup /opt/serviced/var/backups

    If you type "serviced backup" alone at the CLI, it will return usage info, as follows:

    # serviced backup
    Incorrect Usage.

    NAME:
    backup - Dump all templates and services to a tgz file

    USAGE:
    command backup [command options] [arguments...]

    DESCRIPTION:
    serviced backup DIRPATH

    OPTIONS:
    --exclude '--exclude option --exclude option' Subdirectory of the tenant volume to exclude from backup

  • Avatar
    Tim Caiazza

    Thanks for the quick response, unfortunately we are still running into some issues. We are taking the backup from 1 machine and attempting to restore on another, to simulate a DR event. On the new control center machine, should all the applications be wiped so there are no applications installed before we attempt to restore the backup?

    Also since we are trying to restore to a new machine, the IPs will be different. Is the best way to update the all the end points by updating them in the command center before starting the application?

  • Avatar
    Cisco TAC

    Yes, you will want to restore to a "bare" Control Center: i.e., one that does not already have the Resource Manager application deployed. Depending on the version of Control Center, you may need a Host already added; on older versions of CC, the restore process will bring the old Host with it. If the latter, you may find you need to correct a mismatch of the "hostid" by deleting and adding back the Host in CC.

    To fix the IP issues, I think you will only need to check on & re-do the "IP Assignments" under the RM application. If this is a multi-host install, there may be additional work to fix the Virtual IP of the collector pools.

    I am more familiar with the Cisco UCSPM version of this product, but I think everything I said above will apply as well to RM.

    Regards,

    Eric Thirolle

  • Avatar
    Tim Caiazza

    Thanks again. We got this working, by doing what you said. Start with a bare control center and then restore.

    One slightly annoying thing is that the backup and restore does seem to keep any updates to the /opt/zenoss/Products/ZenEvents/SyslogProcessing.py file. But that we can copy over after the restore.

  • Avatar
    Tony Thayer

    It should be noted that on RHEL7 systems, /etc/cron.(hourly|daily|weekly|monthly) only works if cronie is enabled and in use. Standard crontab should be used instead if you aren't using cronie.

  • Avatar
    Manish Panwar **SUSPENDED**

    Hey,
    I would recommend that since it is automated backup, there should be a mechanism that after 3 or 5 consecutive backups, the script should remove a previous backup file to remove dated backups to create more space for new ones.
    This should first check that the new backup created has a size greater than or equal to immediately old backup to avoid incomplete backups to remove old ones.

    Can someone help with any such script or procedure.

  • Avatar
    Cisco TAC

    +1 on this request from Manish for a "logrotate"-type script for the backup files. The most recent backup file will be the most important, so I think the need to save the latest backup file should probably override the need to keep older backups by default.

  • Avatar
    Darryl Dunkin

    Add this line to the script above to keep up to 7 days of backups only:
    find /opt/serviced/var/backups -mtime +7 -delete

    However, if your backups fail to run for 7 days, you won't have any backups, so be sure you are monitoring the results of your backup job at the same time.

  • Avatar
    Manish Panwar **SUSPENDED**

    Hi Everyone,
    In my environment i am using the following 2 scripts which run daily. One creates backup and one removes the oldest one.

    However, it is missing a minimum count check so if your backup is not happening for some reason, it will still remove the oldest backup and it might delete all backups made till yet.

    1. Create a backup script in cron.daily with following contents:
    [root@n2vl-pa-ccmz01 ccm_user]# cat /etc/cron.daily/zenoss_backup
    #!/bin/sh
    # Execute a Control Center backup, writing the results to the backups directory
    ${SERVICED:=/opt/serviced/bin/serviced} backup /opt/serviced/var/backups

    2. Create a script in /root/delete_oldest_backup.sh with following contents:

    [root@n2vl-pa-ccmz01 ccm_user]# cat /root/delete_oldest_backup.sh
    #!/bin/sh
    cd /opt/serviced/var/backups
    `ls -t | sed -e '1,2d' | xargs -d '\n' -r rm`

  • Avatar
    Chris De Coster

    I use this script in my cron.daily to prevent the backupfilesystem to fill up.

    #!/bin/sh
    # Prune backups

    # Set the backup path base on /etc/default/serviced
    # SERVICED_BACKUPS_PATH=/opt/serviced/var/backups
    SERVICED_BACKUPS_PATH=$(grep "^SERVICED_BACKUPS_PATH" /etc/default/serviced | sed 's/^.*=//')
    if [ "$SERVICED_BACKUPS_PATH" == "" ];
    SERVICED_BACKUPS_PATH=$(grep "^#.*SERVICED_BACKUPS_PATH" /etc/default/serviced | sed 's/^.*=//')
    fi;
    echo Using Backup path of $SERVICED_BACKUPS_PATH

    # Goal of this script is to prune the backup directory so that all of the following are true:
    # 1. We never end up creating a state where we end up with no backup.
    # 2. It is always possible to take a backup. If the size of the last back plus 10% doesn't
    # fit on disk, go delete an old one, and keep deleting until we have enough space or #1 applies

    # Loop until we have enough space for a backup, or we have pruned to minimum of only one backup
    while true
    do
    # Count number of backups
    NUM_BACKUPS=$(find $SERVICED_BACKUPS_PATH -maxdepth 1 -type f -name backup-*.tgz | wc -l)

    # If number of backups 1 or less exit since we will never delete the only backup
    if [ $NUM_BACKUPS -lt 2 ]; then
    echo "Didn't find any more backups eligible for deletion"
    exit
    fi;

    echo Found $NUM_BACKUPS backup files

    K_FREE=$(df -k -P $SERVICED_BACKUPS_PATH | tail -1 | awk {'print $4'})
    BYTES_FREE=$(awk -vk=$K_FREE -vb=1024 'BEGIN{printf "%.0f", k * b}')
    echo Amount of space free on $SERVICED_BACKUPS_PATH is $BYTES_FREE bytes

    SIZE_OF_LATEST_BACKUP=$(ls -lt $SERVICED_BACKUPS_PATH/backup*.tgz | head -1 | awk {'print $5'})
    echo Size of latest backup is $SIZE_OF_LATEST_BACKUP bytes

    # Add a percent to size of last backup file, to estimate size of next one
    EXTRA_PERCENT="10.0"
    ESTIMATED_SIZE=$(awk -vs=$SIZE_OF_LATEST_BACKUP -vp=$EXTRA_PERCENT 'BEGIN{printf "%.0f", s * p}')
    echo Estimated next backup size is $ESTIMATED_SIZE

    if [ $ESTIMATED_SIZE -lt $BYTES_FREE ]; then
    echo "We likely have enough space to take another backup. No need for pruning"
    exit
    fi;

    # delete the oldest backup and then go round the loop again
    OLDEST_BACKUP=$(ls -lrt $SERVICED_BACKUPS_PATH/backup*.tgz | head -1 | awk {'print $9'})
    echo Pruning old backup $OLDEST_BACKUP
    rm -f $OLDEST_BACKUP

    done

  • Avatar
    Matthew Burley

    Is there a way to exclude historic events from backups as our backups are now 2.6TB and taking over 48 hours to finish. Need to narrow this down

Powered by Zendesk