Applies To
- Zenoss 5.1 and greater
Summary
It is important to have accurate and tested system backups because they can mitigate problems caused by software or hardware issues. One method to backup Control Center and its installed applications is to use serviced backup. Backups can be performed via the command line or through automated cron jobs (crontab).
The serviced backup command saves a backup of the current system, including the state of all services, and data to a compressed archive file (.tgz). Because serviced backup leverages functionality provided by serviced snapshot, backups can be performed without the need to shutdown zenoss or the docker containers because they are only momentarily suspended to enable reading the data.
NOTE: It is important that there is enough free space to receive and store backups because running low on available disk space will result in errors and impact system performance.
About serviced backup
The default directory that receives backups from serviced backup is /opt/serviced/var/backups.
The serviced backup command includes various options to tailor the command for specific use:
- USAGE
-
serviced backup [command options] [arguments...]
- DESCRIPTION
-
serviced backup DIRPATH
- OPTIONS
-
--exclude '--exclude option --exclude option'
Subdirectory of the tenant volume to exclude from backup
-
--help, -h
- Shows the help for an option
Procedures
Backups
Backups can be initiated from the UI or the command line. Zenoss stores its backups by default in the directory /opt/serviced/var/backups.
Best practices for backup include:
- Weekly backups for production environments.
- Increasing backup frequency to daily during initial deployment or for development environments.
Backup Using Control Center
To backup the system through the Control Center UI, navigate to Control Center > Backup/Restore and click the Create Backup button.
If a backup is run through the UI, the path is controlled by an entry in the /etc/default/serviced file:
# SERVICED_BACKUPS_PATH=/opt/serviced/var/backups
Change this entry to specify a new target directory for the UI initiated backups.
Backup Using the Command Line
To backup your system through the command line:
- As root or a valid Control Center user, backup the system to the default directory /opt/serviced/var/backups with the following command:
serviced backup /opt/serviced/var/backups
A successful backup is indicated by the system displaying the new backup file name, for example:
backup-2016-09-29-163201.tgz
Automating Backup with CentOS/RHEL 7
For automating system backup on CentOS/RHEL 7, crontab alone is not used, instead, bash scripts are used to automate cron, fstrim, zenossdbpack etc. Zenoss is configured with various bash scripts for this purpose.
For examples of how to create appropriate automation scripts, see the files in /etc/cron.daily/, /etc/ cron.hourly/ and /etc/cron.weekly/ in the current installation.
Note: If automation is used to perform backups, care must be exercised to ensure accumulation of old backups does not consume all disk space. Old backups must be curated to maintain available storage space.
Example serviced-backup File
As an example of a script using serviced backup, add a file, called serviced-backup in this example to the appropriate directory with the following contents:
#!/bin/sh # Execute a Control Center backup, writing the results to the backups directory ${SERVICED:=/opt/serviced/bin/serviced} backup /opt/serviced/var/backups
What would make this article more useful is presentation of how to "rotate" the backups. E.g., if you are taking weekly backups, your backup storage area will eventually fill up, which will prevent new backups from occurring. If the customer is not manually managing their backup storage on a weekly basis, they could end up getting a bad surprise when it is time to do a restore (i.e., they could find that all their backups are old). Can you provide a script (that could be part of the serviced-backup script) to cull the oldest backup if disk space is low? Thanks!
Hey there,
I've submitted a request to our KB team to get this article updated to reflect that. Thanks for the feedback!
I'm assuming to restore the backup we would want to do
serviced restore backupfile.tgz
is that correct?
You will want to provide the directory path, not the file name, like this:
serviced backup /opt/serviced/var/backups
If you type "serviced backup" alone at the CLI, it will return usage info, as follows:
# serviced backup
Incorrect Usage.
NAME:
backup - Dump all templates and services to a tgz file
USAGE:
command backup [command options] [arguments...]
DESCRIPTION:
serviced backup DIRPATH
OPTIONS:
--exclude '--exclude option --exclude option' Subdirectory of the tenant volume to exclude from backup
Thanks for the quick response, unfortunately we are still running into some issues. We are taking the backup from 1 machine and attempting to restore on another, to simulate a DR event. On the new control center machine, should all the applications be wiped so there are no applications installed before we attempt to restore the backup?
Also since we are trying to restore to a new machine, the IPs will be different. Is the best way to update the all the end points by updating them in the command center before starting the application?
Yes, you will want to restore to a "bare" Control Center: i.e., one that does not already have the Resource Manager application deployed. Depending on the version of Control Center, you may need a Host already added; on older versions of CC, the restore process will bring the old Host with it. If the latter, you may find you need to correct a mismatch of the "hostid" by deleting and adding back the Host in CC.
To fix the IP issues, I think you will only need to check on & re-do the "IP Assignments" under the RM application. If this is a multi-host install, there may be additional work to fix the Virtual IP of the collector pools.
I am more familiar with the Cisco UCSPM version of this product, but I think everything I said above will apply as well to RM.
Regards,
Eric Thirolle
Thanks again. We got this working, by doing what you said. Start with a bare control center and then restore.
One slightly annoying thing is that the backup and restore does seem to keep any updates to the /opt/zenoss/Products/ZenEvents/SyslogProcessing.py file. But that we can copy over after the restore.
It should be noted that on RHEL7 systems, /etc/cron.(hourly|daily|weekly|monthly) only works if cronie is enabled and in use. Standard crontab should be used instead if you aren't using cronie.
Hey,
I would recommend that since it is automated backup, there should be a mechanism that after 3 or 5 consecutive backups, the script should remove a previous backup file to remove dated backups to create more space for new ones.
This should first check that the new backup created has a size greater than or equal to immediately old backup to avoid incomplete backups to remove old ones.
Can someone help with any such script or procedure.
+1 on this request from Manish for a "logrotate"-type script for the backup files. The most recent backup file will be the most important, so I think the need to save the latest backup file should probably override the need to keep older backups by default.
Add this line to the script above to keep up to 7 days of backups only:
find /opt/serviced/var/backups -mtime +7 -delete
However, if your backups fail to run for 7 days, you won't have any backups, so be sure you are monitoring the results of your backup job at the same time.
Hi Everyone,
In my environment i am using the following 2 scripts which run daily. One creates backup and one removes the oldest one.
However, it is missing a minimum count check so if your backup is not happening for some reason, it will still remove the oldest backup and it might delete all backups made till yet.
1. Create a backup script in cron.daily with following contents:
[root@n2vl-pa-ccmz01 ccm_user]# cat /etc/cron.daily/zenoss_backup
#!/bin/sh
# Execute a Control Center backup, writing the results to the backups directory
${SERVICED:=/opt/serviced/bin/serviced} backup /opt/serviced/var/backups
2. Create a script in /root/delete_oldest_backup.sh with following contents:
[root@n2vl-pa-ccmz01 ccm_user]# cat /root/delete_oldest_backup.sh
#!/bin/sh
cd /opt/serviced/var/backups
`ls -t | sed -e '1,2d' | xargs -d '\n' -r rm`
I use this script in my cron.daily to prevent the backupfilesystem to fill up.
#!/bin/sh
# Prune backups
# Set the backup path base on /etc/default/serviced
# SERVICED_BACKUPS_PATH=/opt/serviced/var/backups
SERVICED_BACKUPS_PATH=$(grep "^SERVICED_BACKUPS_PATH" /etc/default/serviced | sed 's/^.*=//')
if [ "$SERVICED_BACKUPS_PATH" == "" ];
SERVICED_BACKUPS_PATH=$(grep "^#.*SERVICED_BACKUPS_PATH" /etc/default/serviced | sed 's/^.*=//')
fi;
echo Using Backup path of $SERVICED_BACKUPS_PATH
# Goal of this script is to prune the backup directory so that all of the following are true:
# 1. We never end up creating a state where we end up with no backup.
# 2. It is always possible to take a backup. If the size of the last back plus 10% doesn't
# fit on disk, go delete an old one, and keep deleting until we have enough space or #1 applies
# Loop until we have enough space for a backup, or we have pruned to minimum of only one backup
while true
do
# Count number of backups
NUM_BACKUPS=$(find $SERVICED_BACKUPS_PATH -maxdepth 1 -type f -name backup-*.tgz | wc -l)
# If number of backups 1 or less exit since we will never delete the only backup
if [ $NUM_BACKUPS -lt 2 ]; then
echo "Didn't find any more backups eligible for deletion"
exit
fi;
echo Found $NUM_BACKUPS backup files
K_FREE=$(df -k -P $SERVICED_BACKUPS_PATH | tail -1 | awk {'print $4'})
BYTES_FREE=$(awk -vk=$K_FREE -vb=1024 'BEGIN{printf "%.0f", k * b}')
echo Amount of space free on $SERVICED_BACKUPS_PATH is $BYTES_FREE bytes
SIZE_OF_LATEST_BACKUP=$(ls -lt $SERVICED_BACKUPS_PATH/backup*.tgz | head -1 | awk {'print $5'})
echo Size of latest backup is $SIZE_OF_LATEST_BACKUP bytes
# Add a percent to size of last backup file, to estimate size of next one
EXTRA_PERCENT="10.0"
ESTIMATED_SIZE=$(awk -vs=$SIZE_OF_LATEST_BACKUP -vp=$EXTRA_PERCENT 'BEGIN{printf "%.0f", s * p}')
echo Estimated next backup size is $ESTIMATED_SIZE
if [ $ESTIMATED_SIZE -lt $BYTES_FREE ]; then
echo "We likely have enough space to take another backup. No need for pruning"
exit
fi;
# delete the oldest backup and then go round the loop again
OLDEST_BACKUP=$(ls -lrt $SERVICED_BACKUPS_PATH/backup*.tgz | head -1 | awk {'print $9'})
echo Pruning old backup $OLDEST_BACKUP
rm -f $OLDEST_BACKUP
done
Is there a way to exclude historic events from backups as our backups are now 2.6TB and taking over 48 hours to finish. Need to narrow this down