Error: "no space left on device" when backing up with Control Center

Applies To

• Control Center 1.0.x
• Control Center 1.1.x

Summary

When backing up via Control Center, an error including the phrases “Error 500: API error (500)” and “no space left on device” may be encountered if the disk containing the /tmp directory lacks sufficient space.

By default Control Center stages (spools) a considerable amount of data in the /tmp directory during a backup. This data is at least 800 MB on a brand new install with no devices added. The space required for a production instance is considerably more.

Control Center then creates a large tarball (>1.2GB) of the full system backup in the /opt/serviced/var/backups directory. After the tarball is successfully written, the staged data in the /tmp directory is removed.

NOTE: It is extremely important that sufficient space exists on the drive hosting the /tmp directory while the backup is running.

If this error is encountered, a new destination for the staged backup data must be specified.

Procedure

Considerations

• To perform this procedure, it is necessary to identify a Identify a drive with enough space for the spooling files. The size of the most recent successfully saved backup tarball is a good guide to the required amount of storage - assuming the general configuration of the instance has not changed significantly.
• In this example the disk hosting the /opt/serviced/var/backups directory (the default location for the tarball backups) is assumed to have sufficient space to host the spooling files.
• This procedure creates a subdirectory (/opt/serviced/var/backups/tmp) within the .../backups directory to host the spooling files.

Perform the following to specify a new destination for the staged backup data:

1. Identify a drive with enough space for the spooling files.
2. Become root or a user with root authority:

   sudo - su

3. Create the new directory to host the spooling files. In this example, /opt/serviced/var/backups/tmp:

   mkdir /opt/serviced/var/backups/tmp

4. Edit the serviced configuration file /etc/default/serviced to reflect the new location for staging files:
   1. Open the file for editing.
   2. Locate the following line:

      # TMP=/tmp

      NOTE: This line defines the location for the spooling files.

   3. Uncomment the line (remove the # sign).
   4. Change the value from /tmp the new location. In this example, change the following line:
      

      # TMP=/tmp

      to read:

      TMP=/opt/serviced/var/backups/tmp

   5. Save and exit the file.
   6. To make the changes effective, stop Resource Manager, then restart Control Center. Optionally, restart Resource Manager after Control Center is running:

      serviced service stop Zenoss.resmgr
      systemctl stop serviced
      systemctl start serviced
      (optonal) serviced service start Zenoss.resmgr

5. Delete the var/lib/docker/tmp directory and replace it with a symbolic link to the new spooling location as your target. In this example, /opt/serviced/var/backups/tmp:

   rm /var/lib/docker/tmp -R
   ln -s /opt/serviced/var/backups/tmp /var/lib/docker

6. Optionally, monitor the new backup directory when the next backup is in progress. This can verify that the spooling files are being created in the correct location

   NOTE: Use CTRL-C to stop the command.:

   watch ls -laFh /opt/serviced/var/backups/tmp/