Follow

Error: "no space left on device" when backing up with Control Center

Applies To

  • Control Center 1.0.x
  • Control Center 1.1.x

Summary

When backing up via Control Center, an error including the phrases “Error 500: API error (500)” and “no space left on device” may be encountered if the disk containing the /tmp directory lacks sufficient space.

By default Control Center stages (spools) a considerable amount of data in the /tmp directory during a backup. This data is at least 800 MB on a brand new install with no devices added. The space required for a production instance is considerably more.

Control Center then creates a large tarball (>1.2GB) of the full system backup in the /opt/serviced/var/backups directory. After the tarball is successfully written, the staged data in the /tmp directory is removed.

NOTE: It is extremely important that sufficient space exists on the drive hosting the /tmp directory while the backup is running.

If this error is encountered, a new destination for the staged backup data must be specified.

Procedure

Considerations

  • To perform this procedure, it is necessary to identify a Identify a drive with enough space for the spooling files. The size of the most recent successfully saved backup tarball is a good guide to the required amount of storage - assuming the general configuration of the instance has not changed significantly.
  • In this example the disk hosting the /opt/serviced/var/backups directory (the default location for the tarball backups) is assumed to have sufficient space to host the spooling files.
  • This procedure creates a subdirectory (/opt/serviced/var/backups/tmp) within the .../backups directory to host the spooling files.

Perform the following to specify a new destination for the staged backup data:

  1. Identify a drive with enough space for the spooling files.
  2. Become root or a user with root authority:
    sudo - su
  3. Create the new directory to host the spooling files. In this example, /opt/serviced/var/backups/tmp:
    mkdir /opt/serviced/var/backups/tmp
  4. Edit the serviced configuration file /etc/default/serviced to reflect the new location for staging files:
    1. Open the file for editing.
    2. Locate the following line:
      # TMP=/tmp

      NOTE: This line defines the location for the spooling files.

    3. Uncomment the line (remove the # sign).
    4. Change the value from /tmp the new location. In this example, change the following line:
      # TMP=/tmp

      to read:

      TMP=/opt/serviced/var/backups/tmp
    5. Save and exit the file.
    6. To make the changes effective, stop Resource Manager, then restart Control Center. Optionally, restart Resource Manager after Control Center is running:
      serviced service stop Zenoss.resmgr
      systemctl stop serviced
      systemctl start serviced
      (optonal) serviced service start Zenoss.resmgr
  5. Delete the var/lib/docker/tmp directory and replace it with a symbolic link to the new spooling location as your target. In this example, /opt/serviced/var/backups/tmp:
    rm /var/lib/docker/tmp -R
    ln -s /opt/serviced/var/backups/tmp /var/lib/docker
  6. Optionally, monitor the new backup directory when the next backup is in progress. This can verify that the spooling files are being created in the correct location

    NOTE: Use CTRL-C to stop the command.:

    watch ls -laFh /opt/serviced/var/backups/tmp/
Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk