Follow

How To Use zodbscan To Scan ZODB for Dangling References

Applies To

  • Zenoss 5.x
  • Zenoss 4.x

Summary

The zodbscan python script (available as part of the zenoss.toolbox project on Github - https://github.com/zenoss/zenoss.toolbox) scans the zodb database for dangling references. Dangling references are references that no longer point to appropriate content.

If the script discovers a dangling reference it analyzes the pickled object (see note) to determine:

  • The path through the object graph
  • The name of the attribute that references the missing OID
  • Which class it should be

The causes for dangling references can be diverse - issues with ZenDS, issues with ZenPacks, improper zendmd scripting, etc.

Note: The terms POSKeyError, PKE, and Dangling Reference all refer to the same underlying issue - a bad or broken object in zodb.

Note: In Python, pickling implements an algorithm that serializes/de-serializes a Python object structure. Pickling is the process of converting a Python object hierarchy into a byte stream. Unpickling is the inverse operation.

Symptoms of Dangling References

The symptoms of dangling reference issues are diverse because they are related to the location of the error(s) in the Zope database.

When POSKeyErrors or dangling reference errors are present, Zenoss admins may find that:

  • they see a yellow error banner in the web interface flagging a POSKeyError
  • one or more Zenoss daemons have POSKeyErrors detailed in their logs
  • they cannot load certain pages in the web interface
  • they cannot view/delete certain devices/device components
  • they cannot run certain reports

Using the zodbscan Script

The following section describes the arguments and command syntax for the script.

usage:
zodbscan [-h] [-v] [-v10] [--tmpdir TMPDIR] [-s]

zodbscan -
scans ZODB checking objects for dangling references -
https://support.zenoss.com/hc/en-us/articles/203118175

optional arguments:

-h, --help show this help message and exit
-v, --version show program's version number and exit
-v10, --debug debug verbose log output (debug logging)
--tmpdir TMPDIR override the TMPDIR setting

Fixing Errors Found

If zodbscan finds errors, run the following tools:

If these tools do not resolve the error(s) displayed by zodbscan, contact Zenoss support.

Example Script Output

The following is example output from the zodbscan script:

[2016-02-17 18:15:27] Initializing zodbscan v2.0.0 (detailed log at /opt/zenoss/log/toolbox/zodbscan.log)
[2016-02-17 18:15:27] Examining 100792 items in the 'zodb' database:
[2016-02-17 18:15:43]  Verified  [==================================================] 100%

[2016-02-17 18:15:43] Execution finished in 0:00:16
Was this article helpful?
1 out of 3 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    John Nally

    Hi,

    I have an issue where zodb scan higlighted uncontactable objects.
    dbscan reported:
    ** 96.80% of zodb objects not reachable - examine your zenossdbpack settings **

    Zope event log
    ##################################################
    2016-08-23T17:35:16 ERROR ZODB.ConflictResolution Unexpected error
    Traceback (most recent call last):
    File "/opt/zenoss/lib/python2.7/site-packages/ZODB/ConflictResolution.py", line 243, in tryToResolveConflict
    oldData = self.loadSerial(oid, oldSerial)
    File "/opt/zenoss/lib/python2.7/site-packages/relstorage/storage.py", line 513, in loadSerial
    raise POSKeyError(oid)
    POSKeyError: 0x08
    ##################################################
    Afterwards I ran findposkeyerror, zenrelationscan and zencatalog scan but none of them found anything.
    What else can I do to fix this?
    I am running zenoss 5.0 and have documented my checks / errors here:
    http://www.zenoss.org/forum/145906

Powered by Zendesk