How to use findposkeyerror to scan/fix zodb dangling references

Applies To

• Zenoss 5.x
• Zenoss 4.x

Summary

The findposkeyerror python script (available as part of the zenoss.toolbox project on Github - https://github.com/zenoss/zenoss.toolbox) will scan through a path in dmd (the default path is '/') and check each object in that path, as well as its associated attributes and relations, to make sure that all references refer to valid objects. The findposkeyerror script will log any issues it finds and has an option to attempt to repair a subset of PKE issues with the -f flag.

Note:The term POSKeyError (PKE) refers to a POSitional Key Error. Zenoss uses a Zope object database called zodb. All objects within this database contain relationships to other objects - a PKE exists when one (or more) of those relationships is incorrect. When a reference exists that does not correspond to an actual object, a PKE is thrown. The terms POSKeyError, PKE, and Dangling Reference all refer to the same underlying issue.

Symptoms of POSKeyErrors/Dangling References

Due to the low level nature of PKEs, the symptoms are quite diverse. If you see any of the following, a check for PKEs is called for:

• Yellow flare message (error banner) in the UI mentioning a POSKeyError.
• POSKeyErrors mentioned in any Zenoss daemon log files.
• Errors mentioned in the results from the zodbscan, or zenrelationscan scripts.

The following is an example of a POSKeyError error traceback that displays in the Zope UI:

 Traceback (most recent call last):
 File "/usr/local/lib/python/ZEO/zrpc/connection.py", line 407, in handle_request ret = meth(*args)
 File "/usr/local/lib/python/ZODB/FileStorage/FileStorage.py", line 655, in modifiedInVersion pos = self._lookup_pos(oid)
 File "/usr/local/lib/python/ZODB/FileStorage/FileStorage.py", line 534, in _lookup_pos raise POSKeyError(oid) POSKeyError: 0x172024

Important: Although the command findposkeyerror will not make changes to your system, when the command is run with the -f flag set, for example, as findposkeyerror -f, the script will make modifications to your system.

Using the findposkeyerror Script

usage:

findposkeyerror.py [-h] [-v] [-v10] [--tmpdir TMPDIR] [-s] [-f] [-n CYCLES] [-p PATH] [-u]

optional arguments:

-h, --help	show this help message and exit
-v, --version	show program's version number and exit
-v10, --debug	debug verbose log output (debug logging)
--tmpdir TMPDIR	override the TMPDIR setting
-s, --skipEvents	skip creating summary events
-f, --fix	attempt to fix ZenRelationship objects
-n CYCLES, --cycles CYCLES	maximum times to cycle (with --fix)
-p PATH, --path PATH	base path to scan from (Devices.Server)
-u, --unlimitedram	skip transaction.abort() - unbounded RAM, ~40% faster

Example Script Output

[2016-02-18 13:37:45] Initializing findposkeyerror v2.0.0 (detailed log at /opt/zenoss/log/toolbox/findposkeyerror.log)
[2016-02-18 13:37:49] Examining items under the '/' path ():

[2016-02-18 13:38:17]  Cycle 1  | Items Scanned:       133148 | Errors:       0 |

[2016-02-18 13:38:17] Execution finished in 0:00:31