- Zenoss 5.x
- Zenoss 4.x
The findposkeyerror python script (available as part of the zenoss.toolbox project on Github - https://github.com/zenoss/zenoss.toolbox) will scan through a path in dmd (the default path is '/') and check each object in that path, as well as its associated attributes and relations, to make sure that all references refer to valid objects. The findposkeyerror script will log any issues it finds and has an option to attempt to repair a subset of PKE issues with the -f flag.
Note:The term POSKeyError (PKE) refers to a POSitional Key Error. Zenoss uses a Zope object database called zodb. All objects within this database contain relationships to other objects - a PKE exists when one (or more) of those relationships is incorrect. When a reference exists that does not correspond to an actual object, a PKE is thrown. The terms POSKeyError, PKE, and Dangling Reference all refer to the same underlying issue.
Symptoms of POSKeyErrors/Dangling References
Due to the low level nature of PKEs, the symptoms are quite diverse. If you see any of the following, a check for PKEs is called for:
- Yellow flare message (error banner) in the UI mentioning a POSKeyError.
- POSKeyErrors mentioned in any Zenoss daemon log files.
- Errors mentioned in the results from the zodbscan, or zenrelationscan scripts.
The following is an example of a POSKeyError error traceback that displays in the Zope UI:
Traceback (most recent call last): File "/usr/local/lib/python/ZEO/zrpc/connection.py", line 407, in handle_request ret = meth(*args) File "/usr/local/lib/python/ZODB/FileStorage/FileStorage.py", line 655, in modifiedInVersion pos = self._lookup_pos(oid) File "/usr/local/lib/python/ZODB/FileStorage/FileStorage.py", line 534, in _lookup_pos raise POSKeyError(oid) POSKeyError: 0x172024
Important: Although the command findposkeyerror will not make changes to your system, when the command is run with the -f flag set, for example, as findposkeyerror -f, the script will make modifications to your system.
Using the findposkeyerror Script
findposkeyerror.py [-h] [-v] [-v10] [--tmpdir TMPDIR] [-s] [-f] [-n CYCLES] [-p PATH] [-u]
-h, --help show this help message and exit -v, --version show program's version number and exit -v10, --debug debug verbose log output (debug logging) --tmpdir TMPDIR override the TMPDIR setting -s, --skipEvents skip creating summary events -f, --fix attempt to fix ZenRelationship objects -n CYCLES, --cycles CYCLES maximum times to cycle (with --fix) -p PATH, --path PATH base path to scan from (Devices.Server) -u, --unlimitedram skip transaction.abort() - unbounded RAM, ~40% faster
Example Script Output
[2016-02-18 13:37:45] Initializing findposkeyerror v2.0.0 (detailed log at /opt/zenoss/log/toolbox/findposkeyerror.log) [2016-02-18 13:37:49] Examining items under the '/' path (): [2016-02-18 13:38:17] Cycle 1 | Items Scanned: 133148 | Errors: 0 | [2016-02-18 13:38:17] Execution finished in 0:00:31