Applies To
- Zenoss 4.x. and Earlier
Summary
Administrators may encounter a scenario when they expect specific Zenoss daemons to be operating and find that they are not. Often the cause of the problem can traced to one or more daemons' failure to start correctly. Two examples of scenarios that can cause this to occur might be:
- A hard shutdown of the Linux server hosting Zenoss (for example during a power outage). On system reboot, a stray PID may indicate that a daemon is running and therefore it may not be restarted correctly.
- A 'restart' of Zenoss on a system that is swapping RAM. In this case, daemons may fail to fully shut down before attempting a restart.
Procedure
To start Zenoss cleanly, first shut down Zenoss and verify that all daemons apart from those associated with Zends and Rabbitmq have shut down completely. Begin by stopping Zenoss:
zenoss stop
Next, search for processes associated with Zenoss:
ps ax | grep zenoss
On version 4.x of Zenoss, only processes associated with Zends and Rabbitmq should remain running. For example, administrators should see the following process still running after a 'zenoss stop":
/bin/sh /opt/zends/bin/mysqld_safe
/opt/zends/bin/mysqld
/usr/lib64/erlang/erts-5.6.5/bin/beam.smp
On earlier versions of Zenoss, there should be no processes remaining that are associated with Zenoss.
However, after a 'zenoss stop' the Zenoss daemons themselves should no longer be running. For example, the following should not appear in the output of the 'ps' command:
/opt/zenoss/bin/python /opt/zenoss/Products/ZenRRD/zenperfsnmp.py
or
/opt/zenoss/bin/python /opt/zenoss/Products/ZenRRD/zencommand.py
If processes other than those associated with Zends or Rabbitmq are seen to be running after a 'zenoss stop' command has been run, verify that swapping is not delaying the shutdown of the daemons by running 'top' and looking for python process consuming CPU. Also, look for higher than normal percentages of 'wait' time being consumed. If these conditions are observed, give the operating system time to shut down all of the daemons fully. Once the CPU utilization has dropped and no processes associated with Zenoss daemons are observed in top, re-run the ps command detailed above to confirm that all daemons have shut down.
If you continue to see processes associated with Zenoss (but not Zends or Rabbitmq) running that are not being delayed in shutdown by swapping, you may want to forcibly quit these 'stray' or 'stuck' processes. To do so, identify the process' PID and use the 'kill' command to stop the process:
kill [process_id_inserted_here]
or
kill -9 [process_id_inserted_here]
Once you have confirmed that all Zenoss daemons not associated with Zends or Rabbitmq are down, you may start Zenoss:
zenoss start
Example
Note: Not all of the time will you have to kill. This is an example where a rogue process did not stop initially or with a normal kill.
[zenoss@applianceincloset ~]$ zenoss stop Daemon: zenmailtx stopping... Daemon: zenwinperf stopping... Daemon: zenwebtx stopping... Daemon: zenjmx stopping... Daemon: zeneventlog stopping... Daemon: zenwin stopping... Daemon: zenprocess stopping... Daemon: zencommand stopping... Daemon: zenperfsnmp stopping... Daemon: zenmodeler stopping... Daemon: zentrap stopping... Daemon: zenactions stopping... Daemon: zenstatus stopping... Daemon: zensyslog stopping... Daemon: zenping stopping... Daemon: zenhub stopping... Daemon: zopectl . daemon process stopped Daemon: zeoctl . daemon process stopped
[zenoss@applianceincloset ~]$ ps aux | grep py raa-web 3055 0.0 0.6 45972 25580 ? Ssl Feb10 1:36 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon root 3063 0.2 0.6 80592 25752 ? Ssl Feb10 130:04 /usr/bin/python2.4 /usr/bin/raa-service raa-web 13605 0.2 0.5 46232 23472 ? S Mar15 7:19 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon raa-web 15034 0.0 0.5 45972 23404 ? S Mar16 0:00 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon zenoss 17273 7.6 0.6 43644 27340 ? Rl 14:57 0:00 /usr/lib/zope/bin/python /home/zenoss/Products/ZenHub/zenhub.py --configfile /home/zenoss/etc/zenhub.conf --cycle --daemon
Note: raa is an appliance process. Appliance processes don't need to be restarted.
[zenoss@applianceincloset ~]$ kill 17273 [zenoss@applianceincloset ~]$ ps aux | grep py raa-web 3055 0.0 0.6 45972 25580 ? Ssl Feb10 1:36 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon root 3063 0.2 0.6 80592 25752 ? Ssl Feb10 130:04 /usr/bin/python2.4 /usr/bin/raa-service raa-web 13605 0.2 0.5 46232 23472 ? S Mar15 7:20 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon raa-web 15034 0.0 0.5 45972 23404 ? S Mar16 0:00 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon zenoss 17273 2.0 0.6 43620 27316 ? Sl 14:59 0:00 /usr/lib/zope/bin/python /home/zenoss/Products/ZenHub/zenhub.py --configfile /home/zenoss/etc/zenhub.conf --cycle --daemon
Note: The process is still there, it didn't go away.
[zenoss@applianceincloset ~]$ kill -9 17298 [zenoss@applianceincloset ~]$ ps aux | grep py raa-web 3055 0.0 0.6 45972 25580 ? Ssl Feb10 1:36 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon root 3063 0.2 0.6 80592 25752 ? Ssl Feb10 130:04 /usr/bin/python2.4 /usr/bin/raa-service raa-web 13605 0.2 0.5 46232 23472 ? S Mar15 7:20 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon raa-web 15034 0.0 0.5 45972 23404 ? S Mar16 0:00 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon zenoss 17304 0.0 0.0 2912 800 pts/0 S+ 15:00 0:00 grep py
Note: Now the process is gone.
[zenoss@applianceincloset ~]$ zenoss start Daemon: zeoctl . daemon process started, pid=17317 Daemon: zopectl . daemon process started, pid=17321 Daemon: zenhub starting... Daemon: zenping starting... Daemon: zensyslog starting... Daemon: zenstatus starting... Daemon: zenactions starting... Daemon: zentrap starting... Daemon: zenmodeler starting... Daemon: zenperfsnmp starting... Daemon: zencommand starting... Daemon: zenprocess starting... Daemon: zenwin starting... Daemon: zeneventlog starting... Daemon: zenjmx starting... Daemon: zenwebtx starting... Daemon: zenwinperf starting... Daemon: zenmailtx starting...
Comments