Follow

How To Start Zenoss After Encountering a Problem

Applies To

  • Zenoss 4.x. and Earlier

Summary

Administrators may encounter a scenario when they expect specific Zenoss daemons to be operating and find that they are not. Often the cause of the problem can traced to one or more daemons' failure to start correctly. Two examples of scenarios that can cause this to occur might be:

  • A hard shutdown of the Linux server hosting Zenoss (for example during a power outage). On system reboot, a stray PID may indicate that a daemon is running and therefore it may not be restarted correctly.
  • A 'restart' of Zenoss on a system that is swapping RAM. In this case, daemons may fail to fully shut down before attempting a restart.

Procedure

To start Zenoss cleanly, first shut down Zenoss and verify that all daemons apart from those associated with Zends and Rabbitmq have shut down completely. Begin by stopping Zenoss:

 zenoss stop

Next, search for processes associated with Zenoss:

 ps ax | grep zenoss

On version 4.x of Zenoss, only processes associated with Zends and Rabbitmq should remain running. For example, administrators should see the following process still running after a 'zenoss stop":

/bin/sh /opt/zends/bin/mysqld_safe
/opt/zends/bin/mysqld
/usr/lib64/erlang/erts-5.6.5/bin/beam.smp

On earlier versions of Zenoss, there should be no processes remaining that are associated with Zenoss.

However, after a 'zenoss stop' the Zenoss daemons themselves should no longer be running. For example, the following should not appear in the output of the 'ps' command:

/opt/zenoss/bin/python /opt/zenoss/Products/ZenRRD/zenperfsnmp.py

or

/opt/zenoss/bin/python /opt/zenoss/Products/ZenRRD/zencommand.py

If processes other than those associated with Zends or Rabbitmq are seen to be running after a 'zenoss stop' command has been run, verify that swapping is not delaying the shutdown of the daemons by running 'top' and looking for python process consuming CPU. Also, look for higher than normal percentages of 'wait' time being consumed. If these conditions are observed, give the operating system time to shut down all of the daemons fully. Once the CPU utilization has dropped and no processes associated with Zenoss daemons are observed in top, re-run the ps command detailed above to confirm that all daemons have shut down.

If you continue to see processes associated with Zenoss (but not Zends or Rabbitmq) running that are not being delayed in shutdown by swapping, you may want to forcibly quit these 'stray' or 'stuck'  processes. To do so, identify the process' PID and use the 'kill' command to stop the process:

kill [process_id_inserted_here]

or

kill -9 [process_id_inserted_here]

Once you have confirmed that all Zenoss daemons not associated with Zends or Rabbitmq are down, you may start Zenoss:

zenoss start 

Example

Note: Not all of the time will you have to kill. This is an example where a rogue process did not stop initially or with a normal kill.

[zenoss@applianceincloset ~]$ zenoss stop
Daemon: zenmailtx stopping...
Daemon: zenwinperf stopping...
Daemon: zenwebtx stopping...
Daemon: zenjmx stopping...
Daemon: zeneventlog stopping...
Daemon: zenwin stopping...
Daemon: zenprocess stopping...
Daemon: zencommand stopping...
Daemon: zenperfsnmp stopping...
Daemon: zenmodeler stopping...
Daemon: zentrap stopping...
Daemon: zenactions stopping...
Daemon: zenstatus stopping...
Daemon: zensyslog stopping...
Daemon: zenping stopping...
Daemon: zenhub stopping...
Daemon: zopectl . daemon process stopped
Daemon: zeoctl . daemon process stopped
[zenoss@applianceincloset ~]$ ps aux | grep py
raa-web   3055  0.0  0.6  45972 25580 ?        Ssl  Feb10   1:36 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon
root      3063  0.2  0.6  80592 25752 ?        Ssl  Feb10 130:04 /usr/bin/python2.4 /usr/bin/raa-service
raa-web  13605  0.2  0.5  46232 23472 ?        S    Mar15   7:19 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon
raa-web  15034  0.0  0.5  45972 23404 ?        S    Mar16   0:00 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon
zenoss   17273  7.6  0.6  43644 27340 ?        Rl   14:57   0:00 /usr/lib/zope/bin/python /home/zenoss/Products/ZenHub/zenhub.py --configfile /home/zenoss/etc/zenhub.conf --cycle --daemon

Note: raa is an appliance process. Appliance processes don't need to be restarted. 

[zenoss@applianceincloset ~]$ kill 17273
[zenoss@applianceincloset ~]$ ps aux | grep py

raa-web   3055  0.0  0.6  45972 25580 ?        Ssl  Feb10   1:36 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon
root      3063  0.2  0.6  80592 25752 ?        Ssl  Feb10 130:04 /usr/bin/python2.4 /usr/bin/raa-service
raa-web  13605  0.2  0.5  46232 23472 ?        S    Mar15   7:20 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon
raa-web  15034  0.0  0.5  45972 23404 ?        S    Mar16   0:00 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon
zenoss   17273  2.0  0.6  43620 27316 ?        Sl   14:59   0:00 /usr/lib/zope/bin/python /home/zenoss/Products/ZenHub/zenhub.py --configfile /home/zenoss/etc/zenhub.conf --cycle --daemon

Note: The process is still there, it didn't go away.

[zenoss@applianceincloset ~]$ kill -9 17298
[zenoss@applianceincloset ~]$ ps aux | grep py
raa-web   3055  0.0  0.6  45972 25580 ?        Ssl  Feb10   1:36 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon
root      3063  0.2  0.6  80592 25752 ?        Ssl  Feb10 130:04 /usr/bin/python2.4 /usr/bin/raa-service
raa-web  13605  0.2  0.5  46232 23472 ?        S    Mar15   7:20 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon
raa-web  15034  0.0  0.5  45972 23404 ?        S    Mar16   0:00 /usr/bin/python2.4 /usr/bin/raa-web-scgi /etc/raa/prod.cfg --no-daemon
zenoss   17304  0.0  0.0   2912   800 pts/0    S+   15:00   0:00 grep py

Note: Now the process is gone.

[zenoss@applianceincloset ~]$ zenoss start
Daemon: zeoctl . daemon process started, pid=17317
Daemon: zopectl . daemon process started, pid=17321
Daemon: zenhub starting...
Daemon: zenping starting...
Daemon: zensyslog starting...
Daemon: zenstatus starting...
Daemon: zenactions starting...
Daemon: zentrap starting...
Daemon: zenmodeler starting...
Daemon: zenperfsnmp starting...
Daemon: zencommand starting...
Daemon: zenprocess starting...
Daemon: zenwin starting...
Daemon: zeneventlog starting...
Daemon: zenjmx starting...
Daemon: zenwebtx starting...
Daemon: zenwinperf starting...
Daemon: zenmailtx starting...
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk