Follow

How To Troubleshoot Graphs for Devices on Remote Collectors

Applies To

  • Zenoss 4.2.4
  • Zenoss 4.2.3
  • Zenoss 4.1.1

Summary

The Distributed Collector ZenPack allows the configuration of additional servers to collect performance information from devices and share the load in gathering metrics. The graphs generated from the data on these remote collectors are created by the zenrender daemon, instead of the Zope server on the Zenoss master. Therefore, should there be break in network connectivity between a the master and a collector and/or a problem with the zenrender daemon on a collector performance graphs may fail to render. In this article we examine the steps necessary to troubleshoot graphs that fail to render. 

 

Procedure

The following procedures explain how to verify connectivity and manually obtain a sample graph from the remote collector.

The steps documented here assume the following:

  • A single zenhub is in use, for example, one on the Zenoss master server. If the remote collector is associated with a remote hub, you must replace references to the Zenoss master server with the DNS name of the remote hub device.
  • In this example, the remote collector is named remoteCollector. You must replace the remoteCollector string in the examples with the actual name of your remote collector.
  • The names of daemons running on remote collectors are prepended with the name of the collector. For example, zenrender, when running on a collector named remoteColllector, becomes remoteCollector_zenrender. Be sure to substitute the correct daemon names when completing the steps below in your production environment.

 

Troubleshooting Graphs on Zenoss Resource Manager

The following section describes troubleshooting procedures for Zenoss Resource Manager.

Verify Connectivity of Zenoss Master to Remote Collector

Use the following procedures to verify connectivity between the Zenoss master server and the machine hosting the remote collector.

Verify that zenrender is Running on the Collector

To verify zenrender is running and to restart it if necessary, perform the following:

  1. Log on to the remote collector.
  2. Enter the following command as the zenoss user to determine the status of the daemon:
    $ remoteCollector_zenrender status

    For example:

    $ MyCollector_zenrender status 

    If zenrender is running, the command returns a result similar to the following:

    program running; pid=7011

  3. Enter the following command as the zenoss user to restart the daemon, if necessary:
     $ remoteCollector_zenrender restart

Note: If the daemon is running and graphs do not display, continue with the steps to troubleshoot possible network connectivity issues.

Test Connectivity with telnet

The master server connects to the collector on port 8091 to render graphs. Use telnet to test the connection to the remote collector machine on port 8091:

# telnet remote_collector_device 8091

Note: If telnet is unavailable, install it with the following command:

# yum install telnet -y

If the telnet succeeds, output similar to the following displays:

Trying 10.177.nnn.nnn...
Connected to remoteCollector.
Escape character is '^]'

Note: You must manually quit the telnet connection when you finish.

One of the possible errors that telnet can fail with is:

No route to host...

Possible causes for this error can include:

  1. A firewall issue, for example iptables incorrectly configured to block port 8091 between hosts.
  2. Name/address resolution, for example DNS or host table configuration.
  3. A physical break in network connectivity to the collector machine.

A Connection refused message generally means that zenrender is not started or responding.

Search for Hung zenrender Processes with Netstat

Log on to the remote collector and enter the following as the zenoss user:

 $ netstat -nap | egrep 8091

The pid (process ID) returned by the netstat commanded should match the process ID returned when the command service remoteCollector_zenrender status is run. If the process IDs do not match, stop (kill) the daemon identified by the netstat command and restart the zenrender daemon.

Verify the Correct Port for the daemon with netstat

  1. Log on to the remote collector and enter the following as the zenoss user:
    $ netstat -nap | grep zenrender_process_id
  2. Consult the output to verify the zenrender daemon is listening on the default port 8091. If zenrender is listening on a different port, either reconfigure the port to listen on port 8091, or update the collector's settings in the appropriate section of the Resource Manager user interface.

Obtain a Sample Performance Graph from the Zenoss Master

Perform the following process to manually obtain a sample performance graph:

  1. From Zenoss UI, Click the Infrastructure tab to display the Device pane.
  2. In the Device pane, select a device associated with the remote collector to display the Overview pane for the device.
  3. Click the Graphs tab in the left pane to display the Performance Graphs in the right pane.
  4. Select a graph and right-click the graph.
  5. Select Copy Image Location.
  6. Paste the location information into a text file so it can be viewed and copied. The location information yields a long data string, for example:

    http://10.177.nnn.nnn:8091/render?gopts=eNqld ...%3D%3D&drange=129600

    Note: The ellipses (...) indicate many lines of code left out, for brevity.

  7. From the Zenoss master, use the wget command to obtain the image directly from the remote collector.
    Note: Within the location string, you must replace any occurance of the html code for ampersand (&) with an actual ampersand ('&').

    For example, using the line of code from the location, the bold section in our example (above):

    $ wget 'http://remoteCollector:8091/render?gopts=eNq1d...3D&;drange=129600' \
    -O test_graph.png

    Note: The port number in this wget is 8091, not 8090.

    This should successfully connect with a 200 message. For example:

    Connecting to MyCollector.zenoss.loc|10.177.nnn.nnn|:8091... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 15529 (15K) [image/png]
    Saving to: `test_graph.png'
    
    100%[===================================================>] 15,529  --.-K/s   in 0s
    
    2014-nn-nn 17:31:01 (64.7 MB/s) - `test_graph.png' saved [15529/15529]
    

    The test_graph.png file should be a valid PNG format graphic that shows the graph.

    Note: If you copy the file to a desktop to view the graph, ensure the copy is binary instead of text, without conversion such as DOS to Unix.

  8. From the Zenoss master, delete or rename the test_graph.png, for example:
     $ rm test_graph.png
  9. Use the wget command to obtain the image directly from the zenhub process.

    Note: Within the location string, you must replace any occurance of the html code for ampersand (&) with an actual ampersand ('&').

    For example using the line of code from the location:

    $ wget 'http://zenossMaster:8090/MyCollector/render?gopts=eNq1d...3D&;drange=129600' \
    -O test_graph.png

    The wget command should successfully connect with a 200 message and the test_graph.png file should be a valid PNG format graphic that displays the graph.

    If the graphs can be successfully obtained manually, it is likely that the desktop client browsers cannot connect directly to the port to obtain the graphic.

Verify that Desktop Browsers can Connect

To verify whether the browsers can connect directly to the appropriate port to obtain the performance graphic file, enter the URL into the desktop browser to view the page. If the page times out or shows unavailable, this indicates a firewall issue between the client and the Zenoss master. For example, port 8090 unavailable.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    Radovan Djogo **SUSPENDED**

    Hi,

    I have a problem with RM 4.2.5
    zenrender daemon is running on Remote Collector, still if I chose remote to be the collector for any device in fails to render graphs. There are no files generated in /opt/zenoss/perf.

  • Avatar
    Mark Shoemaker

    Hi Radovan,

    If you don't mind, could you please open a support ticket on this issue?

    Thanks!
    Mark Shoemaker, Zenoss

  • Avatar
    Radovan Djogo **SUSPENDED**

    Hi Mark,

    Of course, I just did.
    I was just wondering if I missed something, and thought maybe someone ran into similar issue.

    Thanks

Powered by Zendesk