Follow

Known Issue: ZEN-31334 - Analytics Self Signed Certificates Not Working in Resource Manager 6.3.2 Following urllib3 Update

Problem Description

Following the upgrade of Resource Manager to 6.3.2 (or on a fresh install), Resource Manager is unable to connect to Analytics due to the update of Python urllib3 library, from version 1.10.2 to version 1.22 (see the Resource Manager 6.3.2 release notes). This affects users using a self signed certificate.

Users may notice Analytics data not updating or receiving an Unable to talk to Analytics server flare when navigating in Resource Manager to Reports → Configuration. If attempting to submit new or amended configuration data from Resource Manager for Analytics, users may receive a Failed to connect to <Analytics Internal URL> error flare. The following example is what would be found in the zope event log:

 

2019-02-05T09:33:06 ERROR zen.etl.router Failed to connect to https://zenoss.example.com:443: Request 'https://zenoss.example.com:443/etl/502bc7a8-8450-11e8-817a-0242ac110018' failed: CERTIFICATE_VERIFY_FAILED certificate verify failed (_ssl.c:579)

 

Users are able to sign on to Analytics using Resource Manager authentication.  This only affects the Analytics ETL process.

How to verify urllib3 library is causing your issue

  • Navigate to Resource Manager → Configuration and confirm if you receive the error flare.
  • Amend a value of the configuration settings and submit. Verify the other message and check the zope logs:
    • for i in {0..5} ; do serviced service attach Zope/$i cat /opt/zenoss/log/event.log >> event.log ; done

  • Attach to a zope container and run the following (updating the URL, following is an example from my lab):
    • python -c "import urllib2;z = urllib2.urlopen('https://zenoss.example.com/');x = z.read()"

  • Following is a traceback which would confirm the issue:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
  • If nothing is returned then you're not impacted by the urllib3 issue.

Verify the Analytics certificate hasn't expired and that it has the correct common name

  • In a zope container, run either/both of the following two commands to check the expiry date of the Analytics certificate (updating to your Analytics URL):
    • curl -v https://zenoss.example.com:443

    • openssl s_client -connect zenoss.example.com:443 2>/dev/null | openssl x509 -noout -dates

  • The output would look like the following:
[root@7e0c9c8dfaf0 /]# curl -v https://zenoss.example.com:443
* About to connect() to zenoss.example.com port 443 (#0)
*   Trying 10.90.36.188...
* Connected to zenoss.example.com (10.90.36.188) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Server certificate:
* 	subject: E=root@zenoss.example.com,CN=zenoss.example.com,OU=SomeOrganizationalUnit,O=SomeOrganization,L=SomeCity,ST=SomeState,C=--
* 	start date: Jul 26 14:59:47 2017 GMT
* 	expire date: Jul 26 14:59:47 2018 GMT
* 	common name: zenoss.example.com
* 	issuer: E=root@zenoss.example.com,CN=zenoss.example.com,OU=SomeOrganizationalUnit,O=SomeOrganization,L=SomeCity,ST=SomeState,C=--
* NSS error -8181 (SEC_ERROR_EXPIRED_CERTIFICATE)
* Peer's Certificate has expired.
* Closing connection 0
curl: (60) Peer's Certificate has expired.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.


[root@7e0c9c8dfaf0 /]# openssl s_client -connect zenoss.example.com:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Jul 26 14:59:47 2017 GMT
notAfter=Jul 26 14:59:47 2018 GMT

Create a new Analytics certificate

  • On the Analytics host, create some new working directories:
    • mkdir /tmp/certUpdate
      mkdir /tmp/certUpdate/backup
  • Create the new certificate (this example sets the expiry date for 10 years from now):
    • openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /tmp/certUpdate/localhost.key -out /tmp/certUpdate/localhost.crt
  • Backup the old certificates:
    • mv /etc/pki/tls/certs/localhost.crt /tmp/certUpdate/backup/localhost.crt
      mv /etc/pki/tls/private/localhost.key /tmp/certUpdate/backup/localhost.key
  • Copy the new certificates over:
    • cp /tmp/certUpdate/localhost.key /etc/pki/tls/private/localhost.key
      cp /tmp/certUpdate/localhost.crt /etc/pki/tls/certs/localhost.crt
  • Restart http and the zenoss_analytics service:
    • systemctl restart httpd
      service zenoss_analytics stop
      service zenoss_analytics start
  • Rerun the checks to verify the expiry date has now been updated

Update the certificates for Resource Manager

  • Launch a zope shell:
    • serviced service shell -i -s fixAnalyticsCert zope bash
  • Define your URL (Replace 'zenoss.example.com' with your correct Analytics URL)
    • ANALYTICS=zenoss.example.com
  • Run the check to verify the certificate is still bad:
    • python -c "import urllib2;z = urllib2.urlopen('https://"${ANALYTICS}"');x = z.read()"

  • Navigate to the CA directory and create the certificate pem file:
    • cd /etc/pki/ca-trust/source/anchors/
      openssl s_client -showcerts -connect ${ANALYTICS}:443 </dev/null 2>/dev/null|openssl x509 -outform PEM > analyticsCertfile.pem
  • Update the certificate store (do not do by hand and please note there's no output):
    • update-ca-trust extract
  • Rerun the check to verify the certificate is okay now:
    • python -c "import urllib2;z = urllib2.urlopen('https://"${ANALYTICS}"');x = z.read()"

  • The above should not return anything.
  • Exit the container and commit the snapshot:
    • serviced snapshot commit fixAnalyticsCert
  • Delete the resulting snapshot tag:
    • serviced snapshot rm <tag>
  • Restart Resource Manager services:
    • serviced service restart zenoss.resmgr

 

Issue faced with Resource Manager 6.3.2 and Analytics v5.0.6 & v5.1.0

Related articles

 

https://access.redhat.com/articles/2039753#controlling-and-troubleshooting-certificate-verification-6

Was this article helpful?
3 out of 3 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    Louis McCurry

    Very well written.

Powered by Zendesk