Problem Description
Following the upgrade of Resource Manager to 6.3.2 (or on a fresh install), Resource Manager is unable to connect to Analytics due to the update of Python urllib3
library, from version 1.10.2 to version 1.22 (see the Resource Manager 6.3.2 release notes). This affects users using a self signed certificate.
Users may notice Analytics data not updating or receiving an Unable to talk to Analytics server flare when navigating in Resource Manager to Reports → Configuration. If attempting to submit new or amended configuration data from Resource Manager for Analytics, users may receive a Failed to connect to <Analytics Internal URL> error flare. The following example is what would be found in the zope event log:
2019-02-05T09:33:06 ERROR zen.etl.router Failed to connect to https://zenoss.example.com:443: Request 'https://zenoss.example.com:443/etl/502bc7a8-8450-11e8-817a-0242ac110018' failed: CERTIFICATE_VERIFY_FAILED certificate verify failed (_ssl.c:579)
Users are able to sign on to Analytics using Resource Manager authentication. This only affects the Analytics ETL process.
How to verify urllib3 library is causing your issue
- Navigate to Resource Manager → Configuration and confirm if you receive the error flare.
- Amend a value of the configuration settings and submit. Verify the other message and check the zope logs:
-
for i in {0..5} ; do serviced service attach Zope/$i cat /opt/zenoss/log/event.log >> event.log ; done
-
- Attach to a zope container and run the following (updating the URL, following is an example from my lab):
-
python -c "import urllib2;z = urllib2.urlopen('https://zenoss.example.com/');x = z.read()"
-
- Following is a traceback which would confirm the issue:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
- If nothing is returned then you're not impacted by the urllib3 issue.
Verify the Analytics certificate hasn't expired and that it has the correct common name
- In a zope container, run either/both of the following two commands to check the expiry date of the Analytics certificate (updating to your Analytics URL):
-
curl -v https://zenoss.example.com:443
-
openssl s_client -connect zenoss.example.com:443 2>/dev/null | openssl x509 -noout -dates
-
- The output would look like the following:
[root@7e0c9c8dfaf0 /]# curl -v https://zenoss.example.com:443
* About to connect() to zenoss.example.com port 443 (#0)
* Trying 10.90.36.188...
* Connected to zenoss.example.com (10.90.36.188) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* Server certificate:
* subject: E=root@zenoss.example.com,CN=zenoss.example.com,OU=SomeOrganizationalUnit,O=SomeOrganization,L=SomeCity,ST=SomeState,C=--
* start date: Jul 26 14:59:47 2017 GMT
* expire date: Jul 26 14:59:47 2018 GMT
* common name: zenoss.example.com
* issuer: E=root@zenoss.example.com,CN=zenoss.example.com,OU=SomeOrganizationalUnit,O=SomeOrganization,L=SomeCity,ST=SomeState,C=--
* NSS error -8181 (SEC_ERROR_EXPIRED_CERTIFICATE)
* Peer's Certificate has expired.
* Closing connection 0
curl: (60) Peer's Certificate has expired.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
[root@7e0c9c8dfaf0 /]# openssl s_client -connect zenoss.example.com:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Jul 26 14:59:47 2017 GMT
notAfter=Jul 26 14:59:47 2018 GMT
Create a new Analytics certificate
- On the Analytics host, create some new working directories:
-
mkdir /tmp/certUpdate mkdir /tmp/certUpdate/backup
-
- Create the new certificate (this example sets the expiry date for 10 years from now):
-
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /tmp/certUpdate/localhost.key -out /tmp/certUpdate/localhost.crt -addext "basicConstraints = CA:TRUE"
-
- Backup the old certificates:
-
mv /etc/pki/tls/certs/localhost.crt /tmp/certUpdate/backup/localhost.crt mv /etc/pki/tls/private/localhost.key /tmp/certUpdate/backup/localhost.key
-
- Copy the new certificates over:
-
cp /tmp/certUpdate/localhost.key /etc/pki/tls/private/localhost.key cp /tmp/certUpdate/localhost.crt /etc/pki/tls/certs/localhost.crt
-
- Restart http and the zenoss_analytics service:
-
systemctl restart httpd service zenoss_analytics stop service zenoss_analytics start
-
- Rerun the checks to verify the expiry date has now been updated
Update the certificates for Resource Manager
- Launch a zope shell:
serviced service shell -i -s fixAnalyticsCert zope bash
- Define your URL (Replace '
zenoss.example.com
' with your correct Analytics URL)- ANALYTICS=
zenoss.example.com
- ANALYTICS=
- Run the check to verify the certificate is still bad:
-
python -c "import urllib2;z = urllib2.urlopen('https://"${ANALYTICS}"');x = z.read()"
-
- Navigate to the CA directory and create the certificate pem file:
-
cd /etc/pki/ca-trust/source/anchors/ openssl s_client -showcerts -connect ${ANALYTICS}:443 </dev/null 2>/dev/null|openssl x509 -outform PEM > analyticsCertfile.pem
-
- Update the certificate store (do not do by hand and please note there's no output):
update-ca-trust extract
- Rerun the check to verify the certificate is okay now:
-
python -c "import urllib2;z = urllib2.urlopen('
https://"${ANALYTICS}"'
);x = z.read()"
-
- The above should not return anything.
- Exit the container and commit the snapshot:
serviced snapshot commit fixAnalyticsCert
- Delete the resulting snapshot tag:
serviced snapshot rm <tag>
- Restart Resource Manager services:
serviced service restart zenoss.resmgr
Issue faced with Resource Manager 6.3.2 and Analytics v5.0.6 & v5.1.0
Related articles
Very well written.
Hi ,
Was going through AWS document and found that current CA certificate going to expire 5th-Feb-2020
We are planning to update the RDS instance certificate from rds-ca-2015 to rds-ca-2019 before 5th Feb 2020
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html.
Is there any impact?