Known Issue: ANA-365 - Analytics Performance batches failing since year rollover

Problem Description

Zenoss Engineering has recently identified an issue with Analytics.  The dim_date table contains incorrect settings that will cause performance ETL batches to fail after 1 January 2020.  This issue affects all users of Zenoss Analytics and the following steps should be taken immediately.

Diagnosis

To determine if you are currently experiencing this issue, you can check the contents of the zenoss_analytics.out file on your Analytics server.  Beginning after 1 January 2020, you may see messages like the following:

WARN pool-105-thread-4 org.hibernate.util.JDBCExceptionReporter - SQL Error: 1064, SQLState: 42000
ERROR pool-105-thread-4 org.hibernate.util.JDBCExceptionReporter - You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'NULL' at line 1
Query is:
call createPerfTable(@_jdbc_var_1,@_jdbc_var_2,@_jdbc_var_3)

This failure will also generate ERROR files in the Analytics temp directory.

Remediation

1. Download the attached ANA-365-fix.tar.gz file to your Analytics server and place it in the /opt/zenoss_analytics/bin directory.

2. Extract the contents of ANA-365-fix.tar.gz with tar -xzvf ANA-365-fix.tar.gz.

3.Switch to the zenoss user (su - zenoss) and change to the newly created /opt/zenoss_analytics/bin/analytics_fix/ directory.

4. Execute the fix_perf_batch_issues.sh script with ./fix_perf_batch_issues.sh

This script runs the following (included) scripts in order, and cleans up any ERROR files in the Analytics temp directory:

set_analytics_env.sh - Discovers and sets Analytics-specific environment variables required by the other scripts.
step1.sql - Drops and recreates the dim_date table of the Analytics reporting database.
step2.sql - Adds additional columns to the dim_date table and populates it.
retry_all_failed_perf_batches.sh - Attempts to retry perf batches that failed due to ANA-365.

4. Due to the number and size of missed batches, the final step may take some time to complete.  The status of the batches can be tracked in the Analytics UI to determine when they have finished.