Stages not Sending completely from March 27 to April 11th.

Incident Report for Retention Science

Resolved

Date Started: March 27th
Date Resolved: April 11th

Cause of Incident:

A Migration/Upgrade to MySql v8 per Amazon AWS requirements occured on March 26th.
When the migration happened there was a small bug in the code/query around NULL date values. This caused rows in our AI predictions output to go missing. Any row that had a date or timestamp or a 0 or null or date outside the normal range would cause that row to be removed entirely from the output. This was something the new version of sql was sensitive to that the old version of sql was not.

How we will prevent this from happening in future:

A Database migration is not a frequent occurrence, something that happens once every 4 years so so.
We will do a better job in QAing by involving the Client Success team to QA a few of their largest clients to make sure no anomalies are occurring with the stages and engagement.

Posted Apr 11, 2024 - 11:00 PDT