Analytics and JavaScript delays
Incident Report for Retention Science
Resolved
This incident has been resolved.
Posted Nov 25, 2020 - 23:20 PST
Update
AWS is experiencing an outage that is impacting much of its North America presence: https://techcrunch.com/2020/11/25/amazon-web-services-outage-takes-a-portion-of-the-internet-down-with-it/

This began impacting ReSci clients as of about 6AM PST. We are seeing delays and incorrect statistics / counts in our analytics dashboards as well as a delay in some morning sends.

Our main Cortex dashboard, SMS, data science APIs, events APIs, template tools, and other APIs remain operational as expected.

Here is an update on the affected components.

Promo Blasts:
Operational
As of 1PM PST, all promo blasts that were delayed from this morning (from ~6AM PST) have been sent out.

Realistic Test Sends:
Degraded performance
A small percentage of test-sends are timing out. This is resulting from the system delays we experienced this morning.

Click tracking:
Operational

Open Tracking:
Degraded performance
Some “open” events are missing from analytics and filtering from approximately ~6AM PST until now.

JS Events:
Degraded performance
Some events are missing from analytics and filtering from approximately ~6AM PST until now.

Segment sync to FB and Google
Degraded performance
We are experiencing issues with segments being automatically synced to Google and Facebook.
Posted Nov 25, 2020 - 16:22 PST
Monitoring
Although there is no longer an issue with emails sending, the outage with our infrastructure provider is causing delays in analytics for sends since this morning. Sends prior to today should reflect accurate reporting, and once the outage is resolved we will backfill reports to reflect data accurately. JavaScript tracking of onsite events is also impacted and thus page views/carts/checkouts are currently missing and will be backfilled once the outage is resolved. Open and click tracking are working as expected. We expect a full recovery to take a few hours and will continue to monitor and update as we learn more.
Posted Nov 25, 2020 - 14:27 PST
This incident affected: Data Imports Service and Reporting.