tag:status.retentionscience.com,2005:/historyRetention Science Status - Incident History2024-03-27T19:49:41-07:00Retention Sciencetag:status.retentionscience.com,2005:Incident/198866842024-02-01T12:52:03-08:002024-02-01T12:52:03-08:00Slowness navigating the website, delays sending emails..<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>12:52</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>09:50</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue for immediate repair</p>tag:status.retentionscience.com,2005:Incident/161673952023-02-16T09:00:00-08:002023-02-16T10:32:45-08:00Email Sends Limited<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>09:00</var> PST</small><br><strong>Resolved</strong> - We have identified an issue with our send pipeline causing extremely low throughput, delaying most sends. Only customer accounts using our proprietary SMTP (accounts migrated off of Sendgrid) have been affected. Our team is actively working to restore system performance.</p>tag:status.retentionscience.com,2005:Incident/87839392021-12-07T22:40:19-08:002021-12-07T22:40:19-08:00AWS Outage<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>22:40</var> PST</small><br><strong>Resolved</strong> - This incidence has been resolved. All systems are back online and email sends will resume. Previously scheduled sends should be verified and rescheduled if needed.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>09:55</var> PST</small><br><strong>Monitoring</strong> - There is a widespread AWS outage affecting several components of Cortex. There may be issues loading pages, setting up emails, and sending campaigns during this time. We are monitoring the issue closely and will notify you when systems are back online.</p>tag:status.retentionscience.com,2005:Incident/71892722021-06-07T11:12:36-07:002021-06-07T11:12:36-07:00Click/Open Data Missing in AI Stats<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>11:12</var> PDT</small><br><strong>Resolved</strong> - A patch has been put into place and this incident is now resolved. <br /><br />It may take up to 24 hours for the system to update and reflect stats correctly.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>09:42</var> PDT</small><br><strong>Identified</strong> - Campaign performance metrics are temporarily missing for some accounts - our team is aware of the issue and will update with a resolution soon. Email sends are not impacted.</p>tag:status.retentionscience.com,2005:Incident/71495952021-06-02T11:51:05-07:002021-06-02T11:51:05-07:00Intermittent 502 errors for email clicks<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>11:51</var> PDT</small><br><strong>Resolved</strong> - This incident has now been resolved.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>10:27</var> PDT</small><br><strong>Identified</strong> - We have identified an intermittent issue caused by our click tracking service resulting in some click throughs erroring out and some clicks not properly getting tracked. Sends are NOT impacted. <br /><br />Our team is pushing an update shortly.</p>tag:status.retentionscience.com,2005:Incident/58361732020-12-18T11:58:06-08:002020-12-18T11:58:06-08:00Delayed sends for emails with greater than 5k recipients<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>11:58</var> PST</small><br><strong>Resolved</strong> - This issue has now been resolved. All backed up emails will begin sending and all remaining emails will go out as planned.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>11:27</var> PST</small><br><strong>Identified</strong> - We are experiencing an issue that is preventing emails from going out for list sizes greater than 5000 recipients. We have identified the issue and are currently working on a solution.</p>tag:status.retentionscience.com,2005:Incident/56572362020-11-25T23:20:23-08:002020-11-25T23:20:23-08:00Analytics and JavaScript delays<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>23:20</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>16:22</var> PST</small><br><strong>Update</strong> - AWS is experiencing an outage that is impacting much of its North America presence: https://techcrunch.com/2020/11/25/amazon-web-services-outage-takes-a-portion-of-the-internet-down-with-it/<br /><br />This began impacting ReSci clients as of about 6AM PST. We are seeing delays and incorrect statistics / counts in our analytics dashboards as well as a delay in some morning sends.<br /><br />Our main Cortex dashboard, SMS, data science APIs, events APIs, template tools, and other APIs remain operational as expected.<br /><br />Here is an update on the affected components.<br /><br />Promo Blasts:<br />Operational<br />As of 1PM PST, all promo blasts that were delayed from this morning (from ~6AM PST) have been sent out.<br /><br />Realistic Test Sends:<br />Degraded performance<br />A small percentage of test-sends are timing out. This is resulting from the system delays we experienced this morning. <br /><br />Click tracking:<br />Operational<br /><br />Open Tracking:<br />Degraded performance<br />Some “open” events are missing from analytics and filtering from approximately ~6AM PST until now. <br /><br />JS Events:<br />Degraded performance<br />Some events are missing from analytics and filtering from approximately ~6AM PST until now. <br /><br />Segment sync to FB and Google<br />Degraded performance<br />We are experiencing issues with segments being automatically synced to Google and Facebook.</p><p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>14:27</var> PST</small><br><strong>Monitoring</strong> - Although there is no longer an issue with emails sending, the outage with our infrastructure provider is causing delays in analytics for sends since this morning. Sends prior to today should reflect accurate reporting, and once the outage is resolved we will backfill reports to reflect data accurately. JavaScript tracking of onsite events is also impacted and thus page views/carts/checkouts are currently missing and will be backfilled once the outage is resolved. Open and click tracking are working as expected. We expect a full recovery to take a few hours and will continue to monitor and update as we learn more.</p>tag:status.retentionscience.com,2005:Incident/56551232020-11-25T12:10:41-08:002020-11-25T12:10:41-08:00Analytics and Cortex Delays<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>12:10</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>09:25</var> PST</small><br><strong>Identified</strong> - We have identified an issue with our infrastructure provider since approximately 5:30AM PST that is causing delays in analytics and causing a subset of sends to experience delays. We are carefully monitoring and scaling to address the immediate issue.</p>tag:status.retentionscience.com,2005:Incident/54726882020-11-02T11:23:16-08:002020-11-02T11:23:16-08:00File Manager returns error when adding or updating an image<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>11:23</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>09:51</var> PST</small><br><strong>Identified</strong> - When attempting to add or update an image in the template builder tool, the user is brought to the File Manager portion of the builder which throws an error. Attempting to retry results in the same error message being displayed. <br /><br />This does not affect email sends or reporting - only the ability to upload images when creating templates.</p>tag:status.retentionscience.com,2005:Incident/53166482020-10-14T10:02:18-07:002020-10-14T10:02:18-07:00SMS - Webhooks for Shopify integration outage<p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>10:02</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:23</var> PDT</small><br><strong>Investigating</strong> - We are experiencing an outage with our webhooks for the shopify integration specifically for SMS. Shopify webhooks for email are currently functioning as expected.<br /><br />We are working on a fix. This outage only impacts segmentation capabilities based on shopify attributes. It doesn't impact deliverability of automated or one time campaign sends or capturing leads through modals or any other list building flow.</p>tag:status.retentionscience.com,2005:Incident/52863472020-10-09T06:56:42-07:002020-10-09T06:56:42-07:00Delay in Promo Blast Campaigns<p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>06:56</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved. The scope of this issue affects Promo blasts that were scheduled between 4-6:30AM PST. Any blasts that may have been delayed have now been queued for sending.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>05:51</var> PDT</small><br><strong>Investigating</strong> - We are investigating a network connectivity issue that is causing a delay in the send pipeline. Promo blasts scheduled are being processed but may experience delay until this issue has been resolved. We are actively working through this and will update as soon as we can.</p>tag:status.retentionscience.com,2005:Incident/50182372020-09-04T12:01:26-07:002020-09-04T12:01:26-07:00Cortex Filters and Partial Delay in Sends<p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>12:01</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>10:05</var> PDT</small><br><strong>Identified</strong> - We've identified an issue causing degraded performance for some Cortex filters. This is causing Cortex sends and promo blasts to be delayed for a subset of clients. We are actively working with our database provider to resolve this as soon as possible.<br /><br />Analytics pages may also experience delayed metrics. Our metrics are regenerating and will be fully updated by end of day.</p>tag:status.retentionscience.com,2005:Incident/49140942020-08-22T22:33:56-07:002020-08-22T22:33:56-07:00Cortex Filters and Partial Delay in Sends<p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>22:33</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>12:06</var> PDT</small><br><strong>Identified</strong> - We've identified an issue causing degraded performance for some Cortex filters. This is causing Cortex sends and promo blasts to be delayed for a subset of clients. We are actively working with our database provider to resolve this as soon as possible.<br /><br />Analytics pages may also experience delayed metrics. Our metrics are regenerating and will be fully updated by end of day.</p>tag:status.retentionscience.com,2005:Incident/48400312020-08-12T11:53:47-07:002020-08-12T11:53:47-07:00Intermittent access to Cortex<p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>11:53</var> PDT</small><br><strong>Resolved</strong> - This issue is now resolved.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>11:15</var> PDT</small><br><strong>Investigating</strong> - Some pages within Cortex are intermittently unavailable, but fortunately sends are not affected. We are investigating the issue and expect to have a fix in place soon.</p>tag:status.retentionscience.com,2005:Incident/46836312020-07-22T12:47:30-07:002020-07-22T12:47:30-07:00Intermittent access to Cortex<p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>12:47</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved and full access to Cortex has been restored</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>11:43</var> PDT</small><br><strong>Identified</strong> - We are experiencing intermittent issues with Cortex. Access to Cortex may be limited during this time. We have identified the source issue and expect a fix to be in place momentarily.</p>tag:status.retentionscience.com,2005:Incident/46759022020-07-21T14:41:04-07:002020-07-21T14:41:04-07:00Issue Saving Segments<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>14:41</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>14:11</var> PDT</small><br><strong>Investigating</strong> - We are currently experiencing an issue when trying to save segments. We are in the process of identifying the source issue and will send an update once we know more.</p>tag:status.retentionscience.com,2005:Incident/46147952020-07-14T15:38:04-07:002020-07-14T15:38:04-07:00Delay in transactional sends<p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>15:38</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>14:52</var> PDT</small><br><strong>Investigating</strong> - We are investigating an issue causing delay of transactional sends from our Transactional sends API.</p>tag:status.retentionscience.com,2005:Incident/44085702020-06-19T08:00:50-07:002020-06-19T08:00:50-07:00Slow loading Cortex pages<p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>08:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>07:18</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an issue causing slow loading pages or browser timeout errors for a subset of accounts. This may occur when accessing specific Cortex pages including the Smart Segments or during Email Creation.</p>tag:status.retentionscience.com,2005:Incident/41041372020-05-14T12:05:19-07:002020-05-14T12:05:19-07:00Sporadic Error Appearing in Template Editor<p><small>May <var data-var='date'>14</var>, <var data-var='time'>12:05</var> PDT</small><br><strong>Resolved</strong> - This issue is now resolved.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>12:09</var> PDT</small><br><strong>Identified</strong> - We are seeing a sporadic error in the template editor stating "Errors on BEE Plugin configuration" which is preventing the editing and saving of templates. Logging out and logging back in is a simple workaround as a fix is being put into place. We have identified the source issue and will have it resolved shortly.</p>tag:status.retentionscience.com,2005:Incident/39552012020-04-24T11:44:06-07:002020-04-24T11:44:06-07:00AWS Issue Delaying Sends/Reporting<p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>11:44</var> PDT</small><br><strong>Resolved</strong> - The issue as been resolved and all sends and reporting have caught up. <br /><br />Please note that no sends were dropped and only some sends experienced a delay.<br /><br />Please reach out to help@retentionscience.com or your Customer Success Manager to review and discuss any impact to your specific account.</p><p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>10:43</var> PDT</small><br><strong>Identified</strong> - Due to an issue with AWS, our send capacity is being temporarily reduced resulting in some delays in sends and reporting.<br /><br />We have identified the source issue and expect to have a resolution shortly.</p>tag:status.retentionscience.com,2005:Incident/38112402020-04-18T07:00:35-07:002020-04-18T07:00:35-07:00Drag & Drop Templating Tool Maintenance<p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>07:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>05:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>10:59</var> PDT</small><br><strong>Scheduled</strong> - On Saturday, April 18, 2020, between 5:00 AM and 7:00 AM PDT, we will be performing several important updates on the drag & drop template system, which will be temporarily unavailable during this time.<br /><br />Although the window is 2 hours, actual downtime is expected to be around 30 minutes and we will make every effort to ensure the work is performed as quickly as possible.</p>tag:status.retentionscience.com,2005:Incident/38594132020-04-09T12:46:44-07:002020-04-09T12:46:44-07:00Reporting Partially Degraded<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>12:46</var> PDT</small><br><strong>Resolved</strong> - This issue has now been fixed. This was due to a server related issue that we had between 2PM PST on 4/7 and 2PM PST on 4/8. During that period of time, we were double counting Email Sent numbers on the reporting page, however actual send counts were NOT affected. The numbers you see for “Emails Sent” will return to normal over the next 5 days. Please reach out to help@retentionscience.com for any further questions.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>12:19</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>12:18</var> PDT</small><br><strong>Identified</strong> - The AI Stats page is currently showing a much larger amount of emails than are actually being sent out. The root cause has been identified and is currently being fixed. We will send an update when the fix has been deployed.</p>tag:status.retentionscience.com,2005:Incident/35271982020-02-05T13:38:35-08:002020-02-05T13:38:35-08:00Cortex is not accessible<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>13:38</var> PST</small><br><strong>Resolved</strong> - This issue has been resolved, and Cortex is accessible again. The root cause was a temporary database issue. We apologize for any inconvenience.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>13:23</var> PST</small><br><strong>Investigating</strong> - Cortex is currently not accessible. We are investigating and will have an update shortly.</p>tag:status.retentionscience.com,2005:Incident/32590252019-11-26T14:03:58-08:002019-11-26T14:03:58-08:00Intermittent Cortex access issues<p><small>Nov <var data-var='date'>26</var>, <var data-var='time'>14:03</var> PST</small><br><strong>Resolved</strong> - Access issues have been resolved. This was a temporary network issue, and additional monitoring checks have been added to several components of the system.</p><p><small>Nov <var data-var='date'>26</var>, <var data-var='time'>13:49</var> PST</small><br><strong>Investigating</strong> - We are experiencing intermittent issues with accessing the Cortex platform. You may see a 503 error when trying to access pages. The issue is currently under investigation and we'll have updates for you shortly.</p>tag:status.retentionscience.com,2005:Incident/30699392019-10-04T23:14:51-07:002019-10-04T23:14:51-07:00Routine Database Maintenance<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>23:14</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>23:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>17:53</var> PDT</small><br><strong>Scheduled</strong> - We will be doing routine maintenance on a subset of our database clusters. During this time you may experience temporary issues accessing your Cortex dashboard or with specific API calls. All scheduled campaigns will go out as expected, immediately after our maintenance is complete.</p>