As of 6:29pm we have ensured any failed sends were re-initiated, all error logs reviewed and additional rules setup for internal notices if there are any further issues.
We are marking this incident as fully resolved at this point, but we are also adding additional remediation measures to our email sending systems to ensure the risk for this type of outage is greatly minimized.
Thanks again for your patience and understanding as we continue to improve Nucleus for everyone.
Posted Aug 02, 2022 - 05:21 UTC
Monitoring
As promised, some more updates: 1. Email sending is now re-enabled for the Nucleus (and RebelGive) platform 💜🥳 2. All emails in the queue that were supposed to be sent out (but did not due to the outage) were successfully sent. 💌
Next Steps: 1. We are now working through the logs to ensure that any lingering "failed" emails that should have been sent (but did not initially as the outage started to occur) get found, restored, and retried now that the outage has been fully resolved. We will post an update once that has been completed. 2. We are taking additional steps to implement additional systems to protect us from similar outages in the future, implementing additional error monitoring, and levels of protection that will enhance everyone's experience with Nucleus Messages (and the rest of our email sending system).
Thanks again for your patience as we addressed the issue this morning.
We will post another update once the issue is fully closed out and mark as resolved.
Posted Aug 01, 2022 - 21:08 UTC
Update
The underlying issue has been resolved, and the path forward has been cleared. Email sending system is not re-enabled just yet, but will be soon.
First, we are taking some additional precautionary steps to review in full before reactivating email sending from the Nucleus platform.
After we do that, we will resolve (allow sending) for any scheduled emails that were skipped, paused, or missed during the outage.
Another update coming soon.
Posted Aug 01, 2022 - 20:53 UTC
Update
NOTE: No emails scheduled (or receipt emails etc.) have been lost. Once the issue is resolved, we will be able to re-queue missed emails to send so the system catches up with intended deliveries.
Posted Aug 01, 2022 - 18:25 UTC
Identified
We identified an issue this AM that is currently preventing all email campaigns in Nucleus Messages from sending, as well as preventing Flow confirmation emails and receipts from sending.
This also briefly affected password reset email requests but that has been remedied with a temporary solution to ensure those continue sending in the meantime as we work to resolve the underlying issue.
We will continue to keep this incident updated as we make progress resolving the issue.