From Steven Chan, Goodnotes founder and CEO:
To our Goodnotes community,
I know that many of you rely on Goodnotes for your studies, work, and daily life. That's why I wanted to personally address and apologize to you directly about the service disruption on October 23 that affected some of you.
Simply put, the whole team and I consider downtime like this that could affect your access to your account and notes unacceptable, and we are committed to holding ourselves to a higher standard.
Today, I'd like to share directly with you what happened and what we are doing to better ensure you can always access your notes.
What happened and how we responded
Starting at 00:53 UTC on October 23, we experienced an outage that resulted in some users not being able to sign in or stay signed in to their Goodnotes app. This was primarily due to issues relating to our user management backend system, which handles user sign-in and authentication. The initial trigger was a set of API load tests that inadvertently affected the user management system.
By 01:11 UTC, we had detected issues and mobilized our team to investigate and address. In parallel, our support team was informed and began sharing guidance on the Help Center and in response to support inquiries.
By 02:31 UTC, we had resolved these initial issues, and all users were able to sign in and stay signed in.
While the ability to sign in was restored from that point forward, from 03:39 UTC, related resource allocation bugs began affecting some users. Signing in could be delayed, and a subsequent loading screen was unresponsive, requiring manual dismissal. At the sign-in screen, affected users did not have their default sign-in option indicated.
Additionally, some users who newly signed in during this period did not have their paid plan reflected in their Goodnotes app (though those who were signed in already were not impacted). As these particular issues specifically affected a limited number of users, we did not discover them until 05:11 UTC. We immediately worked to resolve them, and they were fully fixed by 07:50 UTC.
The ability to make new purchases was also affected during this time, including new paid subscriptions, one-time payment plans, or Marketplace purchases. From 03:39 UTC onwards, purchase attempts were gradually more likely to be declined, until 06:00 UTC onwards when all purchase attempts began to be declined. This was resolved by 10:25 UTC.
All these issues related to sign-in, account status, and purchases, remained fully resolved from that point onwards. However, as affected users may have had knock-on problems, our support and communications teams have continued to work 24/7 to address these issues since. To date, our support team has processed thousands of associated support requests, providing the first response to each enquiry well under 4 hours on average.
The impact on users
The most important aspect to me is how users were impacted in this incident, which everyone at Goodnotes takes extremely seriously. The key points are:
- From 00:53 to 02:31 UTC, some users experienced delays or were unable to sign in, including not being able to access their account and documents.
- From 03:39 to 07:50 UTC, some users who newly signed in did not have their paid plan reflected in their app, restricting note creation and premium features.
- From 03:39 to 07:50 UTC, some users were shown all the “sign in with” options, rather than their default account method, resulting in a portion inadvertently creating a new account rather than signing in to their original one. This led to confusion around access to subscriptions, purchases, and documents associated with their original account.
- From 03:39 to 07:50 UTC, some users trying to sign in experienced delays of up to several seconds and also subsequently a loading screen that was stuck and needed to be dismissed manually.
- From 03:39 to 10:25 UTC, some users were not able to make new purchases, including new paid plans or Marketplace purchases.
- Across this period, some users deleted and reinstalled Goodnotes in an attempt to resolve the issue themselves. A portion of them had data that was not fully synced or backed up for various reasons, resulting in loss of that data.
The sign-in outage did not itself directly impact any account data, including documents. The privacy and security of user data remained protected, as this was not a hack or security breach. However, I know that if you faced the situations mentioned above, you may have found it concerning, disruptive, or stressful. We do not take this lightly.
Our commitment to improvement
Your ability to create and save notes, then access them when you need to, is the very foundation of Goodnotes. It should never be in doubt.
This incident has provided hard earned insights about where we can improve the resilience and reliability of our systems. It also underlined how user experience and customer communications play an essential role.
We are taking concrete steps to better guard against disruptions to your experience:
- Making our app more resilient to backend system disruptions: We want to ensure the impact to your Goodnotes experience is minimized when there are issues with our backend systems, including your ability to access your documents.
- Alerting users when data is not synced or backed up properly: We will improve visibility around backup status and options for users, including through reminders and alerts. We will still maintain the option of only using local storage for those who prefer it.
- Improving backend engineering coverage: To help ensure we are always prepared to quickly identify and remediate any backend issues, we are dialing up how much coverage and on-call availability we have around the clock, across time zones.
- Being more prepared to identify and respond to incidents: We’re strengthening our monitoring and alert systems, testing procedures, customer support processes and capabilities, and communication channels to better prepare for the unlikely event of any disruptions.
- Reducing risk of users inadvertently creating new accounts: We understand that users may not always remember which sign-in method or credentials they used to set up their Goodnotes account. We already had planned work to improve the sign-in experience to address this, but are now prioritizing it.
These are just some of our immediate concrete steps. Overall, the whole Goodnotes team is committed to learning and improving from this incident and its impacts, and we will be taking many more actions to come.
Thank you
On behalf of the entire Goodnotes team, I want to again extend our apologies for this incident. We understand the stress and inconvenience you may have faced, especially for those of you who were unable to access critical information when you needed it most. And for anyone who inadvertently deleted some notes or documents in the process of deleting and reinstalling the app, we’re truly sorry. We value the trust you place in us, and we are committed to earning it anew through our actions and improvements.
I also want to thank everyone in Goodnotes, from engineering to operations to customer support, for their tireless work around the clock over the past few days to address this incident, mitigate its impact, and aid our users.
If you're still experiencing any issues related to this incident, please don't hesitate to reach out to our support team at support-incident-2024-10-23@goodnotes.com. We've set this up as a temporary dedicated channel to address any outstanding concerns promptly. If you are already in contact with our support team, you can continue communicating through that existing channel.
Thank you for your patience and continued support. My team and I are wholly committed to always delivering a reliable, secure, and seamless Goodnotes experience that you can count on.
Regards,
Steven