Post-Maintenance Update

[Eric]

Summary

For approximately 17 hours from 2024-02-10 at 12:00 PST until 2024-02-11 at 05:00 PST, some Myth-Weavers systems were intermittently available while the dev team conducted maintenance affecting back-end database infrastructure. We estimate that 150 new sheets created between 16:00 PST and 19:00 PST on 2024-02-10 may be affected by data loss. See below for more details. All maintenance tasks were completed. All systems are currently available, and the root cause of the data loss has been eliminated.

Details

Starting Saturday, 2024-02-10 at 10:00 PST, we evaluated action items to address risk in our infrastructure based on current indicators. After weighing the opportunities and expected downtime against the continued risk, we opted to take immediate action. This decision was made with the knowledge that we had not previously announced a maintenance window, but that the entire dev team was available throughout the weekend (a rare occurrence) to identify and rapidly resolve any contingencies.

At 12:00 PST we began urgent site maintenance to address infrastructure risks posed by coupling with the legacy OGMW database system. We had previously rehearsed and performed part of this maintenance and expected it to last 30-60 minutes, in which OGMW would be down and sheets would be in read-only mode. The migration seemed to go smoothly, but subsequent performance issues revealed undetected errors in the new database structure, which had been not previously been used in read-write mode. This error degraded data access performance and the ability to add new information in some cases.

Continued actions to identify, mitigate, and resolve issues resulted in intermittent availability of most Myth-Weavers systems until 2024-02-10 at 16:00 PST.

While troubleshooting performance issues observed after the initial maintenance effort, our team pointed the sheets service on new Myth-Weavers back to the OGMW database to bring up sheets in read-only mode for user convenience. Unfortunately, this incorrect mapping was left in place for approximately three hours after the sheet services were placed in read-write mode. During that time, both sheet services were in operation but were pointed to different databases until the legacy database was turned off. With the legacy database turned off, the incorrect mapping became apparent, and we immediately repaired the issue.

Between 16:00 and 19:00 PST:

• If you created a new sheet using the new Myth-Weavers sheets UI, that sheet is unfortunately lost. We estimate this affects about 100 sheets.
• If you edited an existing sheet using the new Myth-Weavers sheets UI, changes to the sheet name or sharing were not applied, but changes to the sheet data were applied correctly.
• If you created a new sheet using the OGMW sheets UI, the sheet contents may have been overwritten due to a conflict from the other system. We estimate this may affect up to 50 sheets.

After overnight monitoring, final actions to retire the legacy systems and resume normal operation for the OGMW forums were completed on 2024-02-11 at 05:00 PST.

We apologize for any inconvenience.