Seventh Week Site Update
Taking this coming weekend off, so checking in with an early edition. Still, we shipped 83 changes over a little more than five days.
The short version: seven more reported bugs are fixed, including recurring events that looked broken, old forum links that dead-ended, and a Rich Text switch that refused to let people leave HTML mode. The image rescue is finished: every surviving hotlinked image in the site’s twenty-year history (291,959) is now stored on our own servers. We also load-tested the site to find out how many of us it can actually hold. And the unread-badge timeouts from last week have not recurred.
Seven Reported Bugs, Fixed
As always, every one of these started with a member taking the time to write it up.
-
Recurring events “no longer worked.” This was an implementation issue – repeats could take up to six hours to appear, because the only thing that created them was a background job that ran four times a day. The reporter checked two minutes after saving a daily event and saw one occurrence and nothing else, which is indistinguishable from broken. Repeats now appear the moment you save, on all three calendars: game, community, and personal reminders.
-
Old forum links inside posts went to “Page Not Found.” Links like
showthread.php?p=…from the vBulletin days looked like missing content, but nothing was missing — the server knows exactly where every one of those posts lives now and redirects correctly. The problem was that the page intercepted the click before it ever reached the server. Fixed across all 17 million posts at once, without rewriting a single one of them. -
“Let players add calendar events” didn’t let players add calendar events. A game has two calendar views, and only one of them honored the setting — and it wasn’t the Calendar tab, which is the one players are more likely to use. With the setting on, players now get the New Event button, can click an empty day to create one, and can edit events they created themselves.
-
Switching a post to Rich Text was blocked with “would lose 60% of text.” It would have lost nothing. Out-of-character popovers and fieldset titles get stored slightly differently in Rich Text, and the safety check was counting those words on one side of the comparison and not the other. The estimate is now accurate and if the warning does fire, it’s a warning you can override rather than a wall. If you’d turned mode-switch warnings off, you were hitting this with no explanation at all; you now switch cleanly.
-
Tables with merged header cells lost their column widths. A carefully tuned 15-column character sheet came back with most of its columns squashed to the same narrow width. When a table’s top row had merged cells, the editor measured the wrong number of columns and filled the rest with a default. It now measures the most detailed row instead.
-
One empty field blanked an entire template. A single unfilled placeholder — most often a spell slot on its own line — made the whole template render as nothing. An empty field now leaves just its own spot empty.
- The same report mentioned templates showing old values until you closed every Myth-Weavers tab. That was real too: saving from the sheet viewer never told the post editor anything had changed, so closing all your tabs was genuinely the only way to refresh it. Sheet saves now announce themselves, and the editor picks up new values straight away.
-
Posting a character statblock pasted raw BBcode into the post. The sheet viewer’s Generate Statblock button produces BBcode-formatted text, but since the move to Idunn it was arriving in posts as unformatted source. A statblock inserted into a post now arrives as formatted, editable text. In a postbit or content template it stays linked to your sheet and is rebuilt from your current numbers every time you post. While we were in there, we found that the Dresden Files and Anime D20 statblock generators had never worked at all, and fixed those too. The Post Editor guide now has a section on inserting a statblock, including which of the two behaviors you’ll get where.
The Image Rescue Is Finished
Last week we said the second stage — 2015 to today — would be a 40–60 hour job. It ran just over 62 hours, and it’s done. Across the whole of the site’s history:
- 291,959 images preserved, 57 GB in total, now served from our own storage and out of reach of other websites’ clean-up schedules.
- 184,917 unable to be recovered — the hot-linked server has deleted them, the domain disappeared, or the site blocks hotlinking. There was nothing left to save. This includes 22,309 that might have been recoverable, though it’s likely that a site showing as “temporarily down” for 10+ year old content is probably actually dead.
Again, thank you to all our supporters whose contributions ensure we can afford to preserve your games for the long haul.
How Many Of Us Can The Site Hold?
We ran a load test against the live site on the evening of September 7 and the morning of September 8, to find out where the ceiling is. If the site felt sluggish for a few minutes during either one, that was our tests; each step was cut off the moment it crossed our slowness limit. No requests failed, but they may have taken a few extra seconds.
The good news is Idunn’s architecture allows us to deliver more capability with less resources. Even at the busiest time, we have roughly 70% headroom against our DB (the limiting case) and web servers (even less strained). Right now we’re running a peak of 240 users with no sweat; 400 would be easily doable without having to scale up.
The test also showed that the web server pool was adding a third machine during short traffic spikes, a scale-up that didn’t help because the server wasn’t available until after the spike had already passed. We’ve raised that threshold, and the pointless start-and-stop has disappeared completely in the days since.
Security Housekeeping
- We ran our first security scan of the new site, both signed out and signed in. We fixed all the real gaps: the signed-out pages were missing some protective browser headers the signed-in pages had, and a few small files had none at all. All fixed, with an automated check so the two sets can’t drift apart again.
- The signed-in scan also found a latent caching issue. Some per-member information, like your unread counts, didn’t tell caches “this is private, don’t store it.” Nothing stores it today, so nothing leaked, but it would have mattered the day we put a content delivery network in front of the site (which is on the roadmap). Fixed before that could happen.
- Sign-in now has a second layer of protection against password guessing. Repeated failed sign-ins from one network address get slowed down. It counts only failed attempts, and it’s based on where the attempts come from, never on the account. Nobody can lock you out of your own account by deliberately failing against your email address.
- The signup captcha has worked. We measured before building an automatic “registration raid” breaker, and found that since the captcha went live on August 6 there hasn’t been a single hour that looks like a raid. We’re keeping the design in our hip pocket in case we need to break it out later.
Loose Ends From Last Week
- The unread-badge timeouts have stopped. There were 21 of these errors on September 4 and 5. In the five and a half days since the fix shipped, there have been zero.
- Mark All Read in the very largest games is still not fixed. We said last week it fails above roughly two thousand threads. The same fix took the largest game from over 90 seconds to about 36–40, but it’s still more than the 30-second limit query limit. The remaining cost is a housekeeping issue on the posts table, and the fix is a maintenance pass we’ll schedule carefully to limit any effects on your gaming.
- Search on common words is still slow, and we now know why. We shipped three search improvements over the past month, and none of them moved the needle on timeout rate. This week’s measurements found the reason: most of the time goes to ranking the matches, which means reading every matching post’s stored search data, and no index can shorten that. A search for a common word on a play-by-post site matches an enormous number of posts. That points us toward a dedicated search engine rather than more tuning, and we’re planning it now.
Under The Floorboards
- Blades in the Dark sheets are officially approved. Thanks to the folks at Evil Hat, and specifically the legendary John Harper, we can now offer the 17 Blades in the Dark sheet templates with their original character-sheet artwork. We’ve also reached out to other publishers to begin the same process so we can deliver on your requests; more to follow.
- Moderators can now change a member’s email address. If you’ve lost access to the email you signed up with, a moderator can now sort it out, instead of it having to wait for an admin.
- We’re getting better at seeing how new members get started. Anonymous, aggregate measurements of a new member’s first week helped us confirm how they are reaching and exploring the site. We’ll be using this to improve the onboarding experience, with the goal that we help them find their way to your tables. This update also unearthed a counting bug of our own: starting a new topic wasn’t counted as posting, which on a play-by-post site is quite a bit of the posting. Fixed.
Every Site Update So Far
-
Site Updates — Feature Freeze & the Road to Cutover, June 26 to June 30
-
This announcement
What’s Next
-
If you reported one of the seven bugs, check it. If it’s still wrong, say so on the same report — a reopened report is the summoning signal that we missed the mark.
-
Reset any table whose columns got squashed. The fix stops it happening again, but a table that was already saved with collapsed widths keeps them until you set the widths once more.
-
Try a recurring event again if you gave up on them. The repeats now appear straight away. (A “repeats daily” label on the event itself is still to come.)
-
Keep filing. Seven of this week’s fixes came straight from your reports. Telling us you’re having a problem by posting it in the issue tracker is the fastest way for us to know about it, so we can prioritize fixing what’s broken.
Keep on weaving those myths!