April 1st Login Event: After Action Report
by, 04-03-2009 at 12:02 PM (1003 Views)
Well, I'd like to say that the event was a resounding success and we blew that 430 record right out of the water, but technology stepped in and really gave us a whipping. I do think that we would have made it to 430, honestly. Just one minute before the designated start time at 6pm, we were already at 211 registered members online and somewhere in the neighborhood of 60+ guests. That would put us within about 140 logins from what we needed.
In a cruel twist of fate, at exactly 6pm, the site promptly died a horrible death as, I am guessing, many who were already online refreshed the webpage simultaneously and, I suspect, many more were trying to login for the beginning of the event. Ironically, I think the hardware was reasonably up for the challenge, but an overlooked configuration blew the entire operation.
For you techies out there who will understand this, the maximum connections for mySQL was not set in the configuration file. As a result, it defaulted to 151 maximum simultaneous connections. That wasn't enough to accommodate a few hundred people trying to make a mySQL connection at the exact same time, so it created a huge cascading failure as people got database errors and then furiously tried to refresh the page. I was logged into the backend server at the time, and you could even see that the CPUs were not overly taxed (30% usage,) but the load was insane at between 400 and even all the way up to 900 at times.
All that aside, I still count this event as a success. It was really an awesome thing to see a bunch of P&PG'rs come together online and to watch the shoutbox chat go wild even a couple of hours leading up to the event. I do want to apologize to all of the folks who weren't able to get their names on the list and tried to login only to find the site utterly decimated from 6:00pm Pacific to about 6:35pm. It wasn't an April Fools joke, it was just a low mySQL skill-check on my part that was at fault.
Nonetheless, we did get a confirmed 233 members out for the event, which I think is pretty awesome. If you did try to login and weren't able to vote on this poll. I'd love to get a feel for just how many more members we might have had if the server hadn't crashed.
A number of members have asked -- will we try this again? I think the answer is a definite Yes. As to when, I'm not sure -- perhaps next quarter and certainly before the end of the year. In the meantime, I would like to do more site wide events on a bit smaller scale. I've been thinking that we could do some round table type chat events. If anyone has any ideas, send them my way.
To all of the many members who came out to support the site at this event, let me again extend my gratitude. And if you were one of the many folks who didn't get their names on the log because of the server crash, we've put up a public poll that will let you add your name to the list here: http://www.penandpapergames.com/forums/showthread.php?t=9684.
- Robert A. Howard