Sporadic Site Slowness, 8/29

Last night (8/29) we experienced sporadic site slow-downs for many users from approximately 7:13 pm CDT to 10:29 pm CDT. This was caused by an unoptimized configuration setting on our servers which capped the number of users allowed to access the site at any one time. The issue has been resolved, and the site is now running faster than it ever has.

We're extending due dates by 24 hours for students who might have been affected by the slow-downs.

If you have any questions, comments, or concerns, please contact your TechTA (for instructors) or support@saplinglearning.com (for students). We apologize for any frustration the slowness may have caused.

Sapling Learning in Canada

We're pleased to announce that SaplingLearning.ca is now available for our friends north of the border. This server has the same functionality as SaplingLearning.com, but all information for the Canadian version is stored on servers physically located in Canada.

If you already use Sapling Learning at a Canadian school, your TechTA will let you know when to transition to the new server (most likely at the start of the next semester; if your course is already started for the summer, it won't happen just yet).

If you have any questions about the Canadian server, ask your TechTA, or let us know at casupport@saplinglearning.com.

Post Mortem: What Happened During the Amazon Outage

Thursday morning, 4/21/2011, at 5:26AM CDT, we intended to very briefly take our site down for a routine maintenance task. Had all gone as planned, the site would have been down for only a few seconds while we applied a patch to the front page of the site. Unfortunately, not everything went as planned.

We use the Amazon AWS cloud hosting service because they are very stable, and keep things running much better than any other hosting company we've worked with before. They also make multiple copies of every bit of data we save, which is normally a very good thing (it would take many many hard drives failing simultaneously for any student data to be lost). See more about our choice in this post.

That morning, at 2:47AM CDT, Amazon performed their own routine maintenance on their AWS cloud hosting service. They've posted their own post mortem, but it all boils down to a couple key points:

• During the maintenance, someone made a typo, pointing some of their servers to the wrong back up location. This caused the copying of the data to fail, so, each time something changed on a website, that website and the drives it was on locked up. It also caused the system that lets us restart our server to fail, causing our site to stay stuck offline during our few-second switch.
• Since the servers weren't able to reach the drives they thought they should reach while copying, they reported that something was wrong with those drives. If something is wrong with a drive, Amazon automatically takes that drive offline until they can inspect it and see what was wrong. Many, many drives were reported as broken, so all of those drives went offline. Amazon didn't have enough reserve capacity to handle this drive outage, causing their servers to run out of space.

We were able to get to our data, copy it to another server, and relaunch, but it took a long time for us to do so. Since we keep track of what students enter every time they answer, we have a very large amount of data, and just downloading it and uploading it takes several hours in each direction. We're working to make that less of an issue (see below).

Amazon is still working to make things more bulletproof, but they've already done a lot to prevent these problems from happening in the future. 

• They're implementing more automation and other safeguards to stop the typo from occurring in the first place.
• They've already added a lot of reserve capacity, and are adding more. My guess is hard drive salespeople in Virginia, where the servers are located, made a lot of money that day selling high-capacity drives to Amazon.

We are also taking our own steps to avoid similar issues in the future. By the Fall semester, we hope to mirror our data to multiple servers in both the US and Canada, so that, if another Amazon outage occurs, we can quickly move our site to one of these other locations with very minimal downtime.

We have heard of other homework systems being down for multiple days at a time, and we consider that absolutely unacceptable. It took Amazon going down to take us out for those 13 hours, and we don't plan to let that happen again.

Update on Site Availability

Sapling Learning has been unavailable for much of this morning, starting at about 6 AM Eastern to the present. Sapling is hosted on the Amazon Cloud, and the entire US-EAST-1 Region is experiencing connectivity issues. Sapling Learning, among thousands of other companies, has chosen to host our services on the Amazon Cloud for its excellent reliability and scalability. We are confident that Amazon will have their issues resolved quickly.

When the site is back up, your Technology TAs will grant an extension of 24 hours to any assignment that was due today. Please email your TechTA if you do not want the extension, or if you would prefer a longer extension for your classes.

We will post updates about Amazon's availability here hourly until this issue is resolved. Please visit this site for more details.

We apologize for the inconvenience that this has caused you and your students.

UPDATE (10:55 AM): We are still working to get the site back up. We apologize for the continued confusion. If your students have any questions, please forward them along to support@saplinglearning.com.

UPDATE (10:59 AM): Amazon has posted an explanation of the issue:

8:54 AM PDT We'd like to provide additional color on what were working on right now (please note that we always know more and understand issues better after we fully recover and dive deep into the post mortem). A networking event early this morning triggered a large amount of re-mirroring of EBS volumes in US-EAST-1. This re-mirroring created a shortage of capacity in one of the US-EAST-1 Availability Zones, which impacted new EBS volume creation as well as the pace with which we could re-mirror and recover affected EBS volumes. Additionally, one of our internal control planes for EBS has become inundated such that it's difficult to create new EBS volumes and EBS backed instances. We are working as quickly as possible to add capacity to that one Availability Zone to speed up the re-mirroring, and working to restore the control plane issue. We're starting to see progress on these efforts, but are not there yet. We will continue to provide updates when we have them.

UPDATE (11:39 AM): We're working on setting up the site on the USWEST half of the Amazon cloud, but it will take a while to get the very large amount of data we store transferred. Either through Amazon's fix or our own, we will have the site up as soon as possible. We will update you as soon as we have a time estimate.

UPDATE (1:01 PM): Amazon reports that they have made "significant progress" in stabilizing the cloud servers. We hope to have the issues resolved soon.

UPDATE (1:22 PM): Amazon estimates that, at worst, it will take a "few hours" for the servers to be back up. More information is available through Amazon's RSS feed, but we'll continue to also update here.

UPDATE (2:45 PM): Amazon is reporting some success of launching instances on the cloud servers. We hope to be back up soon.

UPDATE (3:46 PM): We're making progress, but the site is still down.

UPDATE (4:19 PM): We think we're getting very close.

UPDATE (5:01 PM): Amazon has brought up 3 of 4 zones in the USEAST-1 region. We, unforunately, are on the fourth zone. We are continuing to try to work around this outage, and to put safeties in place to avoid extensive outages like this in the future.

UPDATE (6:22 PM): There is a lot of activity from the server guys right now, but we are not disturbing them to get details. It looks like it's getting very close.

UPDATE (6:35 PM): We're up! We pride ourselves on our 99.95% uptime, and apologize for today's frustrations.