Friday, April 29, 2011

Post Mortem: What Happened During the Amazon Outage

Thursday morning, 4/21/2011, at 5:26AM CDT, we intended to very briefly take our site down for a routine maintenance task. Had all gone as planned, the site would have been down for only a few seconds while we applied a patch to the front page of the site. Unfortunately, not everything went as planned.

We use the Amazon AWS cloud hosting service because they are very stable, and keep things running much better than any other hosting company we've worked with before. They also make multiple copies of every bit of data we save, which is normally a very good thing (it would take many many hard drives failing simultaneously for any student data to be lost). See more about our choice in this post.

That morning, at 2:47AM CDT, Amazon performed their own routine maintenance on their AWS cloud hosting service. They've posted their own post mortem, but it all boils down to a couple key points:

  • During the maintenance, someone made a typo, pointing some of their servers to the wrong back up location. This caused the copying of the data to fail, so, each time something changed on a website, that website and the drives it was on locked up. It also caused the system that lets us restart our server to fail, causing our site to stay stuck offline during our few-second switch.
  • Since the servers weren't able to reach the drives they thought they should reach while copying, they reported that something was wrong with those drives. If something is wrong with a drive, Amazon automatically takes that drive offline until they can inspect it and see what was wrong. Many, many drives were reported as broken, so all of those drives went offline. Amazon didn't have enough reserve capacity to handle this drive outage, causing their servers to run out of space.
We were able to get to our data, copy it to another server, and relaunch, but it took a long time for us to do so. Since we keep track of what students enter every time they answer, we have a very large amount of data, and just downloading it and uploading it takes several hours in each direction. We're working to make that less of an issue (see below).

Amazon is still working to make things more bulletproof, but they've already done a lot to prevent these problems from happening in the future. 
  • They're implementing more automation and other safeguards to stop the typo from occurring in the first place.
  • They've already added a lot of reserve capacity, and are adding more. My guess is hard drive salespeople in Virginia, where the servers are located, made a lot of money that day selling high-capacity drives to Amazon.
We are also taking our own steps to avoid similar issues in the future. By the Fall semester, we hope to mirror our data to multiple servers in both the US and Canada, so that, if another Amazon outage occurs, we can quickly move our site to one of these other locations with very minimal downtime.

We have heard of other homework systems being down for multiple days at a time, and we consider that absolutely unacceptable. It took Amazon going down to take us out for those 13 hours, and we don't plan to let that happen again.

Thursday, April 21, 2011

Sapling's Cloud Computing Choice

Sapling Learning, and indeed much of the Internet, has been greatly affected by the current failures within the Amazon Elastic Compute Cloud (EC2). We are very troubled by the outage and the issues that it is creating for professors and students that use our site. We at Sapling wanted to share with you what the Amazon "cloud" is and why Sapling chose to host our site within Amazon's EC2 services.

Amazon EC2 is a cloud web service that provides resizable computing capacity, and allows companies to easily configure their web services in response to load. This easy scalability, combined with Amazon's proven uptime record, is why many prominent businesses, including Netflix, Eli Lilly, Autodesk, Ericsson, Yelp, PBS, and ShareThis, have migrated their Web sites and software to EC2. NASA uses Amazon's servers to process telemetry data from the Mars rovers. Amazon EC2 Service Level Agreement guarantees an Annual Uptime Percentage of 99.95%, and has performed well beyond this measure. 

In 2010, at the same time as Netflix, Sapling began moving our Web hosting to Amazon EC2. Stability and scalability were our main reasons for doing so. Until this month, Sapling was very proud of our 99.99% uptime. To put this in perspective, in 2009 Gmail had only a 99.90% reliability rate, with its stated goal to reach 99.99%. In March 2010, Twitter had 99.74% uptime. Up until today, we have been very happy with our decision, especially considering that our previous hosting company has been down at least two times since we moved to Amazon EC2, and we had not had any issues.

Unfortunately, Sapling and many other technology companies are affected by the current EC2 failure. These include leading Internet companies with millions of users such as Foursquare, Reddit, Hootsuite, and Quora, and because of the pervasive nature of this outage across the Internet, the Amazon failure has been prominently featured in the national news.

We are currently working parallel paths to resolve the solution, one of these will assure restoration of Sapling's site for tomorrow's classes in case Amazon is not able to get its cloud services issues fixed before then. Once we are back up and running, we will then begin to implement a strategy that will guard against even this unprecedented cloud downtime. 

Update on Site Availability

Sapling Learning has been unavailable for much of this morning, starting at about 6 AM Eastern to the present. Sapling is hosted on the Amazon Cloud, and the entire US-EAST-1 Region is experiencing connectivity issues. Sapling Learning, among thousands of other companies, has chosen to host our services on the Amazon Cloud for its excellent reliability and scalability. We are confident that Amazon will have their issues resolved quickly.

When the site is back up, your Technology TAs will grant an extension of 24 hours to any assignment that was due today. Please email your TechTA if you do not want the extension, or if you would prefer a longer extension for your classes.

We will post updates about Amazon's availability here hourly until this issue is resolved. Please visit this site for more details.

We apologize for the inconvenience that this has caused you and your students.

UPDATE (10:55 AM): We are still working to get the site back up. We apologize for the continued confusion. If your students have any questions, please forward them along to support@saplinglearning.com.

UPDATE (10:59 AM): Amazon has posted an explanation of the issue:
8:54 AM PDT We'd like to provide additional color on what were working on right now (please note that we always know more and understand issues better after we fully recover and dive deep into the post mortem). A networking event early this morning triggered a large amount of re-mirroring of EBS volumes in US-EAST-1. This re-mirroring created a shortage of capacity in one of the US-EAST-1 Availability Zones, which impacted new EBS volume creation as well as the pace with which we could re-mirror and recover affected EBS volumes. Additionally, one of our internal control planes for EBS has become inundated such that it's difficult to create new EBS volumes and EBS backed instances. We are working as quickly as possible to add capacity to that one Availability Zone to speed up the re-mirroring, and working to restore the control plane issue. We're starting to see progress on these efforts, but are not there yet. We will continue to provide updates when we have them.
UPDATE (11:39 AM): We're working on setting up the site on the USWEST half of the Amazon cloud, but it will take a while to get the very large amount of data we store transferred. Either through Amazon's fix or our own, we will have the site up as soon as possible. We will update you as soon as we have a time estimate.

UPDATE (1:01 PM): Amazon reports that they have made "significant progress" in stabilizing the cloud servers. We hope to have the issues resolved soon.

UPDATE (1:22 PM): Amazon estimates that, at worst, it will take a "few hours" for the servers to be back up. More information is available through Amazon's RSS feed, but we'll continue to also update here.

UPDATE (2:45 PM): Amazon is reporting some success of launching instances on the cloud servers. We hope to be back up soon.

UPDATE (3:46 PM): We're making progress, but the site is still down.

UPDATE (4:19 PM): We think we're getting very close.

UPDATE (5:01 PM): Amazon has brought up 3 of 4 zones in the USEAST-1 region. We, unforunately, are on the fourth zone. We are continuing to try to work around this outage, and to put safeties in place to avoid extensive outages like this in the future.

UPDATE (6:22 PM): There is a lot of activity from the server guys right now, but we are not disturbing them to get details. It looks like it's getting very close.

UPDATE (6:35 PM): We're up! We pride ourselves on our 99.95% uptime, and apologize for today's frustrations.

Tuesday, April 12, 2011

Sapling Learning for High School

Sapling Learning is pleased to announce online homework and problem-solving practice for high-school Biology, Chemistry, Physics, and IPC aligned to TEKS and STAAR EOC. Learn more at hs.saplinglearning.com. Additionally, we will know in a matter of months whether we've been approved by the TEA, which means that Texas high schools will be able to use TEA allocated funds toward the purchase of Sapling Learning for their classes.

Monday, March 28, 2011

Sapling's Digital Content Ecosystem

This week at the ACS Spring Meeting, we are introducing our new general chemistry eBook, Chemistry 5e by John Olmsted and Greg Williams. This book will be available under our new microcontent publishing model. We will be giving two talks at this conference:

  • Tuesday, 9:15 AM, Disney's Grand California Hotel, Trillium B: Microcontent publishing: A new model for development and dissemination of educational support materials (p. TECH-99 of the on-site program, talk 1340)
  • Tuesday, 3:30 PM - 6:00 PM, Anaheim Convention Center, Room 211A: Independent Online Homework and eBooks (a workshop on editing and earning royalties for content in our online homework and eBook system) (p. 29 of the on-site program)

For those of you unable to attend our sessions, this blog gives a brief overview of how things work.

If you have content you'd like to make available to your students, and you think instructors at other schools might like to use that same content, we'd like to let you get compensated for that effort. For questions, we'll pay a royalty based on the percentage of all questions in a course that you provide. In short, if you provide 50% of the questions used in a course, you'll get 50% of the available question royalty. Similarly, for eBook content, we'll pay a royalty based on the percentage of all eBook sections used in each course. The exact percentages can scale depending how much of the work you do, but the idea is to make it easy for people to mix-and-match content to fit their needs.

If you have content that you think would fit this model, let Brian Arneson know at publish@saplinglearning.com. We'll update the blog over the coming weeks with more details about what this means for your courses and how you can take advantage of this model, both as an author and as an instructor. Let us know in the comments if you have any questions you'd like us to answer in those posts.

Wednesday, March 9, 2011

Sapling Learning v1.15 Release Notes

At the end of this week we're releasing a huge update to SaplingLearning.com. Along with some bug fixes and other improvements, the entire look of the site is changing. We realize it might be a bit of a shock to change the look this dramatically in the middle of the semester, but everything will still be where it always has been (not to mention that there really isn't a time that isn't "in the middle of the semester" for at least one course on our site).

The new look extends throughout the site, and we're hoping it will make everything clearer and easier to use.

In addition to the new look, we're releasing several improvements.

Activities (aka Assignments):
  • Help With This Topic Drawer: We've added a drawer to the Activity Sidebar, containing links to resources related to the topic of the question you're viewing. When these resources are available and resources are enabled for your course, they will appear in this drawer. For example, Marc Loudon's Organic Chemistry textbook is available in organic chemistry courses, with the appropriate sections linked from each question.
  • Activity Sizing: Activities will now properly size to fit your browser window.
  • Sidebar Hiding: You can now collapse the sidebar to give you more space for the assignment itself in small windows.
  • Back-end Improvements: The database structure in which we store activities has been greatly improved. The improvements may not be immediately obvious, but we can now continue to improve the speed at which activities save and load. 
Activity Editing:
  • Improved Conflict Resolution: It is possible, under certain conditions, for the activity editor to get "out of sync," for example if multiple people are editing the assignment. We've improved the checks the activity editor does when you make changes to eliminate the issues this can cause.
  • Authoring Bug Fixes: For those of you who write content for your courses, we hope we've tracked down the last of the issues that cause you to have to log out and back in sometimes while editing your content. If you continue to have issues while authoring, please let me know. 
  • Cleaner Extension Interface: Adding due-date extensions for groups of students should now be considerably less confusing.
Anywhere Activities:
  • Introducing Anywhere Activities: Last month I mentioned that we had a new feature in the works to allow Sapling Learning assignments to be embedded on any webpage. This feature is now ready. We do not yet have an instructor interface implemented to give you the code to add to your page to use these, but, if you have a use for them, let me know and we can get you started. The instructor interface for creating these links (similar to embedding a YouTube video) will be available in a future release.
  • Instructor Editing of Anywhere Activities: When you embed an Anywhere Activity on your site, if you're logged in with your instructor account, you'll be able to edit that activity wherever you have it embedded (rather than returning to SaplingLearning.com to edit it). You'll also be able to jump right to the gradebook or assignment statistics to see how your students are doing on that activity.
User Accounts and Login:
  • Facebook Connect Account Creation: Students can now use Facebook Connect to quickly create their accounts using their information from Facebook (such as name and location).
  • Facebook Connect Login: Students (and instructors) can also connect their Sapling Learning accounts to their Facebook accounts, speeding up the login process.
If you need help sorting out any of the new features, your TechTA should be able to walk you through them, or ask us in person at the Spring 2011 ACS National Meeting, March 27-31 in Anaheim, California.

    Monday, February 28, 2011

    Data Mining: Student Performance by OS

    As a follow-up to last week's post, I decided to take a look at student performance by operating system. This data comes from a single assignment in one large course, to try to eliminate other variables. Within this course, 173 students had logged in during the time frame of the assignment exclusively from Windows machines, and 193 students had logged in exclusively from Macs.
    On average, the Mac-using students scored 6.2 points (out of 100) higher than the Windows-using students. Many students (about 10%) had logged in from both Mac and Windows machines, so I added those to the chart.
    The dual-OS users scored even higher than their Mac-exclusive fellow-students, scoring 3.2 points higher than Mac users, and 9.4 points higher than Windows users.

    While it might be tempting to surmise from this data that Mac users are smarter than Windows users (and, given the attention we expect such a finding would generate, we welcome that over-simplification in posts about this blog), I strongly suspect any effect we're seeing here is simply a measure of socio-economic background. Macs tend to be more expensive than Windows machines, and socio-economic background has often been found to correlate with performance.

    We plan to take another look at these numbers on a larger scale (and with a more complete statistical analysis), but this initial data seemed interesting enough to share.