Sporadic Site Slowness, 8/29

Last night (8/29) we experienced sporadic site slow-downs for many users from approximately 7:13 pm CDT to 10:29 pm CDT. This was caused by an unoptimized configuration setting on our servers which capped the number of users allowed to access the site at any one time. The issue has been resolved, and the site is now running faster than it ever has.

We're extending due dates by 24 hours for students who might have been affected by the slow-downs.

If you have any questions, comments, or concerns, please contact your TechTA (for instructors) or support@saplinglearning.com (for students). We apologize for any frustration the slowness may have caused.

Sapling Learning v1.18 Release Notes

Last week, we released several new features to both the US and Canadian higher-education servers. A lot of the changes in this update were behind-the-scenes, but these are the things you might notice:

Activities

• Instructor Quick-Edit on Activities and Due Dates: Instructors can now edit all of the settings (except "Description") for every assignment in your course at once from the "Activities and Due Dates" page. This should be helpful if you want to change the policies for several assignments at once, for example. We also added a "Save All" button on that page to commit all of your updates (the changes you made don't "stick" until you click the individual save icon for each one or the "Save All" button at the bottom).

Course Settings

• Subject and Term: We now explicitly track the Subject and Term (1st semester of  2, 2nd quarter of 3, etc) for courses, so that we can offer better statistics in future updates.

Users/Logins

• Name Changes: Users can now request name changes after account creation. We changed this to a by-request system to help combat a way we knew might work for cheating purposes, but which it doesn't look like any students had discovered; now they'll never get a chance.
• User Institution Settings: On sign-up, we now ask users to choose their Institution (college, university, etc) instead of to tell us their City, State, and Country. We're putting this in place to allow us to streamline the account-creation and course-selection process for the Spring. We'll also start suggesting the appropriate timezone to users based on their institution to help avoid a rare but annoying issue (when students select the wrong timezone and are therefore told the wrong time for due dates).
• Cleaner Login: The Canada server received the Facebook login fixes and the updated login-box look.

Payment/Enrollment

• New Payment System: We completed overhauled the back-end to handle payments in our system. This will allow us to make several improvements (such as the multi-course discounts described below). The thing you're most likely to encounter right now is that we can upload rosters to enroll students in a course even with for-pay courses now.
• Easier Refunds: We updated our refund system to make it easier for us (or even instructors) to grant refunds to students. After we get a few more safeguards in place, we'll also be able to open up self-directed refunds to students (so they'll be able to request a refund within 60 days of purchase and receive the refund immediately).
• Grace Periods: Students now have two weeks to use most courses for free before they have to pay. We hope this will make things easier for students who later drop the course (they can simply use the course without paying if they aren't sure, then only pay when they've made their final decision).
• Multi-Course Discounts: We can now offer a discount for students who purchase multiple quarters or semesters of a course at once (for example, for $29.99 courses, students can purchase two semesters at $49.99 and save $9.99).

Let us know at feedback@saplinglearning.com if you have any comments about these or any other features, or especially if you have an idea for a neat new feature. We're constantly developing new features, and we have a very long list of things we'd like to do, but it never hurts to hear what people want.

Sapling Learning v1.17 Release Notes

We released a minor update this morning, continuing our data restructuring and fixing a few small issues. The most noticeable change (for those of you who use Firefox 4) is that you can now set due-date extensions in Firefox 4.

We will have a much larger release in about a month, but we do not currently anticipate significant downtime for that update.

Sapling Learning v1.16 Release Notes

This morning we released another update to the site. This one wasn't as noticeable as our last release, but we're hoping you'll find it helpful. Here's what it included.

Activities

• Web Help & Videos Drawer: We've added another drawer to the Activity Sidebar, containing a web search tailored to the subject of the assignment (general chemistry, biochemistry, etc) and the topic of the question (stoichiometry, SN1 reactions, etc). If we don't have anything specific in our Help With This Topic drawer, hopefully the web can help students out. This drawer is disabled if you disable resources for the assignment.
• Better Data Structure: This won't be visible yet, but we've made another improvement to the way we store student responses for assignments. This will allow us to streamline some things in the future (and speed up load times), and to produce much more detailed reports about how students performed on assignments. Once we finish crunching the existing data into this new format (it's happening in the background over the next week or two), we'll show you a preview of what this new format will allow us to show you.

User Accounts and Login:

• Facebook Connect Improvements: We've made our Facebook Connect integration easier to use. This should clear up any problems users have had using Facebook to log into the site. Note: We don't share any information with Facebook, we only use it to make creating accounts and logging in faster and easier.

Course Settings:

• Course Institutions: We've added a formal "Institution" setting to the settings for each course. This will be invisible to you (your TechTA set it already), but it will allow us to make it easier soon for students to find your course.

Gradebook:

• Better Scrolling: We removed the extra horizontal scrollbar in the gradebook, letting the scrollbar at the bottom of your browser window do the work. It looks a little ugly right now if your gradebook extends into the right margin, but we wanted to get the functionality in place right away. We'll clean up around the edges in an upcoming release.

In addition to those changes, we made several minor fixes to the new site, and a few behind-the-scenes bug fixes.

We'll have another major release around July, introducing several big changes for the Fall.

Sapling Learning in Canada

We're pleased to announce that SaplingLearning.ca is now available for our friends north of the border. This server has the same functionality as SaplingLearning.com, but all information for the Canadian version is stored on servers physically located in Canada.

If you already use Sapling Learning at a Canadian school, your TechTA will let you know when to transition to the new server (most likely at the start of the next semester; if your course is already started for the summer, it won't happen just yet).

If you have any questions about the Canadian server, ask your TechTA, or let us know at casupport@saplinglearning.com.

Post Mortem: What Happened During the Amazon Outage

Thursday morning, 4/21/2011, at 5:26AM CDT, we intended to very briefly take our site down for a routine maintenance task. Had all gone as planned, the site would have been down for only a few seconds while we applied a patch to the front page of the site. Unfortunately, not everything went as planned.

We use the Amazon AWS cloud hosting service because they are very stable, and keep things running much better than any other hosting company we've worked with before. They also make multiple copies of every bit of data we save, which is normally a very good thing (it would take many many hard drives failing simultaneously for any student data to be lost). See more about our choice in this post.

That morning, at 2:47AM CDT, Amazon performed their own routine maintenance on their AWS cloud hosting service. They've posted their own post mortem, but it all boils down to a couple key points:

• During the maintenance, someone made a typo, pointing some of their servers to the wrong back up location. This caused the copying of the data to fail, so, each time something changed on a website, that website and the drives it was on locked up. It also caused the system that lets us restart our server to fail, causing our site to stay stuck offline during our few-second switch.
• Since the servers weren't able to reach the drives they thought they should reach while copying, they reported that something was wrong with those drives. If something is wrong with a drive, Amazon automatically takes that drive offline until they can inspect it and see what was wrong. Many, many drives were reported as broken, so all of those drives went offline. Amazon didn't have enough reserve capacity to handle this drive outage, causing their servers to run out of space.

We were able to get to our data, copy it to another server, and relaunch, but it took a long time for us to do so. Since we keep track of what students enter every time they answer, we have a very large amount of data, and just downloading it and uploading it takes several hours in each direction. We're working to make that less of an issue (see below).

Amazon is still working to make things more bulletproof, but they've already done a lot to prevent these problems from happening in the future. 

• They're implementing more automation and other safeguards to stop the typo from occurring in the first place.
• They've already added a lot of reserve capacity, and are adding more. My guess is hard drive salespeople in Virginia, where the servers are located, made a lot of money that day selling high-capacity drives to Amazon.

We are also taking our own steps to avoid similar issues in the future. By the Fall semester, we hope to mirror our data to multiple servers in both the US and Canada, so that, if another Amazon outage occurs, we can quickly move our site to one of these other locations with very minimal downtime.

We have heard of other homework systems being down for multiple days at a time, and we consider that absolutely unacceptable. It took Amazon going down to take us out for those 13 hours, and we don't plan to let that happen again.

Sapling's Cloud Computing Choice

Sapling Learning, and indeed much of the Internet, has been greatly affected by the current failures within the Amazon Elastic Compute Cloud (EC2). We are very troubled by the outage and the issues that it is creating for professors and students that use our site. We at Sapling wanted to share with you what the Amazon "cloud" is and why Sapling chose to host our site within Amazon's EC2 services.

Amazon EC2 is a cloud web service that provides resizable computing capacity, and allows companies to easily configure their web services in response to load. This easy scalability, combined with Amazon's proven uptime record, is why many prominent businesses, including Netflix, Eli Lilly, Autodesk, Ericsson, Yelp, PBS, and ShareThis, have migrated their Web sites and software to EC2. NASA uses Amazon's servers to process telemetry data from the Mars rovers. Amazon EC2 Service Level Agreement guarantees an Annual Uptime Percentage of 99.95%, and has performed well beyond this measure. 

In 2010, at the same time as Netflix, Sapling began moving our Web hosting to Amazon EC2. Stability and scalability were our main reasons for doing so. Until this month, Sapling was very proud of our 99.99% uptime. To put this in perspective, in 2009 Gmail had only a 99.90% reliability rate, with its stated goal to reach 99.99%. In March 2010, Twitter had 99.74% uptime. Up until today, we have been very happy with our decision, especially considering that our previous hosting company has been down at least two times since we moved to Amazon EC2, and we had not had any issues.

Unfortunately, Sapling and many other technology companies are affected by the current EC2 failure. These include leading Internet companies with millions of users such as Foursquare, Reddit, Hootsuite, and Quora, and because of the pervasive nature of this outage across the Internet, the Amazon failure has been prominently featured in the national news.

We are currently working parallel paths to resolve the solution, one of these will assure restoration of Sapling's site for tomorrow's classes in case Amazon is not able to get its cloud services issues fixed before then. Once we are back up and running, we will then begin to implement a strategy that will guard against even this unprecedented cloud downtime.