Sapling Learning v1.16 Release Notes

This morning we released another update to the site. This one wasn't as noticeable as our last release, but we're hoping you'll find it helpful. Here's what it included.

Activities

• Web Help & Videos Drawer: We've added another drawer to the Activity Sidebar, containing a web search tailored to the subject of the assignment (general chemistry, biochemistry, etc) and the topic of the question (stoichiometry, SN1 reactions, etc). If we don't have anything specific in our Help With This Topic drawer, hopefully the web can help students out. This drawer is disabled if you disable resources for the assignment.
• Better Data Structure: This won't be visible yet, but we've made another improvement to the way we store student responses for assignments. This will allow us to streamline some things in the future (and speed up load times), and to produce much more detailed reports about how students performed on assignments. Once we finish crunching the existing data into this new format (it's happening in the background over the next week or two), we'll show you a preview of what this new format will allow us to show you.

User Accounts and Login:

• Facebook Connect Improvements: We've made our Facebook Connect integration easier to use. This should clear up any problems users have had using Facebook to log into the site. Note: We don't share any information with Facebook, we only use it to make creating accounts and logging in faster and easier.

Course Settings:

• Course Institutions: We've added a formal "Institution" setting to the settings for each course. This will be invisible to you (your TechTA set it already), but it will allow us to make it easier soon for students to find your course.

Gradebook:

• Better Scrolling: We removed the extra horizontal scrollbar in the gradebook, letting the scrollbar at the bottom of your browser window do the work. It looks a little ugly right now if your gradebook extends into the right margin, but we wanted to get the functionality in place right away. We'll clean up around the edges in an upcoming release.

In addition to those changes, we made several minor fixes to the new site, and a few behind-the-scenes bug fixes.

We'll have another major release around July, introducing several big changes for the Fall.

Sapling Learning in Canada

We're pleased to announce that SaplingLearning.ca is now available for our friends north of the border. This server has the same functionality as SaplingLearning.com, but all information for the Canadian version is stored on servers physically located in Canada.

If you already use Sapling Learning at a Canadian school, your TechTA will let you know when to transition to the new server (most likely at the start of the next semester; if your course is already started for the summer, it won't happen just yet).

If you have any questions about the Canadian server, ask your TechTA, or let us know at casupport@saplinglearning.com.

Post Mortem: What Happened During the Amazon Outage

Thursday morning, 4/21/2011, at 5:26AM CDT, we intended to very briefly take our site down for a routine maintenance task. Had all gone as planned, the site would have been down for only a few seconds while we applied a patch to the front page of the site. Unfortunately, not everything went as planned.

We use the Amazon AWS cloud hosting service because they are very stable, and keep things running much better than any other hosting company we've worked with before. They also make multiple copies of every bit of data we save, which is normally a very good thing (it would take many many hard drives failing simultaneously for any student data to be lost). See more about our choice in this post.

That morning, at 2:47AM CDT, Amazon performed their own routine maintenance on their AWS cloud hosting service. They've posted their own post mortem, but it all boils down to a couple key points:

• During the maintenance, someone made a typo, pointing some of their servers to the wrong back up location. This caused the copying of the data to fail, so, each time something changed on a website, that website and the drives it was on locked up. It also caused the system that lets us restart our server to fail, causing our site to stay stuck offline during our few-second switch.
• Since the servers weren't able to reach the drives they thought they should reach while copying, they reported that something was wrong with those drives. If something is wrong with a drive, Amazon automatically takes that drive offline until they can inspect it and see what was wrong. Many, many drives were reported as broken, so all of those drives went offline. Amazon didn't have enough reserve capacity to handle this drive outage, causing their servers to run out of space.

We were able to get to our data, copy it to another server, and relaunch, but it took a long time for us to do so. Since we keep track of what students enter every time they answer, we have a very large amount of data, and just downloading it and uploading it takes several hours in each direction. We're working to make that less of an issue (see below).

Amazon is still working to make things more bulletproof, but they've already done a lot to prevent these problems from happening in the future. 

• They're implementing more automation and other safeguards to stop the typo from occurring in the first place.
• They've already added a lot of reserve capacity, and are adding more. My guess is hard drive salespeople in Virginia, where the servers are located, made a lot of money that day selling high-capacity drives to Amazon.

We are also taking our own steps to avoid similar issues in the future. By the Fall semester, we hope to mirror our data to multiple servers in both the US and Canada, so that, if another Amazon outage occurs, we can quickly move our site to one of these other locations with very minimal downtime.

We have heard of other homework systems being down for multiple days at a time, and we consider that absolutely unacceptable. It took Amazon going down to take us out for those 13 hours, and we don't plan to let that happen again.

Sapling's Cloud Computing Choice

Sapling Learning, and indeed much of the Internet, has been greatly affected by the current failures within the Amazon Elastic Compute Cloud (EC2). We are very troubled by the outage and the issues that it is creating for professors and students that use our site. We at Sapling wanted to share with you what the Amazon "cloud" is and why Sapling chose to host our site within Amazon's EC2 services.

Amazon EC2 is a cloud web service that provides resizable computing capacity, and allows companies to easily configure their web services in response to load. This easy scalability, combined with Amazon's proven uptime record, is why many prominent businesses, including Netflix, Eli Lilly, Autodesk, Ericsson, Yelp, PBS, and ShareThis, have migrated their Web sites and software to EC2. NASA uses Amazon's servers to process telemetry data from the Mars rovers. Amazon EC2 Service Level Agreement guarantees an Annual Uptime Percentage of 99.95%, and has performed well beyond this measure. 

In 2010, at the same time as Netflix, Sapling began moving our Web hosting to Amazon EC2. Stability and scalability were our main reasons for doing so. Until this month, Sapling was very proud of our 99.99% uptime. To put this in perspective, in 2009 Gmail had only a 99.90% reliability rate, with its stated goal to reach 99.99%. In March 2010, Twitter had 99.74% uptime. Up until today, we have been very happy with our decision, especially considering that our previous hosting company has been down at least two times since we moved to Amazon EC2, and we had not had any issues.

Unfortunately, Sapling and many other technology companies are affected by the current EC2 failure. These include leading Internet companies with millions of users such as Foursquare, Reddit, Hootsuite, and Quora, and because of the pervasive nature of this outage across the Internet, the Amazon failure has been prominently featured in the national news.

We are currently working parallel paths to resolve the solution, one of these will assure restoration of Sapling's site for tomorrow's classes in case Amazon is not able to get its cloud services issues fixed before then. Once we are back up and running, we will then begin to implement a strategy that will guard against even this unprecedented cloud downtime. 

Update on Site Availability

Sapling Learning has been unavailable for much of this morning, starting at about 6 AM Eastern to the present. Sapling is hosted on the Amazon Cloud, and the entire US-EAST-1 Region is experiencing connectivity issues. Sapling Learning, among thousands of other companies, has chosen to host our services on the Amazon Cloud for its excellent reliability and scalability. We are confident that Amazon will have their issues resolved quickly.

When the site is back up, your Technology TAs will grant an extension of 24 hours to any assignment that was due today. Please email your TechTA if you do not want the extension, or if you would prefer a longer extension for your classes.

We will post updates about Amazon's availability here hourly until this issue is resolved. Please visit this site for more details.

We apologize for the inconvenience that this has caused you and your students.

UPDATE (10:55 AM): We are still working to get the site back up. We apologize for the continued confusion. If your students have any questions, please forward them along to support@saplinglearning.com.

UPDATE (10:59 AM): Amazon has posted an explanation of the issue:

8:54 AM PDT We'd like to provide additional color on what were working on right now (please note that we always know more and understand issues better after we fully recover and dive deep into the post mortem). A networking event early this morning triggered a large amount of re-mirroring of EBS volumes in US-EAST-1. This re-mirroring created a shortage of capacity in one of the US-EAST-1 Availability Zones, which impacted new EBS volume creation as well as the pace with which we could re-mirror and recover affected EBS volumes. Additionally, one of our internal control planes for EBS has become inundated such that it's difficult to create new EBS volumes and EBS backed instances. We are working as quickly as possible to add capacity to that one Availability Zone to speed up the re-mirroring, and working to restore the control plane issue. We're starting to see progress on these efforts, but are not there yet. We will continue to provide updates when we have them.

UPDATE (11:39 AM): We're working on setting up the site on the USWEST half of the Amazon cloud, but it will take a while to get the very large amount of data we store transferred. Either through Amazon's fix or our own, we will have the site up as soon as possible. We will update you as soon as we have a time estimate.

UPDATE (1:01 PM): Amazon reports that they have made "significant progress" in stabilizing the cloud servers. We hope to have the issues resolved soon.

UPDATE (1:22 PM): Amazon estimates that, at worst, it will take a "few hours" for the servers to be back up. More information is available through Amazon's RSS feed, but we'll continue to also update here.

UPDATE (2:45 PM): Amazon is reporting some success of launching instances on the cloud servers. We hope to be back up soon.

UPDATE (3:46 PM): We're making progress, but the site is still down.

UPDATE (4:19 PM): We think we're getting very close.

UPDATE (5:01 PM): Amazon has brought up 3 of 4 zones in the USEAST-1 region. We, unforunately, are on the fourth zone. We are continuing to try to work around this outage, and to put safeties in place to avoid extensive outages like this in the future.

UPDATE (6:22 PM): There is a lot of activity from the server guys right now, but we are not disturbing them to get details. It looks like it's getting very close.

UPDATE (6:35 PM): We're up! We pride ourselves on our 99.95% uptime, and apologize for today's frustrations.

Sapling Learning for High School

Sapling Learning is pleased to announce online homework and problem-solving practice for high-school Biology, Chemistry, Physics, and IPC aligned to TEKS and STAAR EOC. Learn more at hs.saplinglearning.com. Additionally, we will know in a matter of months whether we've been approved by the TEA, which means that Texas high schools will be able to use TEA allocated funds toward the purchase of Sapling Learning for their classes.

Sapling's Digital Content Ecosystem

This week at the ACS Spring Meeting, we are introducing our new general chemistry eBook, Chemistry 5e by John Olmsted and Greg Williams. This book will be available under our new microcontent publishing model. We will be giving two talks at this conference:

• Tuesday, 9:15 AM, Disney's Grand California Hotel, Trillium B: Microcontent publishing: A new model for development and dissemination of educational support materials (p. TECH-99 of the on-site program, talk 1340)
• Tuesday, 3:30 PM - 6:00 PM, Anaheim Convention Center, Room 211A: Independent Online Homework and eBooks (a workshop on editing and earning royalties for content in our online homework and eBook system) (p. 29 of the on-site program)

For those of you unable to attend our sessions, this blog gives a brief overview of how things work.

If you have content you'd like to make available to your students, and you think instructors at other schools might like to use that same content, we'd like to let you get compensated for that effort. For questions, we'll pay a royalty based on the percentage of all questions in a course that you provide. In short, if you provide 50% of the questions used in a course, you'll get 50% of the available question royalty. Similarly, for eBook content, we'll pay a royalty based on the percentage of all eBook sections used in each course. The exact percentages can scale depending how much of the work you do, but the idea is to make it easy for people to mix-and-match content to fit their needs.

If you have content that you think would fit this model, let Brian Arneson know at publish@saplinglearning.com. We'll update the blog over the coming weeks with more details about what this means for your courses and how you can take advantage of this model, both as an author and as an instructor. Let us know in the comments if you have any questions you'd like us to answer in those posts.