As a follow-up to last week's post, I decided to take a look at student performance by operating system. This data comes from a single assignment in one large course, to try to eliminate other variables. Within this course, 173 students had logged in during the time frame of the assignment exclusively from Windows machines, and 193 students had logged in exclusively from Macs.
On average, the Mac-using students scored 6.2 points (out of 100) higher than the Windows-using students. Many students (about 10%) had logged in from both Mac and Windows machines, so I added those to the chart.
The dual-OS users scored even higher than their Mac-exclusive fellow-students, scoring 3.2 points higher than Mac users, and 9.4 points higher than Windows users.
While it might be tempting to surmise from this data that Mac users are smarter than Windows users (and, given the attention we expect such a finding would generate, we welcome that over-simplification in posts about this blog), I strongly suspect any effect we're seeing here is simply a measure of socio-economic background. Macs tend to be more expensive than Windows machines, and socio-economic background has often been found to correlate with performance.
We plan to take another look at these numbers on a larger scale (and with a more complete statistical analysis), but this initial data seemed interesting enough to share.
Showing posts with label statistics. Show all posts
Showing posts with label statistics. Show all posts
Monday, February 28, 2011
Monday, February 21, 2011
Data Mining: 6 Things I Learned from Student Login Times
With students visiting our site every day to do their homework, we've built up a large database of information. In this article (and future articles like it), we'll see what we can see in that data. This data is based on student logins between January 9, 2011, and February 13, 2011, adjusted to the student's local timezone.
#1. Weekends = Friday + Saturday
We noticed this one a while back just from our Google Analytics data: at least as far as students working on homework are concerned, the work week runs Sunday through Thursday, leaving Friday and Saturday for extracurricular activities.
#2. Any Time's a Good Time for Homework
Although more students log in per hour from about 7pm-10pm (their local time) than during other hours, login times spread throughout the day. One hour is 4.2% of a day, and there are 13 time slots (11am through 11pm) that are at or above that percentage of the daily logins. Even at 4am and 5am we get several hundred logins most days, but those are definitely our least-active hours. That's why we wake up early on release days to push releases out starting then.
#3. Students Who Use Macs Prepare for the Weekend
This difference is very small, and it may not be statistically significant, but our Mac users (shown in red here) appear to start their weeks early to prepare for the weekend. A greater percentage of our Mac logins occur on Sunday and Monday compared to Windows users. Conversely, Thursday through Saturday are slightly weighted toward PC.
#4. Early to Bed and Early to Rise Makes a Student (ever-so-slightly-more) Likely to Use a PC
A similar trend appears when I group the data by hour, rather than by day of the week. Through about noon, a greater percentage of our PC-using students log in than do our Mac users. Things mostly level out during the day, and then the Mac users begin to top their PC-using counterparts.
A Break for Pie
Most of our students still use Windows. However, Mac's share is definitely growing. Over the last year, Mac's share of the student pie has grown by about 10%.
Our students use a healthy mix of the four biggest browsers (Internet Explorer, Firefox, Safari, and Google Chrome), plus occasional logins from virtually every other browser out there (including, for example, 9 logins from Sony Playstations). As with OS, student use has definitely changed in the last year, but this pie is more divided. Both IE and Firefox have lost students, while Chrome use has more than doubled (and continues to increase every month). Safari use has also increased, but not everyone switching to Mac is switching to Safari.
#5. Geeks are Slightly More Likely to Work on Saturday
A few trends can be spotted in the browser login data if you squint. For example, the Safari daily logins, unsurprisingly, track with the Mac logins: Safari users get on the site earlier in the week, and then disappear for Friday and Saturday. Somewhat similarly, a smaller percentage of Internet Explorer users log in on Sundays than on other days (when compared to the other browsers). As a Chrome user looking at stats on a Saturday, I also found it interesting that users of Firefox and Chrome log in more on Saturday than do their Safari- and IE-using counterparts. Since Firefox and Chrome require the user to specifically install them (while Safari and IE are the default browsers for Mac and PC, respectively), one (stereotypically unsurprising) way to look at this is that the geekier users log in more on Saturday than do their less-tech-loving cohorts.
#6. IE and Firefox Users Like Mornings, Chrome and Safari Users Like Nights
Like with the Mac/PC time-of-day stats, our Internet Explorer and Firefox users start logging on earlier than our Safari and Chrome users. IE usage drops off faster than Firefox usage, though, until, around midnight, Firefox users are more likely to log on than even night-loving Safari users.
To be clear, I've drawn a lot of conclusions from very slight differences in this article. However, we may have to take a deeper look. Each of these groupings involves hundreds or thousands of data points, so these small differences very well could be significant.
I plan to do more of these data mining expeditions to see what we might see. What else would you like to know?
#1. Weekends = Friday + Saturday
We noticed this one a while back just from our Google Analytics data: at least as far as students working on homework are concerned, the work week runs Sunday through Thursday, leaving Friday and Saturday for extracurricular activities.
#2. Any Time's a Good Time for Homework
Although more students log in per hour from about 7pm-10pm (their local time) than during other hours, login times spread throughout the day. One hour is 4.2% of a day, and there are 13 time slots (11am through 11pm) that are at or above that percentage of the daily logins. Even at 4am and 5am we get several hundred logins most days, but those are definitely our least-active hours. That's why we wake up early on release days to push releases out starting then.
#3. Students Who Use Macs Prepare for the Weekend
This difference is very small, and it may not be statistically significant, but our Mac users (shown in red here) appear to start their weeks early to prepare for the weekend. A greater percentage of our Mac logins occur on Sunday and Monday compared to Windows users. Conversely, Thursday through Saturday are slightly weighted toward PC.
#4. Early to Bed and Early to Rise Makes a Student (ever-so-slightly-more) Likely to Use a PC
A similar trend appears when I group the data by hour, rather than by day of the week. Through about noon, a greater percentage of our PC-using students log in than do our Mac users. Things mostly level out during the day, and then the Mac users begin to top their PC-using counterparts.
A Break for Pie
Most of our students still use Windows. However, Mac's share is definitely growing. Over the last year, Mac's share of the student pie has grown by about 10%.
Our students use a healthy mix of the four biggest browsers (Internet Explorer, Firefox, Safari, and Google Chrome), plus occasional logins from virtually every other browser out there (including, for example, 9 logins from Sony Playstations). As with OS, student use has definitely changed in the last year, but this pie is more divided. Both IE and Firefox have lost students, while Chrome use has more than doubled (and continues to increase every month). Safari use has also increased, but not everyone switching to Mac is switching to Safari.
#5. Geeks are Slightly More Likely to Work on Saturday
A few trends can be spotted in the browser login data if you squint. For example, the Safari daily logins, unsurprisingly, track with the Mac logins: Safari users get on the site earlier in the week, and then disappear for Friday and Saturday. Somewhat similarly, a smaller percentage of Internet Explorer users log in on Sundays than on other days (when compared to the other browsers). As a Chrome user looking at stats on a Saturday, I also found it interesting that users of Firefox and Chrome log in more on Saturday than do their Safari- and IE-using counterparts. Since Firefox and Chrome require the user to specifically install them (while Safari and IE are the default browsers for Mac and PC, respectively), one (stereotypically unsurprising) way to look at this is that the geekier users log in more on Saturday than do their less-tech-loving cohorts.
#6. IE and Firefox Users Like Mornings, Chrome and Safari Users Like Nights
Like with the Mac/PC time-of-day stats, our Internet Explorer and Firefox users start logging on earlier than our Safari and Chrome users. IE usage drops off faster than Firefox usage, though, until, around midnight, Firefox users are more likely to log on than even night-loving Safari users.
To be clear, I've drawn a lot of conclusions from very slight differences in this article. However, we may have to take a deeper look. Each of these groupings involves hundreds or thousands of data points, so these small differences very well could be significant.
I plan to do more of these data mining expeditions to see what we might see. What else would you like to know?
Monday, November 15, 2010
Improving Questions: Sapling’s Quality Control Process
Sapling Learning is committed to creating the highest quality content, but we also devote a considerable effort towards maintaining and improving the efficacy of the existing questions. This post covers what happens to a question after it is "live" (available for use in assignments).
There are two opportunities for question revision during a question's lifetime: revision prompted by instructor or student comments, and revision through periodic, statistical reviews. The types of question flaws caught in each instance tend to be very different in nature.
Occasionally a Tech TA or Sapling Support will receive an email or call from an instructor or student alerting us to a potential error with a question. The question may have a wording issue, the tolerances on the question may be too tight, or, rarely, the correct answer may be incorrectly coded. Regardless, the issue with the question is often specific and the action is immediate: we may remove the question from the assignment, or replace the question with a corrected version.
There can be other issues with a question that are more subtle, and that is why we perform periodic, statistical reviews. During these reviews, Sapling analyzes statistical data for the questions that we have written. The question statistics include information about how many students have viewed the question, the average number of attempts per student on the question, the average number of points each student got on the question, and the difficulty rating that the original question author projected the question to have.* For each difficulty rating (easy, medium, and hard) within a discipline, we look for outliers—that is, questions for which students have taken significantly more or less attempts or have received more or less points than average for that difficulty rating. Those questions are flagged, and we examine the questions to see if we can find anything that might cause the unusual behavior. Sometimes the incorrect feedback could be improved, sometimes an issue only occurs for some values of a randomly generated variable, and sometimes question tolerances are too tight. In the instances where we cannot find a flaw in the question, we consider giving the question a new difficulty rating.
Quickly addressing potential question flaws, regardless of how the flaw is brought to our attention is one way that we continuously improve our question bank. However, there are always opportunities to make this process better. As an instructor, what would you like to see us do to improve our content? Are there any statistics, like average time spent on a question, that you feel we should analyze when determining what questions should be flagged for review?
* The number of attempts on a question and the number of points received for the question are not necessarily giving you the same information if a question has more than one part.
There are two opportunities for question revision during a question's lifetime: revision prompted by instructor or student comments, and revision through periodic, statistical reviews. The types of question flaws caught in each instance tend to be very different in nature.
Occasionally a Tech TA or Sapling Support will receive an email or call from an instructor or student alerting us to a potential error with a question. The question may have a wording issue, the tolerances on the question may be too tight, or, rarely, the correct answer may be incorrectly coded. Regardless, the issue with the question is often specific and the action is immediate: we may remove the question from the assignment, or replace the question with a corrected version.
There can be other issues with a question that are more subtle, and that is why we perform periodic, statistical reviews. During these reviews, Sapling analyzes statistical data for the questions that we have written. The question statistics include information about how many students have viewed the question, the average number of attempts per student on the question, the average number of points each student got on the question, and the difficulty rating that the original question author projected the question to have.* For each difficulty rating (easy, medium, and hard) within a discipline, we look for outliers—that is, questions for which students have taken significantly more or less attempts or have received more or less points than average for that difficulty rating. Those questions are flagged, and we examine the questions to see if we can find anything that might cause the unusual behavior. Sometimes the incorrect feedback could be improved, sometimes an issue only occurs for some values of a randomly generated variable, and sometimes question tolerances are too tight. In the instances where we cannot find a flaw in the question, we consider giving the question a new difficulty rating.
Quickly addressing potential question flaws, regardless of how the flaw is brought to our attention is one way that we continuously improve our question bank. However, there are always opportunities to make this process better. As an instructor, what would you like to see us do to improve our content? Are there any statistics, like average time spent on a question, that you feel we should analyze when determining what questions should be flagged for review?
* The number of attempts on a question and the number of points received for the question are not necessarily giving you the same information if a question has more than one part.
Subscribe to:
Posts (Atom)