Monday, November 15, 2010

Improving Questions: Sapling’s Quality Control Process

Sapling Learning is committed to creating the highest quality content, but we also devote a considerable effort towards maintaining and improving the efficacy of the existing questions. This post covers what happens to a question after it is "live" (available for use in assignments).

There are two opportunities for question revision during a question's lifetime: revision prompted by instructor or student comments, and revision through periodic, statistical reviews. The types of question flaws caught in each instance tend to be very different in nature.

Occasionally a Tech TA or Sapling Support will receive an email or call from an instructor or student alerting us to a potential error with a question. The question may have a wording issue, the tolerances on the question may be too tight, or, rarely, the correct answer may be incorrectly coded. Regardless, the issue with the question is often specific and the action is immediate: we may remove the question from the assignment, or replace the question with a corrected version.

There can be other issues with a question that are more subtle, and that is why we perform periodic, statistical reviews. During these reviews, Sapling analyzes statistical data for the questions that we have written. The question statistics include information about how many students have viewed the question, the average number of attempts per student on the question, the average number of points each student got on the question, and the difficulty rating that the original question author projected the question to have.* For each difficulty rating (easy, medium, and hard) within a discipline, we look for outliers—that is, questions for which students have taken significantly more or less attempts or have received more or less points than average for that difficulty rating. Those questions are flagged, and we examine the questions to see if we can find anything that might cause the unusual behavior. Sometimes the incorrect feedback could be improved, sometimes an issue only occurs for some values of a randomly generated variable, and sometimes question tolerances are too tight. In the instances where we cannot find a flaw in the question, we consider giving the question a new difficulty rating.

Quickly addressing potential question flaws, regardless of how the flaw is brought to our attention is one way that we continuously improve our question bank.  However, there are always opportunities to make this process better. As an instructor, what would you like to see us do to improve our content? Are there any statistics, like average time spent on a question, that you feel we should analyze when determining what questions should be flagged for review?

* The number of attempts on a question and the number of points received for the question are not necessarily giving you the same information if a question has more than one part.

69 comments:

  1. Useful details like this one must be kept and managed so I will put this one on my save list! Thanks for this amazing publish and expecting to publish more of this

    Digital Marketing Company

    ReplyDelete
  2. Your blog is looking good and its having the useful information, so most of the peoples are getting good knowledge to read your article. thnx for your creation.

    Medical Tourism to India

    ReplyDelete
  3. very useful details Thank you for your occupation…

    Seo company bangalore

    ReplyDelete
  4. This is great, That's a great website, it really is what I was looking for, thank you for sharing!
    ymail login | ymail sign in |hotmail sign up |hotmail login

    ReplyDelete
  5. I just want to tell you that I’m all new to blogging and site-building and really loved your web page. Very likely I’m want to bookmark your site . You amazingly have remarkable posts. Thanks a bunch for sharing your blog.
    Regards - www.office.com/setup

    ReplyDelete
  6. I am convinced that your website is convinced that weblog is very new. Probably I am going to bookmark your site. There are certainly impressive stories. Passion for sharing your website with us.
    Regards - www.norton.com/setup

    ReplyDelete
  7. Our My Assignment Help Reviews go through hundreds of websites each month to select only the best. We also offer reviews of such websites which are not up to the mark so that you may know about them as well.

    ReplyDelete
  8. I was impressed, I have to say. In fact, I rarely encounter educational and funny blogs. You may strike a nail on your head. Your idea is wonderful. The problem is one in which adequate individuals are not speaking intellectually. I am pleased that I found this by seeing this and searching for one thing.
    Regards - www.office.com/setup
    www.office.com/setup
    www.office.com/setup
    www.office.com/setup
    www.office.com/setup
    norton.com/setup
    www.office.com/setup
    www.office.com/setup

    ReplyDelete
  9. If you want to know about celebrities bio and their updates? You can visit Celebrity Ping. Here you will get all celebrities information, Alan Tudyk, Ashley Johnson, Bernie Mac, Calista Flockhart, Christina Hendricks, Dafne Keen, Emily, Eva Ekeblad, Freddie Prinze Jr, Jessica Barden, Jessica Henwick, Joan Cusack, Josh Wickersham Hartnett, Juan Gabriel, Kaitlyn Dever, Kathryn Hahn, Katie McGrath, Ksenia Solo, Laura Haddock, Lauren German, Lea Thompson, Loni Anderson, Lori Loughlin, Lucas Till, Maika Monroe, Maitland Ward, Markie Post, Matthew Gray Gubler, Olga Kurylenko, Phoebe Cates, Phoebe Snetsinger, Rebecca Romijn, Rene Russo, Robin Williams, Theo James, etc.

    ReplyDelete
  10. WeberGuide is a product reviewing and informative website that provides a fruitful guide in before and after buying all the accessories. We Publish Information about Home Accessories, Sports, Outdoor and Gardening. Visit our website to see our latest article on Best Kukri Knife, Best Flushing Toilet, Best Air Mattress and so on. For the most part, there are numerous sorts of sites on the web about the best items. In any case, we locate that the greater part of these are ecommerce site where individuals are simply attempting to offer their items. In any case, here on this site, you will get bunches of data which will spare your significant time and help you to pick the best items.

    ReplyDelete

  11. Hey Thanks for sharing this valuable information with us. I will come back to your site and keep sharing this information with us.
    Best Regards - office.com/setup
    office.com/setup
    office.com/setup
    office.com/setup
    office.com/setup

    ReplyDelete
  12. power testo blast
    scurrent health and fitness and wellness. Once you have your ideas set on what you want to achieve it's a lot easier to create to the getting and system that will be expected for you to lessen fat. Use both inner and
    https://newsletterforhealth.com/power-testo-blast/

    ReplyDelete
  13. cerisea medica and flatulence. Also the ingredient Ephedrine in many organic health and fitness and health items causes enough harm. This is why FDA or American Food and Drug Association has banned this particular ingredient. To sum up , I would say that it is not wise to compartmentalize organic items as 'safe' or take
    https://newsletterforhealth.com/cerisea-medica/

    ReplyDelete
  14. Beta keto
    aout needs because the kids take up so much of your energy and attempt but you can find cut out small bits of your energy and attempt depending on how many bodyweight you want to lessen and them that you need to do to get to know your goals. One of the many activities you can do to start with is to watch
    https://newsletterforhealth.com/beta-keto/

    ReplyDelete
  15. Keto fast normally expect to wait a few a few several weeks before locks start to build. While the locks is locking, it will need to be washed. Here is where washing should be extended for a while if possible, so that the locks can be allowed to lock for Two several weeks to about a 30 days without manipulation. When you into
    https://newsletterforhealth/keto-fast/

    ReplyDelete
  16. reviva brain
    You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site. 2 Responses to “Male Hair Loss - Tom’s Story.” 1. George Standing Says: May 25th, 2007 at 11:34 pm I think many people have to remember thinning locks is an ingredient
    https://newsletterforhealth.com/reviva-brain/

    ReplyDelete
  17. rapid slim
    determine they want. The acai berry berry fruits berry slim are found in the type of tablets or items and juices. So you can get any one of them and using it. Body weight Decrease Program with The acai berry berry fruits Berries Slim can help you get rid of 5lbs of bodyweight in just one 7 times which indicates
    https://newsletterforhealth.com/rapid-slim/

    ReplyDelete
  18. maxx power libido enhance the metabolic process and thus help in dropping of the undesirable flab in your physique system. The metabolic process increases due to excellent concentration of catechin polyphenols. This intensifies the amount of fat oxidation and thermogenensis. thermogenic properties. Thermogenic drinks as well as
    https://newsletterforhealth.com/maxx-power-libido/

    ReplyDelete
  19. keto blast
    , , particularly if you have injuries or other limitation, see a professional!* If you think ok starting up something on your own, let me put out a simple plug for one of my favorite online workout websites, Fitness Blender.com. For approachable, achievable and FUN workouts, this site is the best. Run by a
    https://topwellnessblog.com/keto-blast/

    ReplyDelete
  20. Excellent post this is really good blog . thank you for information

    โปรโมชั่นGclub ของทางทีมงานตอนนี้แจกฟรีโบนัส 50%
    เพียงแค่คุณสมัคร Gclub กับทางทีมงานของเราเพียงเท่านั้น
    ร่วมมาเป็นส่วนหนึ่งกับเว็บไซต์คาสิโนออนไลน์ของเราได้เลยค่ะ
    สมัครสล็อตออนไลน์ >>> goldenslot
    สนใจร่วมลงทุนกับเรา สมัครเอเย่น Gclub คลิ๊กได้เลย

    ReplyDelete
  21. Excellent post this is really good blog . thank you for information

    เว็บไซต์คาสิโนออนไลน์ที่ได้คุณภาพอับดับ 1 ของประเทศ
    เป็นเว็บไซต์การพนันออนไลน์ที่มีคนมา สมัคร Gclub Royal1688
    และยังมีหวยให้คุณได้เล่น สมัครหวยออนไลน์ ได้เลย
    สมัครสมาชิกที่นี่ >>> Gclub Royal1688
    ร่วมลงทุนสมัครเอเย่นคาสิโนกับทีมงานของเราได้เลย

    ReplyDelete
  22. Thanks for sharing this marvelous post. I m very pleased to read this article. mcafee.com/activate |norton.com/setup|mcafee.com/activate |office.com/setup

    ReplyDelete
  23. Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative.
    mcafee.com/activate |norton.com/setup|office.com/setup

    ReplyDelete
  24. provexum these problems should avoid man-made elements and stick to natural elements such as cotton, especially when it comes to undergarments. • 3. Soap and laundry detergent– While cleansers and cleansers can keep a clean, clean scent, they are often manufactured with dehydrating ingredients that leech the sebum from your epidermis part, making it sensation complicated, tender and itches.r
    https://newsletterforhealth.com/provexum/

    ReplyDelete
  25. Thanks for sharing this marvelous post. I m very pleased to read this article .I enjoy this site - its so usefull and helpfull.
    mcafee.com/activate | office.com/setup |norton.com/setup mcafee.com/activate |norton.com/setup |office.com/setup

    ReplyDelete
  26. Nice one! thank you so much! Thank you for sharing this post. Your blog posts are more interesting and impressive.

    office.com/setup | office.com/setup | norton.com/setup | office.com/setup

    ReplyDelete
  27. Well i am a regular reader of new blogs and i found this one is also such a nice one... keep it up
    mcafee.com/activate | mcafee.com/activate | www.norton.com/setup

    ReplyDelete
  28. You cannot imagine simply how much time I had spent for this information! Thanks!
    office.com/setup | norton.com/setup | www.mcafee.com/activate

    ReplyDelete
  29. Wonderful blog! I found it while browsing on Yahoo News

    Regards= office.com/setup

    ReplyDelete
  30. office.com/setup- For Downloading, Installing and activating the Office product key, visit www.office.com/setup and Get Started previously Office setup.

    ReplyDelete
  31. After visiting office.com/setup login to microsoft account to buy contract of office setup, Any misery in www.office.com/setup our sting will assert going on for office install.

    ReplyDelete
  32. What a superb post! I have no words to describe this post because everthing is clear with your wonderfull words. I really feel out for world reading your post, it is full of fresh and usefull. I really appreciate, keep the work continue.

    www.norton.com/setup ! norton.com/setup ! office.com/setup ! office.com/setup ! mcafee.com/activate ! mcafee.com/activate

    ReplyDelete
  33. Sign in to enter office setup product key. Know how to benefit, download, install, set in movement, uninstall and reinstall MS office setup.
    office.com/setup
    http://officecom-officeoffice.com/
    call us at +1-888-421-9666[tool free]

    ReplyDelete
  34. Nice one! thank you so much! Thank you for sharing this post. Your blog posts are more interesting and impressive.

    For more visit on: mcafee.com/activate | mcafee.com/activate

    ReplyDelete
  35. I certainly needed to post a note in order to express profound gratitude to you for those astounding exhortation you are composing at this site. norton.com/setup | norton.com/setup | norton.com/setup | office.com/setup

    ReplyDelete
  36. Download office setup with our help team without getting any work done by yourself. For more visit office.com/setup here.

    ReplyDelete

  37. Microsoft Office install has required the removal of the previously installed version of your Office product on the device or system. Office 365 and other subscription offers the various features, which you do not get when you do not purchase the Office product. The office can be used free, as Microsoft provides the trial versions of every tool. visit office setup today.

    ReplyDelete

  38. Download Webroot Geek Squad Antiviurs If you own a device or system and you are connecting it to internet or another device then you must have antivirus software. Virus or any risky threat like Malware, Trojan, Spyware, Rootkit or online hacking or attack can steal your data and damage your system. click webroot safe today.

    ReplyDelete

  39. Makes the web safe for your kids to explore and helps your kids balance their time spent .. Internet Security Plus with Antivirus Protection by..... webroot.com/safe .

    ReplyDelete
  40. Mcafee Activate has recognized to be an effective way of keeping up your devices and your data safe from getting infected by any hackers and virus attackers. For more details visit mcafee activate today.

    ReplyDelete
  41. With Mcafee.com/activate , you can prevent devices and network against threats like malware, worms, spyware and runs in the background and does not cause any problem nor slows down the computer.

    ReplyDelete
  42. To activate Hulu Plus on your device, either use the on-screen keyboard to enter your Hulu log in information or go to hulu activate and enter the device activation code. Either of these methods will allow you to use Hulu on any Hulu-supported device.

    ReplyDelete
  43. Hulu supports a wide variety of devices that you can use to stream with us. You can activate any of them at any time, as well as remove them from your account. click hulu.com/activate today.

    ReplyDelete
  44. Install mcafee antivirus in your computer with high class professionals and best tech team. Just ring us and we are ready to assist you till the last minute of installation - www.mcafee.com/activate today.

    ReplyDelete
  45. Mcafee offers you the McAfee activate option to give best Home network protection, Multi-device protection, Web Protection, and Identity Theft. visit Mcafee.com/activate today.

    ReplyDelete
  46. Are you looking for activating your Roku com link? This can turn your tv into up to date streaming gadget, Rokucomlink you can stay advanced. click roku.com/link here.

    ReplyDelete
  47. This comment has been removed by the author.

    ReplyDelete
  48. mcafee antivirus is widely used aantivirus helps to detect and neutralize computer virus, the mail worms,the trojan programs,and also helps your system free of virus and other malware is quite a daily challenge. for more details visit: mcafee activate today.

    ReplyDelete
  49. roku link this is least difficult approach to stream stimulation to your TV. On your terms. With a great many accessible channels to look over for more details.visit: roku link today.

    ReplyDelete
  50. Install mcafee with activation code from mcafee 25 digit activation code . Activate mcafee with activation code to secure your windows and Mac Pc.

    ReplyDelete
  51. www.OFFICE.COM/SETUP DOWNLOAD, INSTALLATION AND ACTIVATION FOR OFFICE SETUP. Microsoft Office suite contains reliably used ventures and organizations. The most used tasks of Microsoft Office are Word, a word taking care of program, Excel, a spreadsheet program, PowerPoint, a presentation maker, and Access, a database application

    ReplyDelete
  52. Online Custom Essay Services are affordable, with rates that are relatively lower in comparison with other Legitimate Custom Essay Writing Services since they allow learners to estimate charges when placing Legitimate Custom Writing Service Orders.

    ReplyDelete
  53. Thanks for sharing this useful information...
    Technical customer care Norton for Antivirus setup can be availed by dialing the Ñѽřţŏƞ.com/setup, Phone number, +1-855-619-5888 Call us and remain ensured while taking a shot at your PC. http://nortoncomnortonnorton.com
    www. Ñѽřţŏƞ.com/setup
    Norton.com/setup

    ReplyDelete
  54. Thanks for sharing this useful information...
    Make one more Norton setup install account by visiting on www.Ñѽřţŏƞ.com/setup. Use the credentials to download, install, and activate the Norton setup. Get the product activation key from http://nortonwwwnorton.com/
    www.norton.com/setup
    Ñѽřţŏƞ setup

    ReplyDelete
  55. You share a awesome information…
    MS office setup first signs in then put product key for activation your setup in your system then easily done and enjoy express life with office.com/setup. http://officeoffice-us.com office.com/setup www.office.com/setup

    ReplyDelete
  56. Consistent and clear marking is one of the keys to fast, stress-free movement. Some people use the numbering system, while others create detailed color coding systems. No matter which tagging method you use, consider the following:
    شركة نقل عفش بالرياض

    ReplyDelete
  57. Thanks for sharing this with us......this is really appreciating.
    office.com/setup
    mcafee.com/activate
    norton.com.setup

    ReplyDelete
  58. At the point, while you download or introduce software on the device, the first steps for you ought to be to check the system prerequisite. Since a system that doesn't bolster the base system necessity for software will show the similarity issue. Along these lines, think about the most recent version of the AOL Desktop Gold Software system prerequisite and your device details, and try to meet that. If you need any information about AOL gold software contacts us our experts 24/7 available for AOL users.

    ReplyDelete
  59. Fantastic blog... I think it's a very interesting and beneficial website for us.
    www.office.com/setup
    norton.com/setup
    norton.com/setup
    norton.com/setup

    ReplyDelete
  60. I'd really love to be a part of online community where I can get opinions from other knowledgeable individuals that share the same interest. List of Retail Companies in Singapore

    ReplyDelete
  61. Fantastic site you have here but I was curious if you knew of any user discussion forums that cover the same topics discussed in this article? I'd really like to be a part of online community where I can get advice from other knowledgeable individuals that share the same interest. executor of will singapore

    ReplyDelete