John: Based on your intro about opportunities regarding a skill/concept (KC), you could imagine measures of useful in terms of a) does it address a KC that currently has too few opportunities b) does adding it to the course “work” in terms of producing learning transfer in the associated learning curve. Have you tried something like these?

It looks like the main challenge is to pick up good questions. Does it mean that concept extraction problem from educational text is mostly solved? Have you tried to asses the concept modeling inthe same way?

Are any of the ratings assessing coherence/ease of comprehension? some of those questions seemed very hard to parse.

How do the useful / not-useful ratings of the learner sourced questions compare to the computer generated questions?

I've applied the classifier to ~100 questions generated in a chemistry course and it ranked around 85% of them as useful where the computer generated ones was closer to 60% I believe

Do you have some examples of the student inputs to these activities? Would be interesting to see!

In these activities what kind of feedback do students get when they click “Submit and Compare”? Is an example(s) given of a possible good question associated with the video?

Typically the activity is embedded into OLI towards the end of the given module, so we ~hope that the other MCQs they've come across in the materials guides them in their creation (but not TOO much guidance where they copy the format)

Workshop website: https://sites.google.com/andrew.cmu.edu/learnersourcing/home

Exciting work!

It sounds like students might not really know what skills are (even designers have trouble with this!). Has anyone tried more colloquial approaches, like 'what steps did you follow to do this' or 'if you were showing someone else how to do this, what would they need to know?'

are there any ip concerns with student generated q's?

Lots of OLI courses have many submit and compare questions — these seem ripe for such an analysis … we are producing more in scenario-based human tutor training …

nice work!

Are there students that complete multiple different OLI courses?

Thanks for the presentation John!

thank you! very interesting!

Thank you!

🎊