Current Research Projects

service-icon

SAQUET: Scalable Automatic Question Usability Evaluation Toolkit

Automatically evaluate the quality of educational multiple-choice questions according to a 19-critera rubric.

Try out the beta here
service-icon

Learnersourcing in Online Courseware

Students participate in the creation and evaluation of educational content, all while learning in the process.

Read about my recent work
service-icon

Large Language Models for Rubrics

Using LLMs such as GPT-4, I apply rubrics for evaluating the quality of educational questions.

Read about my recent work
service-icon

Crowdsourcing for Education

We have crowdworkers participate in a variety of tasks such as evaluating question quality, providing the skills a question assesses, or creating a hint for a question.

Read about my recent work
service-icon

Xprize Digital Learning Challenge

My team recently won the $1M compeition to modernize, accelerate, and improve the ways in which we identify effective learning tools and processes that improve learning.

Check out the press release

Teaching

Principles of Computing

Over the past few summers, I've had the opportunity to teaching introductory programming in Python to high schoolers, freshmen, and incoming MHCI students at CMU.

Access the open learning materials

Tools for Online Learning

I redesigned and taught a graduate-level course on designing, evaluating, and using educational technology in the classroom that leverage learning science principles.

Checkout the student feedback: 2023, 2024

Introduction to Statistics

I taught a gradudate-level course on introductory statistics, designed to prepare students for their future coursework and assist them in evaluating research designs.

Access the open learning materials

Evidence-Based Educational Design

I updated and taught a graduate-level course on effective strategies for measuring student learning, enhancing student engagement, and assessing educational effectiveness.

I'll have student reviews soon :)

Professional Experience

service-icon

Postdoctoral Researcher

LearnLab

Teaching two graduate-level courses "Evidence-Based Educational Design" and "Tools for Online Learning", while also building tools and researching ways to improve the learning engineering process by harnessing (pedagogically informed) LLM-based methods.

METALS Faculty website
service-icon

Learning Engineer

Eberly Center

Provided faculty support for integrating technologylearning in their courses, enhanced CS courses at the undergraduate and graduate levels, and developed interactive tools to assist faculty in teaching across multiple domains.

Eberly Center's website
service-icon

Data Scientist

Coursera

Devised heuristics to assess skill tagging production model saturation, evaluated alternate skill rescaling approaches for improved student learning modeling, and investigated the effects of post-assessment resource provision on learners.

Coursera's website
service-icon

Graduate Teaching Fellow

Carnegie Mellon University

Conducted classroom observations to provide data-driven feedback to instructors, worked with clients to construct and revise statements of teaching philosophy, and lead microteaching workshops to provide instructors with teaching feedback.

Graduate Teaching Fellow's website
service-icon

Full-Stack Developer

Westat

Created customized frontend websites to fulfill diverse client requirements, developed robust backend systems and managed extensive databases, and designed Android and iOS apps for surveys and location tracking purposes.

Westat's website

Publications

Year Type Publication
2024 Thesis
🤠
Steven Moore (2024).
Toward Automation to Support Creation and Evaluation of Pedagogically Valid Multiple-Choice Question Assessments at Scale.
For the degree ofDoctor of Philosophy in Human-Computer Interaction at Carnegie Mellon University
2024 Conference Steven Moore, Norman Bier, John Stamper (2024).
Assessing Educational Quality: Comparative Analysis of Crowdsourced, Expert, and AI-Driven Rubric Applications.
In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 12, forthcoming
2024 Conference Steven Moore, Eamon Costello, Huy A Nguyen, John Stamper (2024).
An Automatic Question Usability Evaluation Toolkit.
In Proceedings of the 25th International Conference on Artificial Intelligence in Education, pp. 31-46
2024 Conference 🏅 Steven Moore, Robin Schmucker, Tom Mitchell, John Stamper (2024).
Automated Generation and Tagging of Knowledge Components from Multiple-Choice Questions.
In Proceedings of the 11th ACM Conference on Learning @ Scale, pp. 122-133
2024 Poster Ilya Musabirov, Mohi Reza, Steven Moore, Pan Chen, Harsh Kumar, Tong Li, Haochen Song, Jiakai Shi, Koby Choy, Thomas Price, John Stamper, Norman Bier, Nina Deliu, Sofia Villar, Anna Rafferty, Audrey Durand, Joseph Jay Williams (2024).
Platform-based Adaptive Experimental Research in Education: Lessons Learned from Digital Learning Challenge.
In Companion Proceedings in LAK24: 14th International Learning Analytics and Knowledge, pp. 37-40
2023 Conference Steven Moore, Huy A Nguyen, Tianying Chen, John Stamper (2023).
Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods.
In Proceedings of the 18th European Conference on Technology Enhanced Learning, pp. 229-245
2023 Conference Steven Moore, Ellen Fang, Huy A Nguyen, John Stamper (2023).
Crowdsourcing the Evaluation of Multiple-Choice Questions Using Item-Writing Flaws and Bloom’s Taxonomy.
In Proceedings of the 10th ACM Conference on Learning @ Scale, pp. 25-34
2023 Journal Hassan Khosravi, Paul Denny, Steven Moore, John Stamper (2023).
Learnersourcing in the Age of AI: Student, Educator and Machine Partnerships for Content Creation.
In Computers and Education: Artificial Intelligence, 100151
2023 Conference Steven Moore, Huy A Nguyen, Norman Bier, Tanvi Domadia, John Stamper (2023).
Who Writes Tomorrow’s Learning Activities? Exploring Community College Student Participation in Learnersourcing.
In Proceedings of the 17th International Conference of the Learning Sciences
2023 Practioner Report Norman Bier, John Stamper, Steven Moore, Darren Siegel, Ariel Anbar (2023).
OLI Torus: A Next-Generation, Open Platform for Adaptive Courseware Development, Delivery, and Research.
In Companion Proceedings in LAK23: 13th International Learning Analytics and Knowledge, pp. 57-60
2023 Poster Steven Moore, Huy A Nguyen, John Stamper (2023).
Students’ Domain Confidence and their Participation in Optional Learnersourcing Activities.
In Companion Proceedings in LAK23: 13th International Learning Analytics and Knowledge, pp. 147-149
2022 Conference Steven Moore, Huy A Nguyen, Norman Bier, Tanvi Domadia, John Stamper (2022).
Assessing the Quality of Student-Generated Short Answer Questions Using GPT-3.
In Proceedings of the 17th European Conference on Technology Enhanced Learning, pp. 243-257
2022 Conference Huy A Nguyen, Shravya Bhat, Steven Moore, Norman Bier, John Stamper (2022).
Towards Generalized Methods for Automatic Question Generation in Educational Domains.
In Proceedings of the 17th European Conference on Technology Enhanced Learning, pp. 272-284
2022 Poster Shravya Bhat, Huy A Nguyen, Steven Moore, John Stamper, Majd Sakr, Eric Nyberg (2022).
Towards Automated Generation and Evaluation of Questions in Educational Domains.
In Proceedings of the International Conference on Educational Data Mining, pp. 701-704
2022 Conference Steven Moore, Huy A Nguyen, John Stamper (2022).
Leveraging Students to Generate Skill Tags that Inform Learning Analytics.
In Proceedings of the 16th International Conference of the Learning Sciences, pp. 791-798
2022 Conference Steven Moore, Huy A Nguyen, John Stamper (2022).
Participation and Success with Optional Self-Explanation for Students in Online Undergraduate Chemistry Courses.
In Proceedings of the 16th International Conference of the Learning Sciences, pp. 1381-1384
2021 Conference Steven Moore, Huy A Nguyen, John Stamper (2021).
Examining the Effects of Student Participation and Performance on the Quality of Learnersourcing Multiple-Choice Questions.
In Proceedings of the 8th ACM Conference on Learning @ Scale, pp. 209-220
2021 Conference Huy A Nguyen, Michelle Lim, Steven Moore, Majd Sakr, Eric Nyberg, John Stamper (2021).
Exploring Metrics for the Analysis of Code Submissions in an Introductory Data Science Course.
In Proceedings of the 11th International Conference on Learning Analytics & Knowledge, pp. 632-638
2021 Conference Steven Moore, John Stamper, Norman Bier, Mary Jean Blink (2021).
A Human-Centered Approach to Data Driven Iterative Course Improvement.
In Proceedings of the 17th International Conference on Remote Engineering and Virtual Instrumentation, pp. 742-761
2020 Conference Steven Moore, Huy A Nguyen, John Stamper (2020).
Towards Crowdsourcing the Identification of Knowledge Components.
In Proceedings of the 7th ACM Conference on Learning @ Scale, pp. 245-248
2020 Conference Rosalyn Abbott, Conrad Zapanta, Michael Melville, Steven Moore (2020).
WIP: Incorporating Interactive Modules Related to Cell Culture and Plasmid Design into Introduction to Biomedical Engineering.
In American Society of Engineering Education, pp. 234-238
2020 Conference 🏅 Steven Moore, Huy A Nguyen, John Stamper (2020).
Utilizing Crowdsourcing and Topic Modeling to Generate Knowledge Components for Math and Writing Problems.
In Proceedings of the 28th International Conference on Computers in Education, pp. 31-40
2020 Conference Steven Moore, Huy A Nguyen, John Stamper (2020).
Evaluating Crowdsourcing and Topic Modeling in Generating Knowledge Components from Explanations.
In Proceedings of the 21st International Conference on Artificial Intelligence in Education, pp. 398-410
2020 Poster Steven Moore, Huy A Nguyen, John Stamper (2020).
Crowdsourcing Explanations for Improving Assessment Content and Identifying Knowledge Components.
In Proceedings of the 14th International Conference of the Learning Sciences, pp. 2637-2638
2019 Workshop Norman Bier, Steven Moore, Martin Van Velsen (2019).
Instrumenting Courseware and Leveraging Data with the Open Learning Initiative (OLI).
In Companion Proceedings 9th International Learning Analytics & Knowledge Conference, pp. 990-1001
2019 Workshop Steven Moore, John Stamper (2019).
Exploring Expertise through Visualizing Agent Policies and Human Strategies in Open-Ended Games.
In Proceedings of the Workshops of the 12th International Conference on Educational Data Mining, pp. 30-37
2019 Demo John Stamper, Paulo Carvalho, Steven Moore, Kenneth Koedinger (2019).
Tigris: An Online Workflow Tool for Sharing Educational Data and Analytic Methods.
In Companion Proceedings 9th International Learning Analytics & Knowledge Conference, pp. 189
2019 Conference John Stamper, Steven Moore (2019).
Exploring Teachable Humans and Teachable Agents: Human Strategies Versus Agent Policies and the Basis of Expertise.
In Proceedings of the 20th International Conference on Artificial Intelligence in Education, pp. 269-274
2019 Conference 🏅 Steven Moore, John Stamper (2019).
Decision Support for an Adversarial Game Environment Using Automatic Hint Generation.
In Proceedings of the 15th International Conference on Intelligent Tutoring Systems, pp. 82-88

Hosted Workshops

Year Venue Publication
2024 L@S Steven Moore, Anjali Singh, Xinyi Lu, Hyoungwook Jin, Hassan Khosravi, Paul Denny, Christopher Brooks, Xu Wang, Juho Kim, John Stamper (2024).
Learnersourcing: Student-generated Content @ Scale. (2nd annual workshop)
at Learning @ Scale 2024
https://sites.google.com/andrew.cmu.edu/learnersourcing
2024 EDM Neil Heffernan, Rose Wang, Christopher MacLellan, Arto Hellas, Chenglu Li, Candace Walkington, Joshua Littenberg-Tobias, David Joyner, Steven Moore, Adish Singla, Zach Pardos, Maciej Pankiewicz, Juho Kim, Shashank Sonkar, Clayton Cohn, Anthony Botelho, Andrew Lan, Lan Jiang, Mingyu Feng, Tanja Kaser, Eamon Worden (2024).
Leveraging Large Language Models for Next-Generation Educational Technologies.
at Educatonal Data Mining 2024
https://sites.google.com/view/llmworkshopedm/home
2023 NeurIPS Paul Denny, Sumit Gulwani, Neil Heffernan, Tanja Käser, Steven Moore, Anna Rafferty, Adish Singla (2023).
Generative AI for Education (GAIED): Advances, Opportunities, and Challenges.
at Neural Information Processing Systems 2023
https://gaied.org/neurips2023
2023 AIED Steven Moore, Richard Tong, Anjali Singh, Zitao Liu, Xiangen Hu, Yu Lu, Joleen Liang, Chen Cao, Hassan Khosravi, Paul Denny, Chris Brooks, John Stamper (2023).
Empowering Education with LLMs-The Next-Gen Interface and Content Generation.
at Artificial Intelligence in Education 2023
https://ai4ed.cc/workshops/aied2023
2023 LAK Steven Moore, Anjali Singh, Hassan Khosravi, Paul Denny, Chris Brooks, John Stamper (2023).
Partnerships for Cocreating Educational Content.
at Learning Analytics and Knowledge 2023
https://sites.google.com/andrew.cmu.edu/partnerships-for-cocreating-ed
2022 L@S Steven Moore, John Stamper, Christopher Brooks, Paul Denny, Hassan Khosravi (2023).
Learnersourcing: Student-generated Content @ Scale.
at Learning @ Scale 2022
https://sites.google.com/andrew.cmu.edu/learnersourcing (proceedings)