Papers

Link to Full CV

Research Statement

Talks Illustrating Examples of Lab Research

Representative Research



Systems and Tools (using Adaptive Experimentation and other Techniques)

Williams, J. J., Rafferty, A., Tingley, D., Ang, A., Lasecki, W. S., & Kim, J. (2018). Enhancing Online Problems Through Instructor-Centered Tools for Randomized Experiments. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems. [PDF] [Talk Slides[Related Poster] [Video Figure] [Instructions to Use System] [Video of Talk[Talk Transcription]

Reza, M., Kim, J., Bhattacharjee, A., Rafferty, A.N., & Williams, J.J. (2021). The MOOClet Framework: Unifying Experimentation, Dynamic Improvement, & Personalization in Online Courses. To appear in the Eighth Annual ACM Conference on Learning at Scale. [PDF] [Video of Talk]

Williams, J. J., Kim, J., Rafferty, A., Maldonado, S., Gajos, K., Lasecki, W., & Heffernan, N. (2016). AXIS: Generating Explanations at Scale with Learnersourcing and Machine LearningProceedings of the Third Annual ACM Conference on Learning at ScaleNominee for Best Paper Award [top 4]  [PDF] [Slides] 

Adaptive Experimentation – Statistical Issues and Algorithms

Rafferty, A., Ying, H., & Williams, J. J. (2019). Statistical consequences of using multi-armed bandits to conduct adaptive educational experiments. Journal of Educational Data Mining. [PDF]
Shaikh, H., Modiri, A., Williams, J. J., & Rafferty, A. N. (2019) Balancing Student Success and Inferring Personalized Effects in Dynamic Experiments. Proceedings of the 12th International Conference on Educational Data Mining. [PDF] [Poster]

Li, Z., Yee, L., Sauerberg, N., Sakson, I., Williams, J. J., & Rafferty, A. N. (2020). Getting too personal(ized): The importance of feature choice in online adaptive algorithms. Proceedings of the 13th International Conference on Educational Data Mining (pp. 159-170). [PDF]

Empirical Work Using Randomized A/B Comparisons


Zavaleta Bernuy, A., Zheng, Q. Y., Shaikh, H., Petersen, A., & Williams, J. J. (2021). Investigating the Impact of Online Homework Reminders Using Randomized A/B Comparisons. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (pp. 921-927). [PDF] [Slides] 


Price, T.W., Williams, J.J., Solyst, J., Marwan, S. (2020) Engaging Students with Instructor Solutions in Online Programming Homework. In CHI 2020, 38th Annual ACM Conference on Human Factors in Computing Systems. [PDF]

Williams, J. J., Lombrozo, T., Hsu, A., Huber, B., & Kim, J. (2016). Revising Learner Misconceptions Without Feedback: Prompting for Reflection on Anomalous FactsProceedings of CHI (2016), 34th Annual ACM Conference on Human Factors in Computing SystemsHonorable Mention for Best Note [top 5%] [PDF] [Slides][Video of Talk ]


Williams, J. J., Lombrozo, T., & Rehder, B. (2013). The hazards of explanation: overgeneralization in the face of exceptions. Journal of Experimental Psychology: General, 142(4), 1006-1014. [PDF]

Gumport, N. B., Williams, J. J., & Harvey, A. G. (2015). Learning cognitive behavior therapy. Journal of Behavior Therapy and Experimental Psychiatry, 48, 164-169. [PDF]

Archival Refereed Conference Proceedings


Zavaleta Bernuy, A., Zheng, Q. Y., Shaikh, H., Nogas, J., Rafferty, Anna., Petersen, A., & Williams, J. J. (2021). Using Adaptive Experiments to Rapidly Help Students. Artificial Intelligence in Education. AIED 2021 (pp. 422-426). [PDF] [Slides] 

Reza, M., Kim, J., Bhattacharjee, A., Rafferty, A.N., & Williams, J.J. (2021). The MOOClet Framework: Unifying Experimentation, Dynamic Improvement, & Personalization in Online Courses. To appear in the Eighth Annual ACM Conference on Learning at Scale. [PDF] [Video of Talk]


Zavaleta Bernuy, A., Zheng, Q. Y., Shaikh, H., Petersen, A., & Williams, J. J. (2021). Investigating the Impact of Online Homework Reminders Using Randomized A/B Comparisons. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (pp. 921-927). [PDF] [Slides] 


Solyst, J., Thakur, T., Dutta, M., Asano, Y., Petersen, A., & Williams, J. J. (2021). Procrastination and Gaming in an Online Homework System of an Inverted CS1. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (pp. 789-795). [PDF]


Li, Z., Yee, L., Sauerberg, N., Sakson, I., Williams, J. J., & Rafferty, A. N. (2020). Getting too personal(ized): The importance of feature choice in online adaptive algorithms. Proceedings of the 13th International Conference on Educational Data Mining (pp. 159-170). [PDF]

Xia, M., Asano, Y., Williams, J. J., Qu, H., Ma, X. (2020) Using Information Visualization to Promote Students’ Reflection on “Gaming the system” in Online Learning. Proceedings of the Seventh ACM Conference on Learning @ Scale (L@S ’20). [PDF]

Price, T. W., Marwan, S., Winters, M., & Williams, J. J. (2020). An Evaluation of Data-Driven Programming Hints in a Classroom Setting. In the International Conference on Artificial Intelligence in Education. [PDF]

Daskalova, N., Yoon, J., Wang, L., Beltran, G., Araujo, C., Nugent, N., McGeary, J., Williams, J., & Huang, J. (2020) SleepBandits: Guided Flexible Self-Experiments for Sleep. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. [PDF]

Price, T.W., Williams, J.J., Solyst, J., Marwan, S. (2020) Engaging Students with Instructor Solutions in Online Programming Homework. In CHI 2020, 38th Annual ACM Conference on Human Factors in Computing Systems. [PDF]

Asano, Y., Solyst, J., Williams, J. J. (2020). Characterizing and Influencing Students’ Tendency to Write Self-explanations in Online Homework. Proceedings of the 10th International Conference on Learning Analytics & Knowledge. [PDF]

Shaikh, H., Modiri, A., Williams, J. J., & Rafferty, A. N. (2019). Balancing Student Success and Inferring Personalized Effects in Dynamic Experiments. Proceedings of the 12th International Conference on Educational Data Mining. [PDF] [Poster]

Wanigasekara, N., Liang, Y., Goh, S. T., Liu, Y., Williams, J. J., & Rosenblum, D. S. (2019). Learning Multi-Objective Rewards and User Utility Function in Contextual Bandits for Personalized Ranking. Proceedings of the 28th International Joint Conference on Artificial Intelligence. [PDF]

Marwan, S., Williams, J. J., & Price, T. (2019). An Evaluation of the Impact of Automated Programming Hints on Performance and Learning. Proceedings of the 15th International Computing Education Research Conference. [PDF]

Marwan, S., N. Lytle, J. J. Williams and T. W. Price (2019). The Impact of Adding Textual Explanations to Next-step Hints in a Novice Programming Environment. Proceedings of the Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE). [PDF] [Slides]

Zhang, L., Craig, M., Kazakevich, M., & Williams, J. J. (2019). Experience Report: Mini Guest Lectures in a CS1 Course via Video Conferencing. Proceedings of the 1st ACM Global Computing Education Conference. [PDF]

Rafferty, A., Ying, H., & Williams, J. J. (2018) Bandit assignment for educational experiments: Benefits to students versus statistical powerProceedings of the 19th International Conference on Artificial Intelligence in Education. [PDF]

Segal, A., David, Y. B., Williams, J. J., Gal, K., & Shalom, Y. (2018). Combining Difficulty Ranking with Multi-Armed Bandits to Sequence Educational Content. Proceedings of the 19th International Conference on Artificial Intelligence in Education. [PDF] [Extended version on arXiv]

Williams, J. J., Rafferty, A., Tingley, D., Ang, A., Lasecki, W. S., & Kim, J. (2018). Enhancing Online Problems Through Instructor-Centered Tools for Randomized Experiments. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems. [PDF[Related Poster]

Shin, H., Ko, E., Williams, J. J., & Kim, J. (2018). Understanding the Effect of In-Video Prompting on Learners and Instructors. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems. [PDF] [Slides]

Foong, P. S., Zhao, S., Tan, F., & Williams, J. J. (2018). Harvesting Caregiving Knowledge: Design Considerations for Integrating Volunteer Input in Dementia Care. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems. [PDF]

Macina, J., Srba, I., Williams, J. J., & Bielikova, M. (2017).  Educational Question Routing in Online Student Communities. Proceedings of the 10th ACM Conference on Recommender Systems. [PDF]

Williams, J. J., Kim, J., Rafferty, A., Maldonado, S., Gajos, K., Lasecki, W., & Heffernan, N. (2016). AXIS: Generating Explanations at Scale with Learnersourcing and Machine LearningProceedings of the Third Annual ACM Conference on Learning at ScaleNominee for Best Paper Award [top 4] [PDF]

Williams, J. J., Lombrozo, T., Hsu, A., Huber, B., & Kim, J. (2016). Revising Learner Misconceptions Without Feedback: Prompting for Reflection on Anomalous FactsProceedings of CHI (2016), 34th Annual ACM Conference on Human Factors in Computing SystemsHonorable Mention for Best Note [top 5%] [PDF]

Ostrow, K., Selent, D., Wang, Y., VanIngwen, E., Heffernan, N., & Williams, J. J. (2016). The Assessment of Learning Infrastructure (ALI): The Theory, Practice, and Scalability of Automated Assessment6th International Learning Analytics & Knowledge Conference. [PDF]

Whitehill, J., Williams, J. J., Lopez, G., Coleman, C., & Reich, J. (2015). Beyond Prediction: First Steps Toward Automatic Intervention in MOOC Student Stopout. Paper presented at the 8th International Conference of Educational Data Mining, Madrid, Spain. [PDF]

Krause, M., Mogale, M., Pohl, H., & Williams, J. J. (2015). A Playful Game Changer: Fostering Student Retention in Online Education with Social Gamification. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale, 95-102. [PDF]

Pacer, M., Williams, J. J., Chen, X., Lombrozo, T., Griffiths, T. L. (2013). Evaluating computational models of explanation using human judgmentsTwenty Ninth Conference on Uncertainty in Artificial Intelligence. [PDF]

Griffiths, T. L., Lucas, C. G., Williams, J. J., Kalish, M. L. (2008). Modeling human function learning with Gaussian processes. Advances in Neural Information Processing Systems 21. [PDF]


Journal Articles

Pathak, L.E., Aguilera, A., Williams, J.J., Lyles, C.R., Hernandez-Ramos, R., Miramontes, J., Cemballi, A.G., & Figueroa, C. (2021). Combining user centered design and crowdsourcing to develop messaging content for a physical activity smartphone application tailored to low-income patients. To appear in JMIR mHealth and uHealthhttps://doi.org/10.2196/21177. [PDF]

Figueroa, F.A., Aguilera, A., Chakraborty, B., Modiri, A., Aggarwal, J., Deliu, N., Sarkar, U., Williams, J.J., & Lyles, C.R. (2021). Adaptive learning algorithms to optimize mobile applications for behavioral health: guidelines for design decisions. Journal of the American Medical Informatics Association, ocab001, 1-10. https://doi.org/10.1093/jamia/ocab001.

Aguilera, A, Figueroa, C.A., Hernandez-Ramos, R., Sarkar, U., Cemballi, A., Gomez-Pathak, L., Miramontes, J., Yom-Tov, E., Chakraborty, B., Yan, X., Xu, J., Modiri, A., Aggarwal, J., Williams, J.J., & Lyles, C.R. (2020) mHealth app using machine learning to increase physical activity in diabetes and depression: clinical trial protocol for the DIAMANTE Study. BMJ Open. 2020 Aug 20https://bmjopen.bmj.com/content/10/8/e034723. [PDF]

Bernecker, S.L., Williams, J.J., Caporale-Berkowitz, N.A., Wasil, A.R., & Constantino, M.J. (2020). Nonprofessional peer support to improve mental health: randomized trial of a scalable web-based peer counseling courseJournal of Medical Internet Research, 22(9). https://doi.org/10.2196/17164. [PDF]

Kizilcec, R. F., Reich, J., Yeomans, M., Dann, C., Brunskill, E., Lopez, D., Turkay, S., Williams, J., & Tingley, D. (2020). Scaling Up Behavioral Science Interventions in Online Education. Proceedings of the National Academy of Sciences (PNAS). [PDF]

Khrosravi, H., Kitto, K., Williams, J.J. (2020) RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities. To appear in Journal of Learning Analytics. [PDF]

Fischer, C., Pardos, Z. A., Baker, R. S. Williams, J.J., Smyth, P., Yu, R., Slater, S., Baker, R., & Warschauer, M. (2020) Mining Big Data in Education: Affordances and Challenges. To appear in Review of Research in Education. [PDF]

Rafferty, A., Ying, H., & Williams, J. J. (2019). Statistical consequences of using multi-armed bandits to conduct adaptive educational experiments. JEDM | Journal of Educational Data Mining, 11(1), 47-79. [PDF]

Edwards, B.J., Williams, J.J., Gentner, D. & Lombrozo, T. (2019) Explanation recruits comparison in a category-learning task. Cognition. 185, 21-38. [PDF]

Walker, C. M., Lombrozo, T., Williams, J. J., Rafferty, A., & Gopnik, A. (2016). Explaining Constrains Causal Learning in Childhood. Child Development, 88(1), 229 - 246. [PDF]

Heffernan, N., Ostrow, K., Kelly, K., Selent, D., Vanlnwegen, E., Xiong, X., & Williams, J. J. (2016). The Future of Adaptive Learning: Does the Crowd Hold the Key? International Journal of Artificial Intelligence in Education, 1 - 30. [PDF]

Miyamoto, Y. R., Coleman, C. A., Williams, J. J., Whitehill, J., Nesterko, S., & Reich, J. (2015)Beyond Time-on-Task: The Relationship Between Spaced Study and Certification in MOOCs. Journal of Learning Analytics, 2(2), 47 - 69. [PDF]

Ostrow, K., Williams, J. J., & Heffernan, N. (in press). The Future of Adaptive Learning: Infusing Educational Technology with Sound Science. Teachers College Record.

Gumport, N. B., Williams, J. J., & Harvey, A. G. (2015). Learning cognitive behavior therapy. Journal of Behavior Therapy and Experimental Psychiatry, 48, 164-169. [PDF]

Lucas, C. G., Griffiths, T. L., Williams, J. J., Kalish, M. L. (2015). A rational model of function learning. Psychonomic Bulletin & Review, 1-23. [PDF]

Harvey, A.G., Lee, J., Williams, J., Hollon, S. Walker, M.P., Thompson, M. & Smith, R. (2014). Improving Outcome of Psychosocial Treatments by Enhancing Memory and Learning. Perspectives in Psychological Science, 9, 161-179. [PDF]

Williams, J. J., & Griffiths, T. L. (2013). Why are people bad at detecting randomness? A statistical argument. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 1473-1490. [PDF]

Williams, J. J., Lombrozo, T., & Rehder, B. (2013). The hazards of explanation: overgeneralization in the face of exceptions. Journal of Experimental Psychology: General, 142(4), 1006-1014. [PDF]

Williams, J. J., & Lombrozo, T. (2013). Explanation and prior knowledge interact to guide learning. Cognitive Psychology, 66, 55–84. [PDF]

Williams, J. J., & Lombrozo, T. (2010). The role of explanation in discovery and generalization: evidence from category learning. Cognitive Science, 34, 776-806. [PDF]


Workshop Papers

Leung, W., & Williams, J. J. (2019). Enhancing education with instructor-in-the-loop algorithms. Paper presented at the Human+AI Modeling & Design workshop at the annual ACM CHI Conference on Human Factors in Computing

Bernecker, S. L., Williams, J. J., & Constantino, M. J. (2017, May). Enhancing mental health through scalable training for peer counselors. Extended abstract presented at the Computing and Mental Health symposium of the annual ACM CHI Conference on Human Factors in Computing Systems, Denver, CO. [PDF]

Williams, J. J., Li, N, Kim, J., Whitehill, J., Maldonado, S., Pechenizkiy, M., Chu, L., & Heffernan, N. (2015). Using the MOOClet Framework as a Problem Formulation to apply Machine Learning to automatically improve modular online educational resources through Experimentation and Personalization. Paper presented at the Human-Propelled Machine Learning Workshop at the Conference on Neural Information Processing Systems.

Williams, J.J. & Williams, B. A. (2013). Using Randomized Experiments as a Methodological and Conceptual Tool for Improving the Design of Online Learning Environments. Paper presented at the Data Driven Education Workshop at the Conference on Neural Information Processing Systems.
Williams, J.J. (2013). Applying Cognitive Science to Online Learning. Paper presented at the Data Driven Education Workshop at the Conference on Neural Information Processing Systems.
Williams, J.J. (2013). Improving Learning in MOOCs by Applying Cognitive Science. Paper presented at the MOOCshop Workshop, International Conference on Artificial Intelligence in Education, Memphis, TN.
Williams, J.J., & Poldsam, H. (2013). Providing implicit formative feedback by combining self-generated and instructional explanations. Paper presented at the Formative Feedback in Interactive Learning Environments Workshop, at the International Conference on Artificial Intelligence in Education, Memphis, TN.


Book Chapter

Williams, J. J., Kim, J., Glassman, E., Rafferty, A., & Lasecki, W. S. (2016). Making Static Lessons Adaptive through Crowdsourcing & Machine Learning. In R. Sottilare, A. Graesser, X. Hu, A. Olney, B. Nye, and A. Sinatra (Eds.). Design Recommendations for Intelligent Tutoring Systems: Volume 4 - Domain Modeling (pp. 127 - 137). Orlando, FL: U.S. Army Research Laboratory. [PDF]


White Papers

Ho, A. D., Chuang, I. R., Reich, J., Coleman, C. A., Whitehill, J., Northcutt, C.G., Williams, J. J., Hansen, J. D., Lopez, G., & Petersen, R. (March 30, 2015). HarvardX and MITx: Two Years of Open Online Courses Fall 2012-Summer 2014. Retrieved from: http://ssrn.com/abstract=2586847 


Using Online Environments to Simultaneously Conduct Basic Research on Learning & Improve Practical Outcomes (Symposia & Workshops)

Williams, J. J., Heffernan, N., Poquet, O. (2018). Design and Application of Collaborative, Dynamic, Personalized Experimentation. Workshop conducted at the 19th International Conference on Artificial Intelligence in Education. London, UK. [PDF]

Krause, M., Hall, M., Williams, J. J., Caton, S., & Pripc, J. (2016). Connecting Online Work and Online Education at Scale. In CHI'16 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery. 

Williams, J. J., Krause, M., Paritosh, P., Whitehill, J., Reich, J., Kim, J., Mitros, P., Heffernan, N., & Keegan, B. C. (2015). Connecting Collaborative & Crowd Work with Online EducationProceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing (pp. 313-318). [Extended Abstract[Website: tiny.cc/crowdworklearning]

Krause, M., Paritosh, P., & Williams, J. J. (2014). Crowdsourcing, Online Education, and Massive Open Online CoursesWorkshop conducted at the Second AAAI Conference on Human Computation and Crowdsourcing.

Williams, J.J., Goldstone, R.L., Rafferty, A., McClelland, J. M., & Mozer, M. (2014). Computational Models for Learning: From Basic Processes to Real World Education. Symposium conducted at the annual convention of the Association for Psychological Science. San Francisco, CA. [Abstract & Summary] [Youtube video of Symposium]. 

Williams, J.J., Teachman, B.A., Richland, L., Brady, S.T, & Aleahmad, T. (2014). Leveraging the Internet to do Laboratory Research in the Real World. Symposium conducted at the annual convention of the Association for Psychological Science. San Francisco, CA. [Abstract & Summary] [Youtube video of Symposium].

Williams, J.J., Kizilcec, R., Russel, D. R., & Klemmer, S. R. (2014). Learning Innovation at ScaleWorkshop at ACM CHI Conference on Human Factors in Computing Systems. Toronto, Canada. [PDF]

Williams, J. J., Linn, M., Edwards, A., Trumbore, A., Chae, H. S., Natriello, G., Saxberg, B., & Mitros, P. (2014). How online resources can facilitate interdisciplinary collaboration. Featured presentation & panel at the Special Interest Group on Computer and Internet Applications in Education, Annual Meeting of the American Educational Research Association. [Summary and More Information]

Williams, J. J., Renkl, A., Koedinger, K., Stamper, J. (2013). Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society, 113-114. Austin, TX: Cognitive Science Society. [PDF]

Williams, J. J., Saxberg, B., Means, B., Mitros, P. (2013). Online Learning and Psychological Science: Opportunities to integrate research and practice. Symposium conducted at the annual convention of the Association for Psychological Science. [description]