Downloadable version can be found at Academic CV.

Positions & Education 

2018- present Assistant Professor, University of Toronto, Department of Computer Science (Budgetary).

2022- present Department of Mechanical & Industrial Engineering, Courtesy Graduate-Level Appointment.

2022- present Department of Economics, Courtesy Graduate-Level Appointment.

2021- present  Department of Statistical Sciences, Courtesy Graduate-Level Appointment. 

2019- present  Department of Psychology, Courtesy Graduate-Level Appointment. 

2019- present Vector Institute for Artificial Intelligence Faculty Affiliate.

2017- 2018 Assistant Professor, National University of Singapore. Department of Information Systems and Analytics, School of Computing.

2014- 2017 Research Fellow, Harvard University. VPAL (Vice Provost for Advances in Learning) Research Group (& HarvardX). Intelligent Interactive Systems Group, Computer Science, School of Engineering & Applied Sciences.

2015- 2018 Research Scientist, Worcester Polytechnic Institute, Department of Computer Science. 

2013- 2014 Postdoctoral Research Scholar, Stanford University. Open Learning Initiative, Lytics Lab, Office of the Vice Provost for Online Learning.

2007- 2013 Ph.D. Computational Cognitive Science. University of California, Berkeley. Dissertation: The Subsumptive Constraints Account of why  explaining “why?” helps learning. (Advisors: Tania Lombrozo & Tom Griffiths).

2003- 2007 B.Sc. Cognitive Science & Artificial Intelligence, Mathematics. University of Toronto.

Publications 

Archival Refereed Conference Proceedings Papers (35)

AC.35 Bhattacharjee, A., Williams, J.J., Meyerhoff, J., Kumar, A., Mariakakis, A. and Kornfield, R., 2023. Investigating the Role of Context in the Delivery of Text Messages for Supporting Psychological Wellbeing. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23) – Best Paper Award.

AC.34 Reza, M., Zavaleta Bernuy, A., Liu, E., Li, Tong., Liang, Z., Barber, C., & Williams, J. J. (2023) Exam Eustress: Designing Brief Online Interventions for Helping Students Identify Positive Aspects of Stress. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), April 23–28, 2023, Hamburg, Germany. ACM, New York, NY, USA, 13 pages.

AC.33 Ye, R., Chen, P., Mao, Y., Wang-Lin, A., Shaikh, H., Zavaleta Bernuy, A., & Williams, J. J. (2022, September). Behavioral Consequences of Reminder Emails on Students’ Academic Performance: a Real-world Deployment. In The 23rd Annual Conference on Information Technology Education (SIGITE ’22)(pp. 16-22) -- Best Paper Award.

AC.32 Zavaleta Bernuy, A., Han, Z., Shaikh, H., Zheng, Q. Y., Lim, L.A., Rafferty, Anna., Petersen, A., & Williams, J. J. (2022). How can Email Interventions Increase Students’ Completion of Online Homework? A Case Study Using A/B Comparisons. Proceedings of the 12th International Conference on Learning Analytics & Knowledge (pp. 107-108).

AC.31 Bhattacharjee, A., Williams, J.J., Chou, K., Tomlinson, J., Meyerhoff, J., Mariakakis, A., & Kornfield, R. (2022). “I Kind of Bounce off It”: Translating Mental Health Principles into Real Life Through Story-Based Text Messages. Proceedings of the ACM on Human-Computer Interaction (CSCW2), 1-30.

AC.30 Kornfield, R., Meyerhoff, J., Levin, H., Bhattacharjee, A., Williams, J. J., Reddy, M., & Mohr, D. C. (2022). Meeting Users Where They Are: User-centered Design of an Automated Text Messaging Tool to Support the Mental Health of Young Adults. Proceedings of the 2022 ACM Conference on Human Factors in Computing Systems (CHI).

AC.29 Reza, M., Kim, J., Bhattacharjee, A., Rafferty, A.N., & Williams, J.J. (2021). The MOOClet Framework: Unifying Experimentation, Dynamic Improvement, & Personalization in Online Courses. In Proceedings of the Eighth Annual ACM Conference on Learning at Scale. 

AC.28  Yang, K. B., Nagashima, T., Yao, J., Williams, J. J., Holstein, K., & Aleven, V. (2021). Can Crowds Customize Instructional Materials with Minimal Expert Guidance? Exploring Teacher-guided Crowdsourcing for Improving Hints in an AI-based Tutor. In Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1-24.

AC.27  Zavaleta Bernuy, A., Zheng, Q. Y., Shaikh, H., Nogas, J., Rafferty, Anna., Petersen, A., & Williams, J. J. (2021). Using Adaptive Experiments to Rapidly Help Students. Artificial Intelligence in Education. AIED 2021 (pp. 422-426).

AC.26 Zavaleta Bernuy, A., Zheng, Q. Y., Shaikh, H., Petersen, A., & Williams, J. J. (2021). Investigating the Impact of Online Homework Reminders Using Randomized A/B Comparisons. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education.

AC.25 Solyst, J., Thakur, T., Dutta, M., Asano, Y., Petersen, A., & Williams, J. J. (2021). Procrastination and Gaming in an Online Homework System of an Inverted CS1. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 

AC.24 Xia, M., Asano, Y., Williams, J. J., Qu, H., & Ma, X. (2020). Using Information Visualization to Promote Students' Reflection on" Gaming the System" in Online Learning. In Proceedings of the Seventh ACM Conference on Learning@ Scale.

AC.23 Price, T. W., Marwan, S., Winters, M., & Williams, J. J. (2020). An Evaluation of Data-Driven Programming Hints in a Classroom Setting. In the International Conference on Artificial Intelligence in Education. 

AC.22 Li, Z., Yee, L., Sauerberg, N., Sakson, I., Williams, J. J., & Rafferty, A. N. (2020). Getting Too Personal (ized): The Importance of Feature Choice in Online Adaptive Algorithms. In Proceedings of the 13th International Conference on Educational Data Mining. 

AC.21 Daskalova, N., Yoon, J., Wang, L., Beltran, G., Araujo, C., Nugent, N., McGeary, J., Williams, J.J., & Huang, J. (2020) SleepBandits: Guided Flexible Self-Experiments for Sleep. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems.

AC.20 Price, T.W., Williams, J.J., Solyst, J., Marwan, S. (2020) Engaging Students with Instructor Solutions in Online Programming Homework. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems.

AC.19 Asano, Y., Solyst, J., Williams, J. J. (2020). Characterizing and Influencing Students’ Tendency to Write Self-explanations in Online Homework. Proceedings of the 10th International Conference on Learning Analytics & Knowledge.

AC.18 Shaikh, H., Modiri, A., Williams, J. J., & Rafferty, A. N. (2019) Balancing Student Success and Inferring Personalized Effects in Dynamic Experiments. Proceedings of the 12th International Conference on Educational Data Mining.

AC.17 Wanigasekara, N., Liang, Y., Goh, S. T., Liu, Y., Williams, J. J., & Rosenblum, D. S. (2019). Learning Multi-Objective Rewards and User Utility Function in Contextual Bandits for Personalized Ranking. Proceedings on the 28th International Joint Conference on Artificial Intelligence.

AC.16 Marwan, S., Williams, J. J., & Price, T. (2019). An Evaluation of the Impact of Automated Programming Hints on Performance and Learning. Proceedings of the 15th International Computing Education Research Conference (ICER).

AC.15 Marwan, S., Lytle, N., Williams, J. J., & Price, T. W. (2019). The Impact of Adding Textual Explanations to Next-step Hints in a Novice Programming Environment. Proceedings of the Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE). 

AC.14 Zhang, L., Craig, M., Kazakevich, M., & Williams, J. J. (2019). Experience Report: Mini Guest Lectures in a CS1 Course via Video Conferencing. Proceedings of the 1st ACM Global Computing Education Conference.

AC.13 Rafferty, A., Ying, H., & Williams, J. J. (2018) Bandit assignment for educational experiments: Benefits to students versus statistical power. Proceedings on the 19th International Conference on Artificial Intelligence in Education.

AC.12 Segal, A., David, Y. B., Williams, J. J., Gal, K., & Shalom, Y. (2018). Combining Difficulty Ranking with Multi-Armed Bandits to Sequence Educational Content. Proceedings on the International Conference on Artificial Intelligence in Education.

AC.11 Williams, J. J., Rafferty, A., Tingley, D., Ang, A., Lasecki, W. S., & Kim, J. (2018). Enhancing Online Problems Through Instructor-Centered Tools for Randomized Experiments. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems.

AC.10 Shin, H., Ko, E., Williams, J. J., & Kim, J. (2018). Understanding the Effect of In-Video Prompting on Learners and Instructors. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems.

AC.9 Foong, P. S., Zhao, S., Tan, F., & Williams, J. J. (2018). Harvesting Caregiving Knowledge: Design Considerations for Integrating Volunteer Input in Dementia Care. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems.

AC.8 Macina, J., Srba, I., Williams, J. J., & Bielikova, M. (2017).  Educational Question Routing in Online Student Communities. Proceedings of the 10th ACM Conference on Recommender Systems.

AC.7 Williams, J. J., Kim, J., Rafferty, A., Maldonado, S., Gajos, K., Lasecki, W. S., & Heffernan, N. (2016). AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning. Proceedings of the Third Annual ACM Conference on Learning at Scale, 379-388. *Nominee for Best Paper.

AC.6 Williams, J. J., Lombrozo, T., Hsu, A., Huber, B., & Kim, J. (2016). Revising Learner Misconceptions Without Feedback: Prompting for Reflection on Anomalous Facts. Proceedings of CHI (2016), 34th Annual ACM Conference on Human Factors in Computing Systems. *Honorable Mention for Best Paper (top 5%). 

AC.5 Ostrow, K., Selent, D., Wang, Y., VanIngwen, E., Heffernan, N., & Williams, J. J. (2016). The  Assessment of Learning Infrastructure (ALI): The Theory, Practice, and Scalability of Automated Assessment. 6th International Learning Analytics & Knowledge Conference, 279-288.

AC.4 Whitehill, J., Williams, J. J., Lopez, G., Coleman, C., & Reich, J. (2015). Beyond Prediction: First Steps Toward Automatic Intervention in MOOC Student Stopout. In Proceedings of the 8th International Conference on Educational Data Mining. Madrid, Spain: International Educational Data Mining Society.  *Nominee for Best Paper.

AC.3 Krause, M., Mogale, M., Pohl, H., & Williams, J. J. (2015). A Playful Game Changer: Fostering Student Retention in Online Education with Social Gamification. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale, 95-102. 

AC.2 Pacer, M., Williams, J. J., Chen, X., Lombrozo, T., Griffiths, T. L. (2013). Evaluating computational models of explanation using human judgments. In Nicholson, A., & Smythe, P. (Eds.), Proceedings of the Twenty Ninth Conference on Uncertainty in Artificial Intelligence, 498 - 507.

AC.1 Griffiths, T. L., Lucas, C. G., Williams, J. J., Kalish, M. L. (2008). Modeling human function learning with Gaussian processes. Advances in Neural Information Processing Systems 21. 

Refereed Journal Articles (23)

J.23 Bhattacharjee, A., Pang, J., Liu, A., Mariakakis, A., Williams, J.J., 2022. Design Implications for One-Way Text Messaging Services that Support Psychological Wellbeing. ACM Transactions on Computer-Human Interaction (TOCHI).

J.22 Meyerhoff, J., Nguyen, T., Karr, C.J., Reddy, M., Williams, J.J., Bhattacharjee, A., Mohr, D.C. and Kornfield, R., 2022. System design of a text messaging program to support the mental health needs of non-treatment-seeking young adults. Procedia Computer Science, 206, pp.68-80. 

J.21 Cai, W., Grossman, J., Lin, Z. J., Sheng, H., Wei, J. T. Z., Williams, J. J., & Goel, S. (2021). Bandit algorithms to personalize educational chatbots. Machine Learning, 110, 2389-2418. Doi

J.20 Pathak, L.E., Aguilera, A., Williams, J.J., Lyles, C.R., Hernandez-Ramos, R., Miramontes, J., Cemballi, A.G., & Figueroa, C. (2021). Combining user centered design and crowdsourcing to develop messaging content for a physical activity smartphone application tailored to low-income patients. JMIR mHealth and uHealth, 9(5), e21177

J.19 Figueroa, F.A., Aguilera, A., Chakraborty, B., Modiri, A., Aggarwal, J., Deliu, N., Sarkar, U., Williams, J.J., & Lyles, C.R. (2021). Adaptive learning algorithms to optimize mobile applications for behavioral health: guidelines for design decisions. Journal of the American Medical Informatics Association, ocab001, 1-10. doi

J.18 Aguilera, A., Figueroa, C.A., Hernandez-Ramos, R., Sarkar, U., Cemballi, A., Gomez-Pathak, L., Miramontes, J., Yom-Tov, E., Chakraborty, C., Yan, X., Xu, J., Modiri, A., Aggarwal, J., Williams, J.J., & Lyles, C.R. (2020). mHealth app using machine learning to increase physical activity in diabetes and depression: clinical trial protocol for the DIAMANTE Study. BMJ Open, 10, e034723. doi

J.17 Bernecker, S.L., Williams, J.J., Caporale-Berkowitz, N.A., Wasil, A.R., & Constantino, M.J. (2020). Nonprofessional peer support to improve mental health: randomized trial of a scalable web-based peer counseling course. Journal of Medical Internet Research, 22(9), e17164. doi

J.16 Kizilcec, R.F., Reich, J., Yeomans, M., Dann, C., Brunskill, E., Lopez, G., Turkay, S., Williams, J.J., & Tingley, D. (2020). Scaling up behavioral science interventions in online education. Proceedings of the National Academy of Sciences, 117(26), 14900–14905. doi

J.15 Khosravi, H., Kitto, K., Williams, J.J. (2020) RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities. Journal of Learning Analytics, 6(3), 91-105. doi 

J.14 Fischer, C., Pardos, Z. A., Baker, R. S. Williams, J.J., Smyth, P., Yu, R., Slater, S., Baker, R., & Warschauer, M. (2020). Mining Big Data in Education: Affordances and Challenges. Review of Research in Education.

J.13 Rafferty, A., Ying, H., & Williams, J. J. (2019). Statistical consequences of using multi-armed bandits to conduct adaptive educational experiments. Journal of Educational Data Mining.

J.12 Edwards, B.J., Williams, J.J., Gentner, D. & Lombrozo, T. (2019) Explanation recruits comparison in a category-learning task. Cognition. 185, 21-38. doi

J.11 Heffernan, N., Ostrow, K., Kelly, K., Selent, D., Vanlnwegen, E., Xiong, X., & Williams, J. J. (2016). The Future of Adaptive Learning: Does the Crowd Hold the Key? International Journal of Artificial Intelligence in Education, 1 -30. doi

J.10 Miyamoto, Y. R., Coleman, C. A., Williams, J. J., Whitehill, J., Nesterko, S., & Reich, J. (2015). Beyond Time-on-Task: The Relationship Between Spaced Study and Certification in MOOCs. Journal of Learning Analytics, 2(2), 47 - 69.

J.9 Walker, C. M., Lombrozo, T., Williams, J. J., Rafferty, A. N., Gopnik, A. (2016). Explaining Constrains Causal Learning in Childhood. Child Development, 88(1), 229-246.

J.8 Ostrow, K., Williams, J. J., & Heffernan, N. (in press). The Future of Adaptive Learning: Infusing Educational Technology with Sound Science. Teachers College Record.

J.7 Gumport, N. B., Williams, J. J., & Harvey, A. G. (2015). Learning cognitive behavior therapy. Journal of behavior therapy and experimental psychiatry, 48, 164-169. doi 

J.6 Lucas, C. G., Griffiths, T. L., Williams, J. J., Kalish, M. L. (2015). A rational model of function learning. Psychonomic Bulletin & Review, 1-23. doi 

J.5 Harvey, A.G., Lee, J., Williams, J. J., Hollon, S. Walker, M.P., Thompson, M. & Smith, R. (2014). Improving Outcome of Psychosocial Treatments by Enhancing Memory and Learning. Perspectives in Psychological Science, 9, 161-179. doi 

J.4 Williams, J. J., & Griffiths, T. L. (2013). Why are people bad at detecting randomness? A statistical argument. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 1473-1490. doi 

J.3 Williams, J. J., Lombrozo, T., & Rehder, B. (2013). The hazards of explanation: overgeneralization in the face of exceptions. Journal of Experimental Psychology: General, 142(4), 1006-1014. doi

J.2 Williams, J. J., & Lombrozo, T. (2013). Explanation and prior knowledge interact to guide learning. Cognitive Psychology, 66, 55-84. doi

J.1 Williams, J. J., & Lombrozo, T. (2010). The role of explanation in discovery and generalization: evidence from category learning. Cognitive Science, 34, 776-806.  doi 

Book Chapters (2)

B.1 Williams, J. J., Kim, J., Glassman, E., Rafferty, A., & Lasecki, W. S. (2016). Making Static Lessons Adaptive through Crowdsourcing & Machine Learning. In R. Sottilare, A. Graesser, X. Hu, A. Olney, B. Nye, and A. Sinatra (Eds.). Design Recommendations for Intelligent Tutoring Systems: Volume 4 - Domain Modeling (pp. 127 - 137). Orlando, FL: U.S. Army Research Laboratory. 

B.2 Leung, W. & Williams, J.J. (2019). Multi-Armed Bandits in Education. In A. Sinatra, A. Graesser, X. Hu, K. Brawner, & V. Rus (Eds.), Design Recommendations for Intelligent Tutoring Systems: Volume 7 - Self-Improving Systems (pp. 71 - 76). Orlando, FL: U.S. Army Research Laboratory. 

Refereed Conference Proceedings Papers (13) 

RC.13 Oliver, M., Renken, M., & Williams, J. J. (2018). Revising biology misconceptions using an online activity with retrieval practice and explanation prompts. Proceedings of the 13th International Conference of the Learning Sciences. ​

RC.12 Edwards, B. J., Williams, J. J., Gentner, D., & Lombrozo, T. (2014). Effects of comparison and explanation on analogical transfer. In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedings of the 36th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society. 

RC.11 Williams, J. J., Walker, C., Maldonado, S. G., Lombrozo, T. (2013). Effects of Explaining Anomalies on the Generation and Evaluation of Hypotheses. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society. 

RC.10 Edwards, B.J., Williams, J.J., & Lombrozo, T. (2013). Effects of explanation and comparison on category learning. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society.

RC.9 Williams, J. J., Walker, C. M., Lombrozo, T. (2012). Explaining increases belief revision in the face of (many) anomalies. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society (pp. 1149-1154). Austin, TX: Cognitive Science Society. 

RC.8 Walker, C. M., Williams, J. J., Lombrozo, T., & Gopnik, A. (2012). Explaining influences children's reliance on evidence and prior knowledge in causal induction. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society (pp. 114-119). Austin, TX: Cognitive Science Society. 

RC.7 Williams, J. J., Lombrozo, T., & Rehder, B. (2011). Explaining drives the discovery of real and illusory patterns. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 1352–1357). Austin, TX: Cognitive Science Society. 

RC.6 Williams, J. J., & Lombrozo, T. (2010). Explanation constrains learning, and prior knowledge constrains explanation. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 2912–2917). Austin, TX: Cognitive Science Society. 

RC.5 Williams, J. J., Lombrozo, T., & Rehder, B. (2010). Why does explaining help learning? Insight from an explanation impairment effect. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 2906–2911). Austin, TX: Cognitive Science Society.

RC.4 Williams, J. J., & Lombrozo, T. (2009). Explaining promotes discovery: evidence from category learning. In N.A. Taatgen & H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp. 1186-1191). Austin, TX: Cognitive Science Society. 

RC.3 Williams, J.J. & Lombrozo, T. (2009). Explaining promotes discovery: evidence from category learning. In Proceedings of the 35th annual conference of the Society for Philosophy and Psychology. 

RC.2 Williams, J. J., & Griffiths, T. L. (2008). Why are people bad at detecting randomness? Because it is hard. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Meeting of the Cognitive Science Society (pp. 64-70). Austin, TX: Cognitive Science Society. 

RC.1 Williams, J. J., & Mandel, D. R. (2007). Do evaluation frames improve the quality of conditional probability judgment? In D. S. McNamara & J. G. Trafton (Eds.), Proceedings of the 29th Annual Meeting of the Cognitive Science Society (pp. 1653-1658), Mahwah, NJ: Erlbaum.

Extended Abstracts (15)

EA.15 Solyst, J., Asano, Y., Williams, J. J. (2019). The instructor reads what you write: Encouraging introductory programming students to engage in self-explanation online. In the 6th Annual Conference on Digital Experimentation @ MIT. 

EA.14 Liu, J., Yang, C., Wang, H., & Williams, J. J. (2019). NoteStruct: Scaffolding Note-taking while Learning from Online Videos. In CHI’19 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery.

EA.13 Williams, J. J., Rafferty, A., Ang, A., Tingley, D. Lasecki, W. S., & Kim, J. (2017). Connecting Instructors and Learning Scientists via Collaborative Dynamic Experimentation. In CHI'17 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery.

EA.12 Williams, J. J., Rafferty, A., Maldonado, S., Ang, A., Tingley, D., & Kim, J. (2017). MOOClets: A Framework for Dynamic Experimentation and Personalization. Proceedings of the Fourth ACM Conference on Learning@Scale. 

EA.11 Krause, M., Hall, M., Williams, J. J., Caton, S., & Pripc, J. (2016). Connecting Online Work and Online Education at Scale. In CHI'16 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery.

EA.10 Williams, J. J. & Heffernan, N. (2015). A Methodology for Discovering how to Adaptively Personalize to Users using Experimental Comparisons. In Late-Breaking Results Workshop of the 23rd Conference on User Modelling, Adaptation, and Personalization. Dublin, Ireland.

EA.9 Williams, J. J., Krause, M., Paritosh, P., Whitehill, J., Reich, J., Kim, J, Mitros, P., & Heffernan, N. (2015). Connecting Collaborative & Crowd Work with Online Education. Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing (pp. 313-318). Vancouver, BC. 

EA.8 Williams, J. J., Ostrow, K., Xiong, X., Glassman, E., Kim, J., Maldonado, S. G., Reich, J., & Heffernan, N. (2015). Using and Designing Platforms for In Vivo Educational Experiments. Proceedings of the Second ACM Conference on Learning@Scale, 409-412.

EA.7 Williams, J. J., Kim, J., Keegan, B. (2015). Supporting Collaborations between Instructors and Researchers using MOOClets. Proceedings of the Second ACM Conference on Learning@Scale, 413-416. 

EA.6 Williams, J.J., Kizilcec, R., Russell, D., & Klemmer, S. (2014). Learning Innovations at Scale. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery.

EA.5 Williams, J. J., Kovacs, G., Walker, C., Maldonado, S. G., & Lombrozo, T. (2014). Learning Online Via Prompts to Explain. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery. 

EA.4 Williams, J. J., Sohl-Dickstein, J., Heffernan, N., Thille, C., & Mitchell, J. C. (2015). Practical and Scientific Benefits of Experiment-Guided Instructional Design of MOOClets and Educational Modules. In J. Davenport  (chair), Theory Driven Design of Online Learning Systems, Data Structures and Analytic Techniques. Annual Meeting of the American Educational Research Association.

EA.3 Williams, J. J., Maldonado, S., Williams, B. A., Rutherford-Quach, S., & Heffernan, N. (2015). How can digital online educational resources be used to bridge experimental research and practical applications? Embedding In Vivo Experiments in “MOOClets”. Spring 2015 Conference of the Society for Research on Educational Effectiveness, SREE. Washington, D. C. 

EA.2 Williams, J. J. & Lombrozo, T. (2014). Why does explaining “why?” help learning? A Subsumptive Constraints Account. In B. Rittle-Johnson (chair), Different Perspectives on the Role of Explanation and Exploration in Learning. Annual Meeting of the American Educational Research Association.

EA.1 Williams, J. J. & Lombrozo, T. (2010). Explaining promotes discovery: Evidence from category learning. In Proceedings of ICLS ‘10, 9th International Conference of the Learning Sciences, 490-491. 

White Papers & Reports (1)

WP.1 Ho, A. D., Chuang, I. R., Reich, J., Coleman, C. A., Whitehill, J., Northcutt, C.G., Williams, J. J., Hansen, J. D., Lopez, G., & Petersen, R. (March 30, 2015). HarvardX and MITx: Two Years of Open Online Courses Fall 2012-Summer 2014. 

Workshop Papers (9)

W.9 Reza, M., Musabirov, I., Liut, M., Laundry, N., & Williams, J. J. (2023) A/B Testing as a Pedagogical Tool for Experiment-Inspired Design in HCI Classrooms. Paper presented at The 5th Annual Symposium on HCI Education (EduCHI). 

W.8 Reza, M., Chowdhury, A., Li, A., Gandhamaneni, M., Williams, J. J. Experimenting with Experimentation: Rethinking The Role of Experimentation in Educational Design. Paper presented at the 3rd Annual Workshop on A/B Testing and Platform-Enabled Learning Research

W.7 Leung, W., & Williams, J. J. (2019). Enhancing education with instructor-in-the-loop algorithms. Paper presented at the Human+AI Modeling & Design workshop at the annual ACM CHI Conference on Human Factors in Computing. 

W.6 Bernecker, S. L., Williams, J. J., & Constantino, M. J. (2017). Enhancing mental health through scalable training for peer counselors. Paper presented at the Computing and Mental Health symposium of the annual ACM CHI Conference on Human Factors in Computing Systems. 

W.5 Williams, J. J., Li, N, Kim, J., Whitehill, J., Maldonado, S., Pechenizkiy, M., Chu, L., & Heffernan, N. (2015). Using the MOOClet Framework as a Problem Formulation to apply Machine Learning to automatically improve modular online educational resources through Experimentation and Personalization. Paper presented at the Human-Propelled Machine Learning Workshop at the Conference on Neural Information Processing Systems. 

W.4 Williams, J.J. & Williams, B. A. (2013). Using Randomized Experiments as a Methodological and Conceptual Tool for Improving the Design of Online Learning Environments. Paper presented at the Data Driven Education Workshop at the Conference on Neural Information Processing Systems. 

W.3 Williams, J.J. (2013). Applying Cognitive Science to Online Learning. Paper presented at the Data Driven Education Workshop at the Conference on Neural Information Processing Systems.

W.2 Williams, J.J. (2013). Improving Learning in MOOCs by Applying Cognitive Science. Paper presented at the MOOCshop Workshop, International Conference on Artificial Intelligence in Education, Memphis, TN. 

W.1 Williams, J.J., & Poldsam, H. (2013). Providing implicit formative feedback by combining self-generated and instructional explanations. Paper presented at the Formative Feedback in Interactive Learning Environments Workshop, at the International Conference on Artificial Intelligence in Education, Memphis, TN.

Fellowships, Grants, and Honors

Grants (11)

G.11 Microsoft: Personalized Messaging Intervention for Young Non-Treatment Seeking Adults Delivered as a Public Health Service. 2022. Co-Principal Investigator. 

G.10 NSF National Science Foundation: Frameworks for Intelligent Adaptive Experimentation: Enhancing and Tailoring Digital Education. (Award #2209819). August 1, 2022 - July 31, 2027. Co-Principal Investigator.

G.9 U.S Office of Naval Research: Investigating the Analysis of Data from Adaptive Experiments. (#N00014-21-1-2576). $427,872. 2021 - 2024. Principal Investigator. 

G.8 Microsoft: Personalized Messaging Intervention for Young Non-Treatment Seeking Adults Delivered as a Public Health Service. 2021. Co-Principal Investigator. 

G.7 NSERC: Enhancing Online User-Facing Systems through Dynamic Experimentation. $140,000, 2019 - 2025. Principal Investigator. #RGPIN-2019-06968

G.6 Connaught Fund: Continually Improving User Interfaces. $20,000, 2019 - 2021. Principal Investigator.

G.5 Office of Naval Research: Personalizing Explanations in Online Problems Using Multi-Armed Contextual Bandits. (N00014-18-1-2755). $499,277. 2018-2021. Principal Investigator.

G.4 National Science Foundation: CIF21 DIBBs: PD: Enhancing and Personalizing Educational Resources through Tools for Experimentation. (1724889) $544,644. 2017 - 2020. Co-Principal Investigator.

G.3 National Science Foundation: SI2-SSE. Adding Research Accounts to the ASSISTments Platform: Helping Researchers Do Randomized Controlled Studies with Thousands of Students. (1440753)  $486,000. 2014 - 2017. Co-Principal Investigator. 

G.2 Gates Foundation & Athabasca MOOC Research Initiative. Investigating the benefits of embedding motivational messages in online exercises. 2013-2014. ($22 500). Principal Investigator. 

G.1 National Institutes of Health Grant Supplement. Improving Learning from Online Cognitive Therapy (2012, $100 000). Co-wrote successful award with Principal Investigator Allison Harvey.

Fellowships

Natural Sciences & Engineering Research Council of Canada Post-Graduate Doctoral Scholarship (2009–2012). 

University of California at Berkeley Regents Intern Fellowship (2007-2009).

Advisory Board Member

“Domain Modeling Techniques.” 2015 collaborative project by Army Research Lab, University of Memphis IIS, and ADL. 

Honors & Awards

Vector Best Poster Award, Toronto Machine Learning Conference (2018).

Nominee for Best Paper Award, 3rd ACM Conference on Learning at Scale (2016). [top 4 papers]

Nominee for Best Paper Award, 8th Conference on Educational Data Mining (2015). [top 3 papers]

Honorable Mention, CHI. [top 5% papers]

Outstanding (Undergraduate Student) Mentor Award, UC Berkeley (2013). 

Semi-finalist, Best poster award, 35th Conference of Society for Philosophy and Psychology (2009).

Trinidad and Tobago Undergraduate Full National Scholarship (2003-2007). 

Professional Activities 

Chairing

CHI (Computer Human Interaction) Subcommittee Co-Chair: Learning, Education and Families 

EDM (Educational Data Mining) Workshops Co-Chair 

Reviewing

Statistical Machine Learning. NeurIPS (Advances in Neural Information Processing Systems), AAAI (Association for the Advancement of Artificial Intelligence).

Human-Computer Interaction. ACM Learning at Scale, CHI (Association for Computing Machinery Conference on Human Factors in Computing Systems), CSCW (Computer Supported Cooperative Work and Social Computing), Human-Computer Interaction (Journal), MobileHCI (International Conference on Human-Computer Interaction with Mobile Devices and Services).

Cognitive Science & Psychology. Annual Conference of the Cognitive Science Society, Social Cognition, Society for Philosophy and Psychology, WIREs (Wiley Interdisciplinary Reviews) Cognitive Science, Journal of Experimental Psychology: Learning, Memory & Cognition, TopiCS in Cognitive Science.

Education. ACM Learning at Scale, EDM (Educational Data Mining Society Conference), Educational Psychology Review, Journal of Science Education and Technology (JOST), IEEE Transactions on Learning Technologies, Instructional Science, Cambridge University Press. 

Program Committees

AAAI, Association for the Advancement of Artificial Intelligence

ACM Learning at Scale

ACM UMAP, User Modeling, Adaptation and Personalization

AIED, Artificial Intelligence in Education

CHI (Specific Applications), ACM Conference on Human Factors in Computing Systems

CogSci, Annual Meeting of the Cognitive Science Society

EDM, Educational Data Mining Conference

HComp (AAAI Conference on Human Computation and Crowdsourcing)

ITS, Conference on Intelligent Tutoring Systems

LWMOOCs, Learning with MOOCs conference

Symposia & Workshops Organized/Chaired (16)

S.15 Ritter, S., Heffernan, N., Williams, J. J., Settles, B., Grimaldi, P., Lomas, D. (2020). Educational A/B Testing at Scale. ACM Learning @ Scale Workshop 2020 (LAS). 

S.14 Azizsoltani, H., Ausin, M. S., Rafferty, A., Williams, J. J., Kim, Y. J., Barnes, T., Chi, M. (2019). Reinforcement Learning for Educational Data Mining. Educational Data Mining Workshop (EDM).

S.13 Williams, J. J., Heffernan, N., & Poquet, O. (2018). Design and Application of Collaborative, Dynamic, Personalized Experimentation. Workshop conducted at the 19th International Conference on Artificial Intelligence in Education (AIED).

S.12 Williams, J. J., Abbasi, Y., Doshi-Velez, F. (2015). Machine Learning From and For Adaptive User Technologies: From Active Learning & Experimentation to Optimization & Personalization. 29th Annual Conference on Neural Information Processing Systems (NeurIPS). 

S.11 Krause, M., Hall, M., Williams, J. J., Caton, S., & Pripc, J. (2016). Connecting Online Work and Online Education at Scale. In CHI'16 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery.

S.10 Williams, J. J., Krause, M., Paritosh, P., Whitehill, J., Reich, J., Kim, J, Mitros, P., & Heffernan, N. (2015). Connecting Collaborative & Crowd Work with Online Education. Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing (pp. 313-318). Vancouver, BC. 

S.9 Williams, J.J., Kizilcec, R., Russell, D., & Klemmer, S. (2014). Learning Innovations at Scale. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery. 

S.8 Williams, J. J., Linn, M., Edwards, A., Trumbore, A., Chae, H. S., Natriello, G., Siemens, G., & Mitros, P. (2014). How online resources can facilitate interdisciplinary collaboration. Featured presentation & panel at the Special Interest Group on Computer and Internet Applications in Education, Annual Meeting of the American Educational Research Association.

S.7 Krause, M., Paritosh, P., & Williams, J. J. (2014). Crowdsourcing, Online Education, and Massive Open Online Courses. Workshop conducted at the Second AAAI Conference on Human Computation and Crowdsourcing. 

S.6 Williams, J.J., Goldstone, R.L., Rafferty, A., McClelland, J, & Mozer, M. (2014). Computational Models for Learning: From Basic Processes to Real World Education. Symposium conducted at the annual convention of the Association for Psychological Science. San Francisco, CA. 

S.5 Williams, J.J., Teachman, B.A., Richland, L., Brady, S.T, & Aleahmad, T. (2014). Leveraging the Internet to do Laboratory Research in the Real World. Symposium conducted at the annual convention of the Association for Psychological Science. San Francisco, CA. 

S.4 Williams, J. J., Renkl, A., Koedinger, K., Stamper, J. (2013). Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society. 

S.3 Williams, J. J., Saxberg, B., Means, B., Mitros, P. (2013). Online Learning and Psychological Science: Opportunities to integrate research and practice. Symposium conducted at the annual convention of the Association for Psychological Science. 

S.2 Williams, J. J., Legare, C., Gentner, D., McNamara, D. (2013). Enhancing education: The role of comparing and explaining examples in promoting abstraction and transfer. Symposium conducted at the annual convention of the Association for Psychological Science.

S.1 Chi, M. T. H., DeJong, G., Legare, C., Lombrozo, T., Williams, J. J. (2011). Explanation-based mechanisms for learning: An interdisciplinary approach. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 2756–2757). Austin, TX: Cognitive Science Society. 

Summary of Invited Talks

University of Michigan (Academic Innovation at Michigan Analytics), University of Edinburgh (School of Informatics), University of Pittsburgh (Intelligent Systems Program AI Forum), Brown University (Department of Cognitive, Linguistic, and Psychological Sciences), Worcester Polytechnic Institute (Computer Science), University of Freiburg (Educational Psychology), University of Amsterdam (Psychology), Army Research Lab Generalized Intelligent Framework for Tutoring Symposium, Carnegie Mellon University (Human Computer Interaction Institute, School of Computer Science), Qualtrics Insight Summit, McGraw Hill Education Meetup on Predictive Analytics for Education, MIT (CSAIL Computational Cognitive Science Group), Harvard University (HarvardX), Udacity, Declara, Stanford University (Psychology, Cognition Frisem), Arizona State University (Learning Sciences Institute), Coursera, Stanford University (Instructional Design Special Interest Group), Stanford Psychological Interventions in Educational Settings (PIES) group, Stanford University (Human Computer Interaction brown Bag Seminar), International Association for K-12 Online Learning (iNACOL) Webinar Series, EdX, Pittsburgh Science of Learning Center LearnLab Summer School, Stanford University (Lytics Online Education Lab), UC Berkeley (Graduate School of Education), UC Berkeley (Psychology), Google (Tech Talk), Khan Academy, EdX, Bloomsburg Corporate Advisory Council, Vector Institute for Artificial Intelligence (Vector Talks), Element AI, Spring Symposium on Artificial Intelligence for K-12 Education. 

University College London Interaction Centre (UCLIC). Perpetually Enhancing and Personalizing Technology for Learning & Health Behavior Change: Using Randomized A/B Experiments to integrate Psychology, Crowdsourcing & Statistical Machine Learning

MILA (a Quebec Institute for Machine Learning & Artificial Intelligence). Combining Reinforcement Learning & Human Computation for A/B Experimentation: Perpetually Enhancing and Personalizing User Interfaces

North Carolina State University, Computer Science & Center for Educational Informatics. Enhancing & personalizing educational & health technology: Randomized A/B Experiments to bridge human-computer interaction, psychology, and statistical machine learning

University of Washington Design. Use. Build. (DUB). Perpetually Enhancing and Personalizing Technology for Learning & Health Behavior Change: Using Randomized A/B Experiments to integrate Human-Computer Interaction, Psychology, Crowdsourcing & Statistical Machine Learning

Stanford Pervasive Wellbeing Technology Lab (Medical School). Combining Reinforcement Learning, Psychology & Human Computation for Randomized A/B Experimentation: Perpetually Enhancing and Personalizing User Interfaces.

University of Alberta Tea Talk. Combining Reinforcement Learning & Human Computation for A/B Experimentation: Perpetually Enhancing and Personalizing User Interfaces.

Carnegie Mellon University LearnLab. Applying Multi-Armed Bandits (Reinforcement Learning)

to A/B Testing to Enhance & Personalize Education.

University of Toronto Social-Personality Research Group. Conducting Adaptive Field Experiments that

Enhance and Personalize Education and Health Technology.

University of Columbia, Statistics. Challenges & Opportunities in using Reinforcement Learning for Dynamic Field Experimentation in User Interfaces: Tradeoffs between Statistical Inference and Enhancing User Experience.

University of Pennsylvania Duckworth Lab. Doing Psychology Research in the Real World: Using Technology for Adaptive Field Experiments in Education and Health.

Office of Naval Research Program Meeting. Enhancing Digital Resources using Multi-Armed Bandits for Crowdsourced & Dynamic Experimentation

Teaching Experience

Undergraduate courses taught

• Human-Computer Interaction Winter 2021, Fall 2021, Winter 2022

  • Taught 4th year undergraduate course at the University of Toronto (Computer Science, CSC 428), focused on understanding human behaviour as it applies to user interfaces. 

  • Cross-listed as graduate course 2514

• Introduction to Computer Programming: Synchronous Inverted In Person (Winter 2020) 

  • Taught CS1 1st year undergraduate course at University of Toronto (Computer Science, CSC 108), introduction to programming using python. 

• Introduction to Computer Programming: Online Asynchronous (Winter 2019; Winter 2021) 

  • Taught CS1 1st year undergraduate course at University of Toronto (Computer Science, CSC 108), introduction to programming using python. 

• Research Opportunity Program 

  • Supervisor for a 2nd/3rd year undergraduate research course at University of Toronto (Computer Science, CSC299/CSC399). Supervised 19 undergraduate students from 2020-2021 as they completed individual research projects in my lab. 

• Computer Science Research Project Courses 

  • Supervisor for 4th year undergraduate research course at University of Toronto (Computer Science, CSC 494/495). Supervised 2 undergraduate students from 2019-2020 as they completed individual research projects in my lab. 

• Social Media Network Analysis 

  • Taught 4th year undergraduate course at National University of Singapore (Information Systems & Analytics, School of Computing). 

Graduate courses taught

• Intelligent Adaptive Interventions: Designing Intelligent Self-Improving Systems Through Human Computation, Randomized A/B Experiments and Statistical Machine Learning  (Fall 2018, 2019, 2020, 2021)

  • Graduate seminar at University of Toronto (Computer Science, CSC 2558, Multidisciplinary Topics in HCI), covering approaches from multiple disciplines (HCI, machine learning, psychology/cognitive science) related to use of randomized A/B experiments to enhance user-facing technology. 

Guest Lecturer for MOOC on Applications of Calculus 

  Created 4 lecture videos on applying statistics to educational assessment (item response theory) and dynamic experimentation (multi-armed bandits).

Instructor for Short Courses

CHI Course “Online A/B Tests & Experiments” 

“In Vivo Experimentation” Track,  Graduate Summer School

Carnegie Mellon Learning Technology Summer School.

Scientific and Technology mentor for Lean Launchpad course (based on NSF Innovation Corps)

Taught by Steve Blank and Jerry Engels (2013). Mentored Ed-Tech & other startups, using the Launchpad Central online tool and in-person classes similar to what the instructors founded as NSF’s Innovation Corps (which teaches scientists to translate research into businesses). 

Group Leader for Workshop teaching management skills to Stanford Graduate Students

Taught management skills to a group of Stanford graduate students, through a program sponsored by the Vice Provost for Graduate Education (2014).

Mentor for "Educational Data Mining" Track, Graduate Summer School

Carnegie Mellon Learning Technology Summer School.

Teaching Assistant/Graduate Student Instructor

Research and Data Analysis in Psychology. 

Basic Issues in Cognition. 

K12 Tutoring & Teaching Experience

Classroom tutor placed to support new math teachers, 9th grade Geometry and Algebra. 

Developed & delivered K-12 & university problem-based classes on chemistry.

Mentoring and Leadership

Graduate Student Supervision

(Angela Zavaleta Bernuy, Computer Science Master's Student, University of Toronto, 2020-present)

(Ananya Bhattacharjee, Computer Science PhD student, University of Toronto, 2020-present)

(Mohi Reza, Computer Science PhD student, University of Toronto, 2020-present)

(Harsh Kumar, Computer Science PhD student, University of Toronto, 2021-present)

(Ilya Musabirov, Computer Science PhD student, University of Toronto, 2021-present)

(Tong Li, Statistics PhD student, University of Toronto, 2021-present)

(Arghavan Modiri, Computer Science Master's Student, University of Toronto, 2019-2021)

(Jacob Nogas, Computer Science Master's Student, University of Toronto, 2019-2021)

(co-advisor: Nina Deliu, PhD, University of Rome, 2020-2021)

(Sam Cox, Computer Science PhD Student, National University of Singapore, 2018-2019)

PhD Dissertation Committee Member

(Lillio Mok, University of Toronto, Computer Science)

(Laura Niss; University of Michigan, Statistics)

(Mohammad Kian Kianpisheh, University of Toronto, Computer Science)

(Vaunam Venkadasalam, University of Toronto, Ontario Institute for Studies in Education – OISE)

(Zhicong Lu, University of Toronto, Computer Science)

(Amna Liaquat, University of Toronto, Computer Science)

(Devangini Patel, National University of Singapore, Computer Science)

(Merrin Oliver, Georgia State University, Psychology, 2017)

Graduate Student Examination/Prelim Committees

(Laura Niss; Prelim Committee Member; University of Michigan, Statistics)

(Ding Dan & Luo Kai; Graduate Paper Examiner; National University of Singapore, Information Systems)

(Yuanxin Xiang; Master's Thesis Examiner; National University of Singapore, Computer Science)

Mentor for NIH PhD Training Grant on Online Cognitive Therapy

(Samantha Bernecker, University of Massachusetts Amherst, 2013-2017).

Mentorship of Undergraduate Students at University of Toronto (2018-2021)

Yuya Asano, Ben Prystawski, Jai Aggarwal, Samantha Quinto, Gavin Zheng.

Mentorship of 27 Undergraduate Students at UC Berkeley (2008-2013)

Students: Stephen Frey, Joyce Liu, Lindsay Reno, Christopher Gioia, Jerry Ling, Tram Dinh, Tim Ko, Summer Kim, Evan Kim, Lucie Vosicka, Jing Wang, Emily Margolin, Vanessa Ing, Sean Trott, Kelly Whiteford, Adam Krause, Samuel Maldonado, Norielle Adricula, Amanda Hu, Dhruba Banerjee, Preeti Talwai, Christopher Hur, Hava Edelstein, Jared Lorenz, Ania Jarosewicz, Olivia Lin, David Oh.

Outstanding Graduate Student Mentor Award, UC Berkeley (2013) 

Departmental award for mentoring undergraduate students in conducting research.