We are accepting applications for PhD (and Master's) programs for Fall 2023 entry. Apply to UofT programs in Computer Science, Psychology or Statistics, ideally by November. Students can apply through Computer Science, or through Psychology or Statistics (where I have cross-appointments and can supervise students). Our lab is highly interdisciplinary and currently includes students from HCI, Machine Learning, Psychology, Mental Health, Education, Economics, and Statistics – no one student is expected to have all these skills, they all teach other!
If you want to learn more about the lab culture, read and comment Value Statement http://tiny.cc/valuesiai2021 (request access). You can also get a sense of our lab culture by looking at these videos which some students kindly recorded for me on my birthday! Full 9 minute version and shorter prototype A and shorter prototype B some students created.
Most of my past publications focus on education and learning, but I have a substantive set of work on health behaviour and habit change (e.g. encouraging exercise, mental health), and on projects in areas as wide-ranging as behavioural economics (encouraging charitable donations) and workforce development.
As a graduate student, you would set the agenda for research questions in collaboration with me. Just to give examples, I'm particularly interested in creating systems that combine rigorous randomized experiments with crowdsourcing and human computation, applications of statistical machine learning (e.g. bandits & reinforcement learning, NLP, recommender systems), and theories from cognitive, clinical and social psychology (e.g. self-explanation, analogical comparison, growth mindset, teaching cognitive behaviour therapy).
To find out more about what I do, you can read my Research Statement, read one or two relevant Papers and email me with a summary or reflections & questions, or look at these four talks I've given to HCI, Psychology, Machine Learning, Statistics. You can choose whichever are most relevant to the area you want to work in: Talks Illustrating Examples of Lab Research. Based on alignment of interests and time, students may have opportunities to collaborate with people in U of T's Computer Science Education research group (e.g. Andrew Petersen), the Vector Institute/Artificial Intelligence/Machine Learning group (e.g. Amir Massoud Farahmand, Marzyeh Ghassemi), HCI people at DGP (e.g. Tovi Grossman, Fanny Chevalier), Psychology Department (e.g. Cendri Hutcherson, Mickey Inzlicht), the Education School OISE, and many other areas like Computational Social Science (e.g. Ashton Anderson). Graduate students will play a key role in deciding which projects are pursued, but illustrative examples of potential research directions are:
If you're interested, please apply for the Ph.D program in Computer Science, or Psychology, or Statistics, and list me as a potential advisor. Note that the Master's is a research program available to Canadian Citizens or Permanent Residents (and the MScAC is a professional master's without a research component).
You can also send an email to iaiinterest@googlegroups.com with information about yourself, what relevant research experience you have, what parts of my website you've looked at and what you found interesting about them, what topics you're interested in and why, and why you want to pursue a PhD program.
Talks Illustrating Examples of Lab ResearchHCI (Human-Computer Interaction) targeted talk (UWashington DUB/HCI series)Slides: tiny.cc/iaislides Recording: tiny.cc/iairecording Google slides with otter transcription: https://docs.google.com/presentation/d/1m8XaqRXqX6DfZItmyqJbUcW1bPSv3Bspl2nJKTS1_wQ/edit#slide=id.g9685929bb8_2_1294 Short Title: Enhancing and Personalizing Technology Through Dynamic Experimentation Long Title: Perpetually Enhancing and Personalizing Technology for Learning & Health Behavior Change: Using Randomized A/B Experiments to integrate Human-Computer Interaction, Psychology, Crowdsourcing & Statistical Machine Learning How can we transform the everyday technology people use into intelligent, self-improving systems? Our group applies statistical machine learning algorithms to analyze randomized A/B experiments and give the most effective conditions to future users. Ongoing work includes comparing different explanations for concepts in digital lessons/problems, getting people to exercise by testing motivational text messages, and discovering how to personalize micro-interventions to reduce stress and improve mental health. One example system crowdsourced explanations for how to solve math problems from students and teachers, and conducted an A/B experiment to identify which explanations other students rated as being helpful. We used algorithms for multi-armed bandits that analyze data in order to estimate the probability that each explanation is the best, and adaptively weight randomization to present better explanations to future learners (LAS 2016, CHI 2018). This generated explanations that helped learning as much as those of a real instructor. Ongoing work aims to personalize, by discovering which conditions are effective for subgroups of users. We use randomized A/B experiments in technology as an engine for practical improvement, in tandem with advancing research in HCI, psychological theory, statistics, and machine learning Machine Learning targeted talk (Vector, MILA-McGill/Stanford, Microsoft Research New York)Vector Institute for Artificial Intelligence Jan 2021: Slides: Vector Talk Shared Adapting Real-World Experimentation To Balance Enhancement of User Experiences with Statistically Robust Scientific Discovery.pptx Short Title: Adapting Real-World Experimentation To Balance Enhancement of User Experiences with Statistically Robust Scientific Discovery Long Title: Perpetually Enhancing User Interfaces in Tandem with Advancing Scientific Research in Education & Mental Health: Enabling Reliable Statistical Analysis of the Data Collected by Algorithms that Trade Off Exploration & Exploitation How can we transform the everyday technology people use into intelligent, self-improving systems? For example, how can we perpetually enhance text messages for managing stress, or personalize explanations in online courses? Our work explores the use of randomized adaptive experiments that test alternative actions (e.g. text messages, explanations), aiming to gain greater statistical confidence about the value of actions, in tandem with rapidly using this data to give better actions to future users. To help characterize the problems that arise in statistical analysis of data collected while trading off exploration and exploitation, we present a real-world case study of applying the multi-armed bandit algorithm TS (Thompson Sampling) to adaptive experiments. TS aims to assign people to actions in proportion to the probability those actions are optimal. We present empirical results on how the reliability of statistical analysis is impacted by Thompson Sampling, compared to a traditional experiment using uniform random assignment. This helps characterize a substantial problem to be solved – using a reward maximizing algorithm can cause substantial issues in statistical analysis of the data. More precisely, an adaptive algorithm can increase both false positives (believing actions have different effects when they do not) and false negatives (failing to detect differences between actions). We show how statistical analyses can be modified to take into account properties of the algorithm, but that these do not fully address the problem raised. We therefore introduce an algorithm which assigns a proportion of participants uniformly randomly and the remaining participants via Thompson sampling. The probability that a participant is assigned using Uniform Random (UR) allocation is set to the posterior probability that the difference between two arms is 'small' (below a certain threshold), allowing for more UR exploration when there is little or no reward to be gained by exploiting. The resulting data can enable more accurate statistical inferences from hypothesis testing by detecting small effects when they exist (reducing false negatives), and reducing false positives. The work we present aims to surface the underappreciated complexity of using adaptive experimentation to both enable scientific/statistical discovery and help real-world users The current work takes a first step towards computationally characterizing some of the problems that arise, and what potential solutions might look like, in order to inform and invite multidisciplinary collaboration between researchers in machine learning, statistics, and the social-behavioral sciences. Alternative Talks/Previous VersionsTalk at MILA (Montreal Machine Learning & Artificial Intelligence) Dec 2019; Recording: https://bluejeans.com/playback/s/WffXkZQ5VLWAXi15vrsIR8C4ym0rUgIHuftE7jBvjVfdIBRpepJDqJ9vaJfDfIQd Psychology targeted talk (Social-Personality & Cognitive Psych at U of T)Indiana University Psychology Talk Dec 2019 Slides: IU CogSci v2.pptx Recording: Indiana Psychology Talk Dec 2019.mp4 . U of T Social Psychology Research Group talk Sep 2019: Slides: SPRG Talk v5.pptx and Recording: SPRG Talk (rough, unedited).mp4. Title: Conducting Adaptive Field Experiments that Enhance and Personalize Education and Health Technology Understanding people’s complex real-world thinking is a challenge for psychology, while human-computer interaction aims to build computational systems that can behave intelligently in the real-world. This talk presents a framework for redesigning the everyday websites people interact with to function as: (1) Micro-laboratories for psychological experimentation and data collection, (2) Intelligent adaptive agents that implement machine learning algorithms to dynamically discover how to optimize and personalize people’s learning and reasoning. I present an example of how this framework is used to embed randomized experiments into-real world online educational contexts – like learning to solve math problems– and machine learning used for automated experimentation. Explanations (and experimental conditions) are crowdsourced from teachers and scientists, and reinforcement learning algorithms for multi-armed bandits used in real-time to discover the best explanations for optimizing learning. Ongoing research examines tools for managing stress by applying approaches like cognitive behavior therapy, helping university students self-regulate and plan, and encouraging health behavior change through a collaboration with Goodlife to get people to go to the gym.Statistics targeted talk (Cambridge, UMichigan, Columbia)Title: Adapting Real-World Experimentation To Balance Enhancement of User Experiences with Statistically Robust Scientific Discovery
Cambridge University May 2021 Slides: tiny.cc/jstatstalk Recording: https://www.youtube.com/watch?v=vV1ShFophpM How can we transform the everyday technology people use into intelligent, self-improving systems? For example, how can we perpetually enhance text messages for managing stress, or personalize explanations in online courses? Our work explores the use of randomized adaptive experiments that test alternative actions (e.g. text messages, explanations), aiming to gain greater statistical confidence about the value of actions, in tandem with rapidly using this data to give better actions to future users.To help characterize the problems that arise in statistical analysis of data collected while trading off exploration and exploitation, we present a real-world case study of applying the multi-armed bandit algorithm TS (Thompson Sampling) to adaptive experiments, providing more empirical context to issues raised by past work on adaptive clinical trials. TS aims to assign people to actions in proportion to the probability those actions are optimal. The empirical results help characterize how a reward maximizing algorithm can increase both false positives (believing actions have different effects when they do not) and false negatives (failing to detect differences between actions). We explore two methods that take into account properties of the TS algorithm used to collect data: inverse-probability weighting and an 'algorithm-induced hypothesis test' that uses non-parametric simulations under the null. These help but do not fully address the problems raised. We therefore introduce an algorithm which assigns a proportion of participants uniformly randomly and the remaining participants via Thompson sampling. The probability that a participant is assigned using Uniform Random (UR) allocation is set to the posterior probability that the difference between two arms is 'small' (below a certain threshold), allowing for more UR exploration when there is little or no reward to be gained by exploiting. The resulting data can enable more accurate statistical inferences from hypothesis testing by detecting small effects when they exist (reducing false negatives), and reducing false positives. The work we present aims to surface the underappreciated complexity of using adaptive experimentation to both enable scientific/statistical discovery and help real-world users. We conduct field deployments and provide software the community can use to evaluate statistical tests and algorithms in complex real-world applications. This helps provide first steps towards integrating two key approaches: (1) How can we modify statistical tests to better match the properties of the algorithms that collect data in adaptive experiments? (2) How can we modify algorithms for adaptive experimentation to be more "statistically considerate" in being better suited to inference and analysis of data, while maximizing chances of giving participants useful arms? Tackling these questions requires multidisciplinary collaboration between researchers in machine learning, statistics, and the social-behavioral sciences. This is joint work with Nina Deliu, Sofia Villar, Audrey Durand, Anna Rafferty and others. 11 min TEDx Talk for broad audiencebit.ly/tedxwilliams Is the internet replacing teachers? If you liked what you see and are ready to join: Optionally, write to iaiinterest@googlegroups.com with information about yourself, relevant research experience, what topics you're interested in and why, and why you want to pursue a PhD program. Look at Computer Science (for HCI, ML, CS Education), Psychology, or Statistics program descriptions and requirements Follow Application Checklist & Process at School of Graduate Studies website BioJoseph Jay Williams is an Assistant Professor in Computer Science at the University of Toronto, has courtesy appointments in Statistical Sciences & Psychology (to accept & supervise PhD students), and is a Vector Institute for Artificial Intelligence Faculty Affiliate. He leads the Intelligent Adaptive Interventions research group of 11 graduate students across HCI, Psychology, applied ML, & Statistics (www.intadaptint.org). He was previously an Assistant Professor at the National University of Singapore's School of Computing in the department of Information Systems & Analytics, a Research Fellow at Harvard's Office of the Vice Provost for Advances in Learning, and a member of the Intelligent Interactive Systems Group in Computer Science. He completed a postdoc at Stanford University in Summer 2014, working with the Office of the Vice Provost for Online Learning and the Open Learning Initiative. He received his PhD from UC Berkeley in Computational Cognitive Science (with Tom Griffiths and Tania Lombrozo), where he applied Bayesian statistics and machine learning to model how people learn and reason, and used randomized experiments to develop theories of how people learn by explaining to themselves and others. He received his B.Sc. from University of Toronto in Cognitive Science, Artificial Intelligence and Mathematics, and is originally from Trinidad and Tobago. More information about the Intelligent Adaptive Intervention group's research and papers is at www.josephjaywilliams.com.
|