Course on Dynamic A/B Testing

UofT Grad Course: Designing Intelligent Self-Improving Systems Through Human Computation, Randomized A/B Experiments and Statistical Machine Learning

CSC2558: Topics in Multidisciplinary HCI: Designing Intelligent Self-Improving Systems Through Human Computation, Randomized A/B Experiments and Statistical Machine Learning

Course enrollment, Auditing, and other details

Anyone is welcome to attend the first few classes. Auditors are welcome, especially upper-year students who want to work on projects on dynamic experimentation. Computer science students can register immediately. Students in other departments are welcome to take the course, and will likely get a spot, but ACORN will only allow enrollment after a certain date.  

Access to Materials

To get access to the course materials (slides, readings, recorded lectures) as they become available, request to join the google group "Info For Grad Course: Designing Intelligent Self-Improving Systems". The course be provided as a synchronous (and asynchronous) online course, and a wide range of materials have been put together that allow self-paced study.

This course introduces students to computational principles for designing user-facing systems that are intelligent and continually improving, drawing on interdisciplinary work in crowdsourcing, human computation, psychology, experimental design, and statistical machine learning. Students will learn principles for enhancing and personalizing adaptive user-facing systems through randomized A/B experimental comparisons. (Examples of systems include lessons in online courses, activities for mental health, apps for encouraging exercise and other health behavior change, marketing and product design). Students will learn how to design and conduct randomized A/B experiments that are collaborative, dynamic, and personalized. 

Collaborative experiments can require combining multiple stakeholders in design: drawing on theories from social and behavioral sciences to design alternative versions of a user-facing system (e.g. educational theories about learning, psychology of goal-setting, clinical, and public health insights into cognitive behaviour therapy), the practical experience of designers, and using crowdsourcing techniques to have users themselves participate in designing improvements to be experimentally evaluated. 

Dynamic and personalized experiments require using statistics and machine learning or other artificial intelligence techniques to discover in real time which conditions are effective (on average, and for subgroups of users). These models and algorithms may include reinforcement learning, multi-armed (contextual) bandits, and Adaptive Clinical Trials. The course will combine multiple disciplines and involve collaborative work (e.g. teams of students with backgrounds in social and behavioural science, human-computer interaction and crowdsourcing, and statistics or machine learning).

Social, behavioural and health science students will also learn about statistics and machine learning techniques for dynamically analyzing data from experiments, in order to improve outcomes for future participants. They will learn to apply algorithms that analyze data to present more beneficial conditions more frequently to future participants (e.g. transition from 50/50 randomization to weighted randomization of 60/40, 80/20…). In addition, modelling of participant-treatment interactions (e.g. condition A is more effective for people with attitude X) can lead to the personalization of technology (giving different conditions to different subgroups).

Social, behavioural and health science students are not required to have any programming experience, as they will have the opportunity to collaborate with computer science students in building web software. They will also be able to learn about statistics and machine learning techniques by collaborating with grad students who specialize in these areas. In return, social, behavioural and health science students can provide computer science students with insights into how to apply social and behavioural theories to enhancing technology, and how to design rigorous and principled experiments.

The course will be seminar-style and project-based, where an ideal outcome is that a student can gain input and learn how to apply computer science techniques to a set of experiments or research questions central to their work.

Examples of relevant papers to discuss

Kizilcec, R. F., Saltarelli, A. J., Reich, J., & Cohen, G. L. (2017). Closing global achievement gaps in MOOCs. Science, 355(6322), 251-252. [PDF]

Meurer, W. J., Lewis, R. J., Tagle, D., Fetters, M. D., Legocki, L., Berry, S., … Barsan, W. G. (2012). An Overview of the Adaptive Designs Accelerating Promising Trials Into Treatments (ADAPT-IT) Project. Annals of Emergency Medicine, 60(4), 451–457. [PDF]

Scott, S. L. (2010). A modern Bayesian look at the multi‐armed bandit. Applied Stochastic Models in Business and Industry, 26(6), 639-658. [PDF]

Williams, J. J., Rafferty, A., Tingley, D., Ang, A., Lasecki, W. S., & Kim, J. (2018). Enhancing Online Problems Through Instructor-Centered Tools for Randomized Experiments. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems. [PDF] [Talk Slides] [Video Figure] [Video of Talk]

Williams, J. J., Lombrozo, T., Hsu, A., Huber, B., & Kim, J. (2016). Revising Learner Misconceptions Without Feedback: Prompting for Reflection on Anomalous Facts. Proceedings of CHI (2016), 34th Annual ACM Conference on Human Factors in Computing Systems.  [PDF] [Slides] [Video of Talk ]

Williams, J. J., Kim, J., Rafferty, A., Maldonado, S., Gajos, K., Lasecki, W., & Heffernan, N. (2016). AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning. Proceedings of the Third Annual ACM Conference on Learning at Scale.  [PDF] [Slides]

Lomas, J. D., Forlizzi, J., Poonwala, N., Patel, N., Shodhan, S., Patel, K., ... & Brunskill, E. (2016, May). Interface Design Optimization as a Multi-Armed Bandit Problem. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 4142-4153). ACM.

Expanded (rougher) list of papers

Examples of weekly topics are: 

• Human-Computer Interaction research on crowdsourcing and human computation

  • Getting users to help improve a system by suggesting new system components (e.g. explanations, questions to ask people to reflect) which machine learning algorithms can then test out, to discover what is working. 

  • Example: AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning

• Applications of algorithms from reinforcement learning, Bayesian optimisation and Recommender Systems 

  • Doing real-time data analysis of which messages motivate people to respond to emails; discovering what works for different users, and rolling these out to future people. 

  • Example: Combining Dynamic A/B Experimentation and Recommender Systems in MOOCs

• Cognitive Science research in designing user studies and A/B comparisons 

  • Determining which lessons or motivational messages help people learn.

  • Example: Connecting Instructors and Learning Scientists via Collaborative Dynamic Experimentation

Recommended (not required) preparation is one of the following courses (few students will have multiple): human-computer interaction, psychology/economics/public health/business course on theories of behaviour/learning/decision making, experimental design, machine learning, introductory statistics.