Talks"Excellent Online Education" (EdX)Title: Excellent online education Abstract: The expanding world of online learning has provided environments that compete along many dimensions, but one is paramount: The depth, speed, and longevity of student learning. Despite the breakneck pace of online development, the challenge of producing excellent education can be most effectively met by building on what is already known: incorporating the best scientific research on learning, and surveying outstanding educational practice. Examples of evidence-based principles for improving learning in an online course are presented in the context of the EdX platform: Teaching for Transfer, Teaching Incremental Theories of Intelligence, Problem-Based Learning, Case-based reasoning, Use of Analogies, Comparison of Examples, Retrieval Practice, and Prompting for Explanations. By building a foundation on current research and making experiments (A/B testing) a ubiquitous feature of everyday instruction, I present a Cognitive Science perspective on how online courses can become the dominant source of education research and innovation.
"How to Use Scientific Research to Improve Khan Academy Exercises, and How to Use Khan Academy Exercises to Improve Scientific Research." (Khan Academy)Title: How to use scientific research to improve Khan Academy exercises, and how to use Khan Academy exercises to improve scientific research. Abstract: How should we design Khan Academy exercises to produce excellent student learning? How should problems be presented and solutions be communicated, and can we make them more interactive? Cognitive Science research has three insights to answer these and many other practical questions. (1) Avoid reinventing the wheel – review & apply the scientific insights from thousands of previous studies. I will present on Using Cognitive Science Research on Learning to Improve Education. (2) Solve practical problems like a scientist – conduct experiments to evaluate learning whenever possible. I will discuss proposed experiments. (3) "Good designers copy, great designers steal". I report on my exploration of numerous examples of different approaches to the "art" of developing exercises, mining them for insights to synthesize and combine. In turn, engaging in this attempt to use science to improve the practical problem of learning exercises can improve science: By evaluating theories in a real-world environment, disseminating scientific insights in explicit "product form" as thoughtfully crafted exercises, and by unifying knowledge across topics and disciplines around a shared practical problem.
"Using Cognitive Science to Improve E-learning" (Bloomsburg Corporate Advisory Council Talk)"How can Cognitive Science improve Online Learning at Google and Google in Education?" (Google Tech Talks)Presentation on Slideshare Title: How can Cognitive Science improve Online Learning at Google and Google in Education? Abstract: Knowledge and technology that maximizes human learning has financial value for Google in customer education and internal training, as well as social value for the public initiatives of Google in Education. Recent research in Cognitive Science provides complementary insights to those gained from practical experience and the research in Computer Science, Education and other Learning Sciences. This talk considers how learning can be improved by: (1) Asking questions and requesting explanations; (2) Presenting specific examples to illustrate abstract principles; (3) Using tests as pedagogical rather than assessment tools. Moreover, online education provides the unique opportunity of hybrid research that is simultaneously applied and academic. Online environments satisfy the scientific requirements of randomized experiments and precise control, as well as the practical need for ecological validity, fidelity, and scalable dissemination. The Cognitive Science focus on identifying both similarities and differences across learning contexts positions it well for doing research that simultaneously advances public education and a corporate mission. In addition to presenting ongoing research at Khan Academy and MOOCs like EdX, I discuss how analogous principles can be explored in teaching end-users Google Power Search, internal training, and customer education. Bio: Joseph Jay Williams does Cognitive Science research on how generating explanations promotes learning, and Online Education work on improving learning from mathematics exercises (Khan Academy), increasing motivation to learn by changing people's beliefs about intelligence (Project for Education Research that Scales), teaching metacognitive & learning strategies in Massive Open Online Courses (EdX), and using technology to change educational and health habits. He is finishing his PhD in Psychology at UC Berkeley, and also has interests in consulting for corporate e-learning and training, web development for online education, using journalism to disseminate research to practitioners, and education in online search and problem-solving for students and entrepreneurs.
"Using Research to Maximize the Practical Impact of Online Educational Resources" (UC Berkeley Graduate School of Education)
Title: Using research to maximize the practical impact of online educational resources
Abstract: From K-12 students to adults in the workforce, people are increasingly learning from online educational resources like videos, lessons, and interactive exercises. This has made the benefits and costs of online learning a source of lively debate. This talk presents a framework for creating and improving online educational resources that: (1) Complements the real-world instructional needs of teachers and students by providing instantly accessible and constantly improving online lessons and exercises; (2) Uses technology/software that allows rapid authoring & revision of resources by instructors and researchers; (3) Supports "in vivo" experiments that compare different instructional methods; (4) Collects practical and scientifically validated measures of learning; (5) Is guided by theory and scientific evidence from the cognitive and learning sciences. This framework is illustrated using ongoing research that: (a) Boosts community college students' grades through brief online lessons that teach them that their intelligence is malleable, (b) Uses laboratory experiments to iteratively develop online lessons to teach general learning strategies to K-12 and undergraduate students; (c) Promotes learning on KhanAcademy.org by prompting students to generate explanations and answer conceptual questions about the mathematics exercises they are solving.
Feel free to pose questions/comments before and after the talk using the form at tiny.cc/questionscomments.
Bio: Joseph Jay Williams does research on learning from – and the development of – online educational resources. This includes online lessons that teach K-12 and undergraduate students about the malleability of intelligence (to increase motivation), technology for changing educational habits such as learning strategies, and augmenting online mathematics exercises by adding questions that foster understanding, support adaptive learning, and increase interactivity. He is a founding member of the Learning, Education And Research Network (LEARN) at Berkeley, part of the Project for Education Research that Scales (PERTS) at Stanford, has practical experience teaching statistics, and is receiving his PhD from Berkeley’s Psychology Department. His PhD research examined the role of explanation in learning by using experiments and computational modeling, synthesized and applied cognitive science and education research to promoting learning of cognitive-behavioral therapy, and examined technology for changing educational and health habits. He also writes and gives talks about evidence-based best practices for teaching and learning, provides evaluations of educational technology products & authoring tools for e-learning, and has served as a consultant for educational technology startups as well as a science and technology mentor for Steve Blank’s entrepreneurship class in the Business School.
"Supporting Instructors in MOOCS: Using Cognitive Science Research to Guide Pedagogy and Instructional Design" (EdX/MIT (Instructor Support))Title: Supporting instructors in MOOCS: using cognitive science research to guide pedagogy and instructional design
Abstract: How can online learning platforms provide useful information about pedagogy to instructors teaching online, while ensuring that course teams are not constrained in leveraging their teaching expertise to personalize their MOOC? The scientific literature on learning and education provides hundreds of detailed studies, which can be synthesized to identify effective instructional strategies, and mined for examples of how an instructional strategy can be implemented in a specific environment, set of educational materials, or student population. This talk illustrates this approach, by presenting a worksheet guide that supports MOOC designers in using two instructional strategies: increasing student motivation to think through challenges by designing exercises which encourage students to see their intelligence as malleable, and enhancing deep understanding with questions and prompts for students to explain. The talk explains how these two instructional strategies are motivated by both existing literature and recently conducted experimental studies. It also presents the specific details of how the guide is targeted at MOOC instructors and provides them with multiple actionable strategies they can use in their courses.
"How Can Online and Blended Education Be Used to Close the Loop Between Research and Practice?" (Stanford Lytics Lab)Abstract: The development of online and blended learning offers a variety of new opportunities. This talk considers one approach to using online educational resources to close the loop between research on learning and the practical delivery of online & blended education, motivated by the interdisciplinary field of cognitive science. Not only are online educational resources (e.g. videos, mathematics exercises) ecologically valid, scalable, and delivered instantaneously, but they also provide a laboratory environment that supports randomized assignment to instructional conditions, experimental control, and automatic collection of dependent measures. I discuss how my research – on how generating explanations promotes learning – attempts to close the research-practice loop by going back and forth between: reviewing primary literature, synthesizing practical implications, conducting basic experimental psychology research, online studies that use educational materials with convenience samples from Mechanical Turk, and embedded "in vivo" experiments to improve learning from Khan Academy's mathematics exercises.
I also describe the collaborative approach that has worked well in this context: involving researchers (from psychology, education, computer science) as well as practitioners (teachers, students, designers), and using the internet to support diverse, multi-site collaboration. I close with a brief overview of the resources I have found especially valuable for connecting research & practice in online & blended education: research literature & reviews in the cognitive and learning sciences, online platforms and ed-tech products that are especially well-suited for research, funding & grant opportunities, and specific ways to facilitate future researcher-practitioner collaborations.
"Using Online Experiments to Enhance Educational Research & Practice" (Pittsburgh Science of Learning Center Summer School, at Carnegie Mellon)"How Online Educational Resources and the Internet Can Facilitate Diverse Research-Practice and Cross-Disciplinary Collaborations" (Panel at AERA)(Invited Panel Presentation at AERA 2014 SIG Computers & Internet Applications in Education)
Presenters: Joseph Jay Williams, Ann Edwards & Maria Mendiburo, Anne Trumbore, Marcia Linn, Hui Soo Chae & Gary Natriello, Piotr Mitros, George Siemens
Title: How Online Educational Resources and the Internet Can Facilitate Diverse Research-Practice and Cross-Disciplinary Collaborations
Abstract: Increasing use of online educational resources and the Internet bring students together around shared resources. Can a similar benefit be found for educators and researchers? This invited panel considers this question through presentations which explain how collaborative work in both research and practice was supported by the affordances of the Internet or the affordances provided by working on online educational resources. As educational materials (like sequences of lessons or interactive exercises) become digital resources on the Internet, the reduction in space and time constraints makes them more widely available for students. Similarly, researchers working to improve and understand online resources can use the Internet to collaborate and consult with other researchers and practitioners who span a diverse range of disciplines, perspectives, geographic locations, and schedules. "Improving Online Educational Resources Using Cognitive Science (and Online Collaborations between Cognitive Scientists and Educators)" (iNACOL Webinar Series)Title: Improving Online Educational Resources using Cognitive Science (and Online Collaborations between Cognitive Scientists and Educators)
Description: This webinar provides practical information on how to use published research findings and make contact with cognitive scientists in order to improve K-12 and university students’ learning from digital online resources, like Khan Academy videos or interactive mathematics exercises. The webinar focuses on how students’ motivation and grades have been increased by helping them believe they can take charge of their learning and become smarter, and how students can be supported in reflective thinking and seeking deep understanding, when questions and prompts for students to explain are inserted in videos and interactive exercises. Links to further reading and implementation guides will be provided, from a curated collection of practical but scientifically based principles (e.g. www.josephjaywilliams.com/education or tiny.cc/improveonlinelearning).
The webinar will also consider how research and practice can be more closely linked by the transition from spoken lectures and pen-and-paper assignments to digital online videos and exercises. Online Educational Resources create a “Real-World Laboratory” where scientists can measure learning outcomes for hundreds of students and do experiments that directly compare the benefits of different versions of online videos and exercises that use different instructional methods. Educators can also advise scientists on doing more practical research, by sharing their understanding and wisdom about the challenges students face in learning from a particular set of videos and exercises. While conversations and collaborations between educators and researchers can be difficult to coordinate, the webinar explains simple systems for sharing and commenting on a set of online resources, and strategies for connecting researchers and educators interested in the same educational challenges. Please feel free to post questions and thoughts at tiny.cc/inacolwebinar before, during and after the webinar.
"How Online Educational Resources Provide Novel Affordances for Conducting Practical Interventions and Doing Psychology Experiments" (Stanford Psychological Interventions in Educational Settings (PIES) Group)Title: How online educational resources provide novel affordances for conducting practical interventions and doing psychology
Context: Stanford Psychological Interventions in Educational Settings (PIES) group
"Doing Online Learning Research With Both Scientific and Financial Value" (Stanford Lytics Lab)Title: Doing online learning research with both scientific and financial value
Context: Stanford Lytics Lab
"How to Use Online Resources to Facilitate Collaboration Across Disciplines" (National Science Foundation Seventh Annual Inter-Science of Learning Centers Conference)Title: How to use Online Resources to Facilitate Collaboration across Disciplines
Presenters: Joseph Jay Williams and Cristina Zepeda
Context: Workshop presented at iSLC 2014
"Embedding Experiments in Online Educational Resources to Understand and Improve Learning" (Arizona State University)Title: Embedding Experiments in Online Educational Resources to Understand and Improve Learning
Context: Invited talk at Arizona State University
Abstract: A series of experiments is presented which leverages digital online educational resources (like mathematics exercises on Khan Academy or videos in MOOCs/Massive Open Online Courses) to build from lab-based psychological research on learning and motivation to randomized experiments embedded in real-world educational products. This work comprises the complementary threads of (1) understanding how cognitive processing (engaged by generating explanations) underlies learning (through generating explanations), and (2) understanding how beliefs (about the malleability of intelligence) influence motivation.
The first line of research investigates how people learn through generating explanations (concerning domains from artificial categories to statistics), providing evidence for the novel “Subsumptive Constraints” account of why explaining “why?” helps learning – by driving a search for underlying patterns and principles. Two ongoing experiments embedded in Khan Academy math exercises and a biology MOOC extend this work to prompting people to generate explanations while studying worked examples and videos.
The second line of research finds that students’ motivation is increased through the minimal addition of messages about the malleability of intelligence – they attempt and solve more math problems – although there is no such effect of encouraging messages that do not emphasize intelligence’s malleability.
Since digital educational resources bring students’ authentic learning into a medium with the affordances of a laboratory – randomized assignment, experimental control, and data collection – they provide a novel opportunity to bridge research and practice across multiple disciplines.
"Scientifically Guided Benchmarking: Using Research to Identify Practical Improvements to Modular Educational Resources" (Coursera)Title: Scientifically Guided Benchmarking: Using Research to Identify Practical Improvements to Modular Educational Resources
Context: Invited talk at Coursera
Information: tiny.cc/courseratalk
Abstract: How can existing literature and research methodologies be used practically to support deciding between the many options faced by instructors and engineers? This talk presents one approach, in which an interdisciplinary scientific knowledge base about learning is combined with benchmarking or adapting best practices from examples of educational technology. This involves a dual focus on selectively synthesizing existing research and combining insights from multiple practical technologies, which can support the identification and implementation of the high quality pedagogical principles most appropriate for a target context/platform.
This approach is illustrated with randomized experiments or A/B tests in Khan Academy’s mathematics exercises that use messages to increase motivation and teach learning & problem-solving strategies, in-progress studies to boost motivation and increase learning from MOOC videos, and examples of interactive online tools to help people practice strategies for learning in MOOCs, and apply concepts from a management workshop to everyday interactions. Based on this work, ideas about potential directions for MOOCs & Online Learning with practical, financial, and scientific value are presented for discussion and feedback: The value of focusing offerings, development & research on modular online educational resources or “MOOClets” that are at a smaller grain size than full courses (e.g. lessons, videos, assignments); facilitating iterative revision & improvement of content & interactive exercises; supporting rapid collaborative feedback on online resources; and providing technology support for students to learn generalizable skills along with content – like strategies for learning online, solving problems, and effective interpersonal collaboration and management. Background information for the presentation and these ideas that might be relevant to people at Coursera is available at tiny.cc/courseratalk. "Experiment-Focused Instructional Design: Using A/B Testing in Blended Learning Resources as Both a Conceptual and Empirical Tool" (Stanford Instructional Design Special Interest Group)Title: Experiment-Focused Instructional Design: Using A/B Testing in Blended Learning Resources as both a Conceptual and Empirical Tool
Abstract: A powerful feature of educational resources that are digital and cloud-based is that instructional designers can iteratively improve them, using the A/B testing affordances of software to explore how students' learning can be helped by changes to a resource. This talk considers the value of Experiment-Focused Design as a strategy for bringing the instructional and pedagogical expertise of instructors and designers to bear in specific contexts. Experiment-Focused Design refers to conceptually and/or empirically evaluating design decisions (like how to create text and video lessons, homework assignments, exercises for peer collaboration) in terms of how randomly assigning students to alternative versions of a resource would impact particular measures of student learning.
The presentation explains how designers can easily use software like Qualtrics to do qualitative and quantitative experimentation, in settings from small in-class groups to MOOCs to convenience samples available on Amazon Mechanical Turk. Examples available for illustration include: changing text-based lessons, interactive online mathematics exercises (e.g. Khan Academy), designing activities to structure peer discussion, digital tools to support study strategies, problem-solving skills, and application of concepts from a management course to real-life. The examples presented can also be tailored to the interest of Stanford IDs –please provide suggestions or requests at tiny.cc/idinput or to josephjaywilliams@stanford.edu "Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research" (Declara)Title: Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research Context: Invited talk at Declara "Using Modular Design & “MOOClets” to connect Learning to Real-World Tasks and promote Generalizable Skills" (Udacity)Title: Using Modular Design & “MOOClets” to connect Learning to Real-World Tasks and promote Generalizable Skills Context: Invited talk at Udacity "MOOClet-Driven Research & Development: Leveraging Iterative, Collaborative Development of Modules to align Course Production & Research" (HarvardX)Title: MOOClet-Driven Research & Development: Leveraging Iterative, Collaborative Development of Modules to align Course Production & Research Context: Invited talk at HarvardX "Using technology to bridge cognitive research with practical impact on motivation and learning" (CMU)Title: Using technology to bridge cognitive research with practical impact on motivation and learning Context: Invited talk at Carnegie Mellon University "Enhancing Education Research and Practice Using Qualtrics" (Qualtrics)Context: Talk at the 2015 Qualtrics Insight Summit "MOOClets: A Framework for real-time Adaptive Personalization using A/B Experiments" (McGraw Hill Meetup)Title: MOOClets: A Framework for real-time Adaptive Personalization using A/B
Experiments Context: Talk at the McGraw Hill Education Meetup on Predictive Analytics for Education. "The MOOClet Formalism & API: Enabling active machine learning to modify and personalize user-facing software components" (MIT)Title: The MOOClet Formalism & API: Enabling active machine learning to
modify and personalize user-facing software components Context: Talk at Josh Tenenbaum's Computational Cognitive Science Group. |