We may earn an affiliate commission when you visit our partners.
Course image
Scott Klemmer and Jacob O. Wobbrock

You may never be sure whether you have an effective user experience until you have tested it with users. In this course, you’ll learn how to design user-centered experiments, how to run such experiments, and how to analyze data from these experiments in order to evaluate and validate user experiences. You will work through real-world examples of experiments from the fields of UX, IxD, and HCI, understanding issues in experiment design and analysis. You will analyze multiple data sets using recipes given to you in the R statistical programming language -- no prior programming experience is assumed or required, but you will be required to read, understand, and modify code snippets provided to you. By the end of the course, you will be able to knowledgeably design, run, and analyze your own experiments that give statistical weight to your designs.

Enroll now

What's inside

Syllabus

Basic Experiment Design Concepts
In this module, you will learn basic concepts relevant to the design and analysis of experiments, including mean comparisons, variance, statistical significance, practical significance, sampling, inclusion and exclusion criteria, and informed consent. You’ll also learn to think of an experiment in terms of usability, its participants, apparatus, procedure, and design & analysis. This module covers lecture videos 1-2.
Read more
Tests of Proportions
In this module, you will learn how to analyze user preferences (or other tallies) using tests of proportions. You will also get up and running with R and RStudio. Topics covered include independent and dependent variables, variable types, exploratory data analysis, p-values, asymptotic tests, exact tests, one-sample tests, two-sample tests, Chi-Square test, G-test, Fisher’s exact test, binomial test, multinomial test, post hoc tests, and pairwise comparisons. This module covers lecture videos 3-9.
The T-Test
In this module, you will learn how to design and analyze a simple website A/B test. Topics include measurement error, independent variables as factors, factor levels, between-subjects factors, within-subjects factors, dependent variables as responses, response types, balanced designs, and how to report a t-test. You will perform your first analysis of variance in the form of an independent-samples t-test. This module covers lecture videos 10-11.
Validity in Design and Analysis
In this module, you will learn about how to ensure that your data is valid through the design of experiments, and that your analyses are valid by understanding and testing for certain assumptions. Topics include how to achieve experimental control, confounds, ecological validity, the three assumptions of ANOVA, data distributions, residuals, normality, homoscedasticity, parametric versus nonparametric tests, the Shapiro-Wilk test, the Kolmogorov-Smirnov test, Levene’s test, the Brown-Forsythe test, and the Mann-Whitney U test. This module covers lecture videos 12-15.
One-Factor Between-Subjects Experiments
In this module, you will learn about one-factor between-subjects experiments. The experiment examined will be a between-subjects study of task completion time with various programming tools. You will understand and analyze data from two-level factors and three-level factors using the independent-samples t-test, Mann-Whitney U test, one-way ANOVA, and Kruskal-Wallis test. You will learn how to report an F-test. You will also understand omnibus tests and how they relate to post hoc pairwise comparisons with adjustments for multiple comparisons. This module covers lecture videos 16-18.
One-Factor Within-Subjects Experiments
In this module, you will learn about one-factor within-subjects experiments, also known as repeated measures designs. The experiment examined will be a within-subjects study of subjects searching for contacts in a smartphone contacts manager, including the analysis of times, errors, and effort Likert-type scale ratings. You will learn counterbalancing strategies to avoid carryover effects, including full counterbalancing, Latin Squares, and balanced Latin Squares. You will understand and analyze data from two-level factors and three-level factors using the paired-samples t-test, Wilcoxon signed-rank test, one-way repeated measures ANOVA, and Friedman test. This module covers lecture videos 19-23.
Factorial Experiment Designs
In this module, you will learn about experiments with multiple factors and factorial ANOVAs. The experiment examined will be text entry performance on different smartphone keyboards while sitting, standing, and walking. Topics include mixed factorial designs, interaction effects, factorial ANOVAs, and the Aligned Rank Transform as a nonparametric factorial ANOVA. This module covers lecture videos 24-27.
Generalizing the Response
In this module, you will learn about analyses for non-normal or non-numeric responses for between-subjects experiments using Generalized Linear Models (GLM). We will revisit three previous experiments and analyze them using generalized models. Topics include a review of response distributions, nominal logistic regression, ordinal logistic regression, and Poisson regression. This module covers lecture videos 28-29.
The Power of Mixed Effects Models
In this module, you will learn about mixed effects models, specifically Linear Mixed Models (LMM) and Generalized Linear Mixed Models (GLMM). We will revisit our prior experiment on text entry performance on smartphones but this time, keeping every single measurement trial as part of the analysis. The full set of analyses covered in this course will also be reviewed. This module covers lecture videos 30-33.

Good to know

Know what's good
, what to watch for
, and possible dealbreakers
Presents foundational concepts in experimental design and analysis, providing a strong grounding for beginners
Applies R programming language for statistical analysis, equipping learners with practical skills in data analysis
Covers a wide range of statistical methods, from basic to advanced, deepening learners' understanding of user experience evaluation
Provides real-world examples and case studies, demonstrating the practical application of user experience testing methods
Taught by Scott Klemmer and Jacob O. Wobbrock, renowned researchers and educators in the field of human-computer interaction
Includes interactive exercises and assignments to reinforce learning and provide hands-on experience

Save this course

Save Designing, Running, and Analyzing Experiments to your list so you can find it easily later:
Save

Reviews summary

Experimental design and statistical analysis

Learners say this advanced course in designing, running, and analyzing experiments is well structured and provides a solid foundation in statistical techniques. However, some students have expressed frustration due to its high difficulty level, narrow focus on R coding, and lack of real-world application. Despite these concerns, the course has received positive feedback for its clear explanations and practical examples. Before enrolling, learners should consider their background in statistics and R programming and weigh the potential benefits against the challenges.
Practical Scenarios
"I never leave reviews for anything, but I wanted to sincerely thank the instructors for the work put into teaching this course."
"I've taken a few stats courses many years ago. I wanted a good overview of how to do a range of statistical tests within R and an introduction to GLM, LMM and GLMM."
"The sample code provided by the instructor is an EXTREMELY valuable starting point."
Clear Concepts
"The explanations are super clear and concise and I'm amazed at how well some concepts are explained, as compared to some other resources."
"This course has made it friendly for me to onboard and tackle the problems."
"I've read many negative reviews before starting the course and was very nervous at first. But even without any programming background, it was possible to follow the instructions and complete the coding tasks for me."
Well-Organized
"It is rather intense (which means - very informative) course and I enjoyed it very much."
"This course is well paced, the instructor very clear, and the provided files very helpful when taking the assessments."
Outdated Packages
"Seeing all the negative reviews of this unit really discouraged me, but now, having completed it with full marks with zero prior experience in stastistical analysis, I think the hate is unwarranted."
"Many of the packages were outdated and I saw numerous comments within the forum about students unable to install or run the packages required for the assessments."
"The course is out of date with regards to the software needed to complete it."
Condescending Tone
"If a majority of students in this course are struggling, that is an issue with the professor, the content, the way the content is being delivered, or all three."
"It is appalling to me that he repeatedly mentions that the "code is right, the problem must be you" within several of his scant responses on the forum."
R-Studio Focus
"It focus more on proceeding technically statistical tests than explaining why we do them and how they work."
"It's extremely complicated, assumes a lot of previous knowledge, and feels hardly applicable or related to the previous courses."
"This course is more like 'running data analysis tasks on Rstudio' and that's about it."
"This course is targeted towards a very narrow audience who are already highly familiar with R and R coding as well as related stats."
Advanced
"This course is part of the Interaction Design Specialization and if you take that into consideration this course was to advance for the most of the designers taking the specialization."
"This course is targeted towards a very narrow audience who are already highly familiar with R and R coding as well as related stats."
"This course is NOT appropriate for the Interaction Design Specialization. It was far too detailed, complex, in depth and advanced!"
"This was a challenging course for a beginner/novice"

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Designing, Running, and Analyzing Experiments with these activities:
Review statistics concepts
Refresh your knowledge of statistics concepts such as mean, variance, and hypothesis testing.
Browse courses on Statistics
Show steps
  • Review your notes from a previous statistics course
  • Take a practice quiz on statistics concepts
  • Watch a video tutorial on hypothesis testing
Learn about R programming
Complete tutorials on R programming to gain the skills necessary for data analysis.
Browse courses on R Programming
Show steps
  • Find a beginner-friendly R programming tutorial
  • Follow the tutorial and complete the exercises
  • Create a small R program of your own
Discuss user experience design principles with peers
Engage in discussions with peers to exchange ideas and reinforce your understanding of user experience design principles.
Browse courses on User Experience Design
Show steps
  • Find a study group or online forum
  • Participate in discussions and share your thoughts
  • Listen to others' perspectives and learn from their experiences
Five other activities
Expand to see all activities and additional details
Show all eight activities
Calculate t-tests
Practice calculating t-tests to solidify your understanding of hypothesis testing.
Browse courses on T-Tests
Show steps
  • Review the formula for a t-test
  • Find a dataset with two groups
  • Calculate the t-statistic
  • Find the p-value
  • Interpret the results
Design a user experience experiment
Create a user experience experiment design to apply the concepts you've learned in the course.
Browse courses on Experiment Design
Show steps
  • Define your research question
  • Choose your participants
  • Design your experiment
  • Collect your data
  • Analyze your results
Read 'Designing User Interfaces' by Bill Moggridge
Expand your knowledge of user experience design by reading a classic book in the field.
View Melania on Amazon
Show steps
  • Read the book
  • Take notes on the key concepts
  • Apply the concepts to your own work
Develop a user experience prototype
Create a user experience prototype to demonstrate your mastery of the design process and showcase your skills.
Browse courses on Design
Show steps
  • Brainstorm ideas for your prototype
  • Sketch out your design
  • Build a functional prototype
  • Test your prototype with users
  • Refine your design based on feedback
Contribute to an open-source user experience project
Engage in open-source collaboration to contribute to real-world user experience projects.
Browse courses on Open Source
Show steps
  • Find an open-source user experience project
  • Review the project's documentation
  • Identify a way to contribute
  • Submit a pull request with your changes
  • Collaborate with other contributors

Career center

Learners who complete Designing, Running, and Analyzing Experiments will develop knowledge and skills that may be useful to these careers:
UX Designer
As a UX Designer, you will be responsible for designing and evaluating user experiences. This course will provide you with the skills you need to create user-friendly and effective user interfaces. You will learn how to conduct user research, design prototypes, and test your designs with users. This course will also help you to develop the analytical skills you need to evaluate user data and make informed design decisions.
Product Manager
As a Product Manager, you will be responsible for managing the development and launch of new products. This course will provide you with the skills you need to define product requirements, create product roadmaps, and track product progress. You will also learn how to conduct user research and analyze user data to make informed product decisions.
Data Analyst
As a Data Analyst, you will be responsible for collecting, analyzing, and interpreting data. This course will provide you with the skills you need to clean and prepare data, conduct statistical analysis, and create data visualizations. You will also learn how to communicate your findings to stakeholders in a clear and concise way.
Business Analyst
As a Business Analyst, you will be responsible for analyzing business processes and identifying opportunities for improvement. This course will provide you with the skills you need to gather requirements, create process maps, and develop solutions to business problems. You will also learn how to communicate your findings to stakeholders in a clear and concise way.
User Researcher
As a User Researcher, you will be responsible for conducting user research and providing insights to design teams. This course will provide you with the skills you need to design and conduct user studies, analyze user data, and create user personas. You will also learn how to communicate your findings to stakeholders in a clear and concise way.
Software Engineer
As a Software Engineer, you will be responsible for designing, developing, and testing software applications. This course will provide you with the skills you need to write clean and efficient code, design user interfaces, and debug software defects. You will also learn how to work in a team environment and use version control systems.
Marketing Manager
As a Marketing Manager, you will be responsible for developing and executing marketing campaigns. This course will provide you with the skills you need to create marketing plans, manage budgets, and track campaign performance. You will also learn how to use social media, email marketing, and other marketing channels to reach your target audience.
Sales Manager
As a Sales Manager, you will be responsible for managing a sales team and achieving sales goals. This course will provide you with the skills you need to develop sales strategies, motivate your team, and close deals. You will also learn how to use sales tools and techniques to maximize your sales potential.
Project Manager
As a Project Manager, you will be responsible for planning, executing, and closing projects. This course will provide you with the skills you need to develop project plans, manage budgets, and track project progress. You will also learn how to communicate with stakeholders and resolve project issues.
Operations Manager
As an Operations Manager, you will be responsible for managing the day-to-day operations of a business. This course will provide you with the skills you need to develop and implement operational plans, manage staff, and ensure that the business runs smoothly. You will also learn how to use data to improve operational efficiency.
Quality Assurance Analyst
As a Quality Assurance Analyst, you will be responsible for testing software applications and ensuring that they meet quality standards. This course will provide you with the skills you need to design and execute test cases, analyze test results, and report defects. You will also learn how to use quality assurance tools and techniques to improve the quality of software applications.
Information Architect
As an Information Architect, you will be responsible for designing and organizing the content of websites and other digital products. This course will provide you with the skills you need to understand user needs, create sitemaps, and design navigation systems. You will also learn how to use information architecture tools and techniques to improve the usability of digital products.
Interaction Designer
As an Interaction Designer, you will be responsible for designing the user experience of websites and other digital products. This course will provide you with the skills you need to understand user needs, create prototypes, and design user interfaces. You will also learn how to use interaction design tools and techniques to improve the usability of digital products.
Visual Designer
As a Visual Designer, you will be responsible for creating the visual design of websites and other digital products. This course will provide you with the skills you need to understand visual design principles, create mockups, and design user interfaces. You will also learn how to use visual design tools and techniques to improve the usability of digital products.
Technical Writer
As a Technical Writer, you will be responsible for creating documentation for software products and other technical topics. This course will provide you with the skills you need to understand technical concepts, write clear and concise documentation, and use documentation tools and techniques.

Reading list

We've selected 12 books that we think will supplement your learning. Use these to develop background knowledge, enrich your coursework, and gain a deeper understanding of the topics covered in Designing, Running, and Analyzing Experiments.
Provides a comprehensive overview of statistical learning methods. It covers topics such as linear regression, logistic regression, and decision trees. It could provide additional depth and breadth on statistical learning methods beyond what is covered in the course.
Provides a comprehensive overview of modern experimental design techniques. It covers topics such as factorial designs, response surface designs, and mixture designs. It could provide additional depth and breadth on experimental design techniques beyond what is covered in the course.
Provides a comprehensive overview of the design of everyday things. It covers topics such as user research, interaction design, and usability testing. It could provide additional practical insights and examples for the course.
Provides a comprehensive overview of the design of human-centered products and services. It covers topics such as user research, interaction design, and usability testing. It could provide additional practical insights and examples for the course.
Provides a comprehensive overview of usability engineering principles. It covers topics such as user research, interaction design, and usability testing. It could provide additional practical insights and examples for the course.
Provides a practical guide to web usability. It covers topics such as user research, interaction design, and usability testing. It could provide additional practical insights and examples for the course.
Provides a comprehensive overview of the design and conduct of clinical research. It covers topics such as study design, data collection, and data analysis. It could provide additional depth and breadth on the design and conduct of clinical research beyond what is covered in the course.
Provides a comprehensive overview of statistical power analysis for the behavioral sciences. It covers topics such as effect size, sample size determination, and power analysis. It could provide additional depth and breadth on statistical power analysis beyond what is covered in the course.
Provides a comprehensive overview of experimental and quasi-experimental designs for research. It covers topics such as true experimental designs, quasi-experimental designs, and mixed methods designs. It could provide additional background and a different perspective on the experimental design concepts covered in the course.
Provides a conceptual framework for understanding statistical models and their relationships to experimental design and analysis. It provides a comprehensive overview of experimental design, data analysis, and model comparison. It could provide additional background and a different perspective on the concepts covered in the course.
Provides a comprehensive overview of statistical methods used in psychology. It covers topics such as descriptive statistics, inferential statistics, and multivariate analysis. It could provide additional background and a different perspective on the statistical concepts covered in the course.

Share

Help others find this course page by sharing it with your friends and followers:

Similar courses

Here are nine courses similar to Designing, Running, and Analyzing Experiments.
Experimental Design and Recommendations
Most relevant
Experimental Design Basics
Most relevant
Practical Predictive Analytics: Models and Methods
Most relevant
Advanced Statistical Analysis and Tools
Most relevant
UX Research at Scale: Surveys, Analytics, Online Testing
Manufacturing Process Control II
Six Sigma Part 2: Analyze, Improve, Control
Hands-On Chaos Engineering with AWS Fault Injection...
UX User Testing
Our mission

OpenCourser helps millions of learners each year. People visit us to learn workspace skills, ace their exams, and nurture their curiosity.

Our extensive catalog contains over 50,000 courses and twice as many books. Browse by search, by topic, or even by career interests. We'll match you to the right resources quickly.

Find this site helpful? Tell a friend about us.

Affiliate disclosure

We're supported by our community of learners. When you purchase or subscribe to courses and programs or purchase books, we may earn a commission from our partners.

Your purchases help us maintain our catalog and keep our servers humming without ads.

Thank you for supporting OpenCourser.

© 2016 - 2024 OpenCourser