Designing, Running, and Analyzing Experiments from Coursera

What's inside

Syllabus

Basic Experiment Design Concepts

In this module, you will learn basic concepts relevant to the design and analysis of experiments, including mean comparisons, variance, statistical significance, practical significance, sampling, inclusion and exclusion criteria, and informed consent. You’ll also learn to think of an experiment in terms of usability, its participants, apparatus, procedure, and design & analysis. This module covers lecture videos 1-2.

Tests of Proportions

In this module, you will learn how to analyze user preferences (or other tallies) using tests of proportions. You will also get up and running with R and RStudio. Topics covered include independent and dependent variables, variable types, exploratory data analysis, p-values, asymptotic tests, exact tests, one-sample tests, two-sample tests, Chi-Square test, G-test, Fisher’s exact test, binomial test, multinomial test, post hoc tests, and pairwise comparisons. This module covers lecture videos 3-9.

The T-Test

In this module, you will learn how to design and analyze a simple website A/B test. Topics include measurement error, independent variables as factors, factor levels, between-subjects factors, within-subjects factors, dependent variables as responses, response types, balanced designs, and how to report a t-test. You will perform your first analysis of variance in the form of an independent-samples t-test. This module covers lecture videos 10-11.

Validity in Design and Analysis

In this module, you will learn about how to ensure that your data is valid through the design of experiments, and that your analyses are valid by understanding and testing for certain assumptions. Topics include how to achieve experimental control, confounds, ecological validity, the three assumptions of ANOVA, data distributions, residuals, normality, homoscedasticity, parametric versus nonparametric tests, the Shapiro-Wilk test, the Kolmogorov-Smirnov test, Levene’s test, the Brown-Forsythe test, and the Mann-Whitney U test. This module covers lecture videos 12-15.

One-Factor Between-Subjects Experiments

In this module, you will learn about one-factor between-subjects experiments. The experiment examined will be a between-subjects study of task completion time with various programming tools. You will understand and analyze data from two-level factors and three-level factors using the independent-samples t-test, Mann-Whitney U test, one-way ANOVA, and Kruskal-Wallis test. You will learn how to report an F-test. You will also understand omnibus tests and how they relate to post hoc pairwise comparisons with adjustments for multiple comparisons. This module covers lecture videos 16-18.

One-Factor Within-Subjects Experiments

In this module, you will learn about one-factor within-subjects experiments, also known as repeated measures designs. The experiment examined will be a within-subjects study of subjects searching for contacts in a smartphone contacts manager, including the analysis of times, errors, and effort Likert-type scale ratings. You will learn counterbalancing strategies to avoid carryover effects, including full counterbalancing, Latin Squares, and balanced Latin Squares. You will understand and analyze data from two-level factors and three-level factors using the paired-samples t-test, Wilcoxon signed-rank test, one-way repeated measures ANOVA, and Friedman test. This module covers lecture videos 19-23.

Factorial Experiment Designs

In this module, you will learn about experiments with multiple factors and factorial ANOVAs. The experiment examined will be text entry performance on different smartphone keyboards while sitting, standing, and walking. Topics include mixed factorial designs, interaction effects, factorial ANOVAs, and the Aligned Rank Transform as a nonparametric factorial ANOVA. This module covers lecture videos 24-27.

Generalizing the Response

In this module, you will learn about analyses for non-normal or non-numeric responses for between-subjects experiments using Generalized Linear Models (GLM). We will revisit three previous experiments and analyze them using generalized models. Topics include a review of response distributions, nominal logistic regression, ordinal logistic regression, and Poisson regression. This module covers lecture videos 28-29.

The Power of Mixed Effects Models

In this module, you will learn about mixed effects models, specifically Linear Mixed Models (LMM) and Generalized Linear Mixed Models (GLMM). We will revisit our prior experiment on text entry performance on smartphones but this time, keeping every single measurement trial as part of the analysis. The full set of analyses covered in this course will also be reviewed. This module covers lecture videos 30-33.

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Presents foundational concepts in experimental design and analysis, providing a strong grounding for beginners

Applies R programming language for statistical analysis, equipping learners with practical skills in data analysis

Covers a wide range of statistical methods, from basic to advanced, deepening learners' understanding of user experience evaluation

Provides real-world examples and case studies, demonstrating the practical application of user experience testing methods

Taught by Scott Klemmer and Jacob O. Wobbrock, renowned researchers and educators in the field of human-computer interaction

Includes interactive exercises and assignments to reinforce learning and provide hands-on experience

Reviews summary

Experimental design and statistical analysis

Learners say this advanced course in designing, running, and analyzing experiments is well structured and provides a solid foundation in statistical techniques. However, some students have expressed frustration due to its high difficulty level, narrow focus on R coding, and lack of real-world application. Despite these concerns, the course has received positive feedback for its clear explanations and practical examples. Before enrolling, learners should consider their background in statistics and R programming and weigh the potential benefits against the challenges.

Practical Scenarios

"I never leave reviews for anything, but I wanted to sincerely thank the instructors for the work put into teaching this course."

"I've taken a few stats courses many years ago. I wanted a good overview of how to do a range of statistical tests within R and an introduction to GLM, LMM and GLMM."

"The sample code provided by the instructor is an EXTREMELY valuable starting point."

Clear Concepts

"The explanations are super clear and concise and I'm amazed at how well some concepts are explained, as compared to some other resources."

"This course has made it friendly for me to onboard and tackle the problems."

"I've read many negative reviews before starting the course and was very nervous at first. But even without any programming background, it was possible to follow the instructions and complete the coding tasks for me."

Well-Organized

"It is rather intense (which means - very informative) course and I enjoyed it very much."

"This course is well paced, the instructor very clear, and the provided files very helpful when taking the assessments."

Outdated Packages

"Seeing all the negative reviews of this unit really discouraged me, but now, having completed it with full marks with zero prior experience in stastistical analysis, I think the hate is unwarranted."

"Many of the packages were outdated and I saw numerous comments within the forum about students unable to install or run the packages required for the assessments."

"The course is out of date with regards to the software needed to complete it."

Condescending Tone

"If a majority of students in this course are struggling, that is an issue with the professor, the content, the way the content is being delivered, or all three."

"It is appalling to me that he repeatedly mentions that the "code is right, the problem must be you" within several of his scant responses on the forum."

R-Studio Focus

"It focus more on proceeding technically statistical tests than explaining why we do them and how they work."

"It's extremely complicated, assumes a lot of previous knowledge, and feels hardly applicable or related to the previous courses."

"This course is more like 'running data analysis tasks on Rstudio' and that's about it."

"This course is targeted towards a very narrow audience who are already highly familiar with R and R coding as well as related stats."

Advanced

"This course is part of the Interaction Design Specialization and if you take that into consideration this course was to advance for the most of the designers taking the specialization."

"This course is targeted towards a very narrow audience who are already highly familiar with R and R coding as well as related stats."

"This course is NOT appropriate for the Interaction Design Specialization. It was far too detailed, complex, in depth and advanced!"

"This was a challenging course for a beginner/novice"

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Designing, Running, and Analyzing Experiments with these activities:

Review statistics concepts

Show steps

Refresh your knowledge of statistics concepts such as mean, variance, and hypothesis testing.

Browse courses on Statistics

Show steps

Review your notes from a previous statistics course
Take a practice quiz on statistics concepts
Watch a video tutorial on hypothesis testing

Learn about R programming

Show steps

Complete tutorials on R programming to gain the skills necessary for data analysis.

Browse courses on R Programming

Show steps

Find a beginner-friendly R programming tutorial
Follow the tutorial and complete the exercises
Create a small R program of your own

Discuss user experience design principles with peers

Show steps

Engage in discussions with peers to exchange ideas and reinforce your understanding of user experience design principles.

Browse courses on User Experience Design

Show steps

Find a study group or online forum
Participate in discussions and share your thoughts
Listen to others' perspectives and learn from their experiences

Five other activities

Expand to see all activities and additional details

Show all eight activities

Calculate t-tests

Show steps

Practice calculating t-tests to solidify your understanding of hypothesis testing.

Browse courses on T-Tests

Show steps

Review the formula for a t-test
Find a dataset with two groups
Calculate the t-statistic
Find the p-value
Interpret the results

Design a user experience experiment

Show steps

Create a user experience experiment design to apply the concepts you've learned in the course.

Browse courses on Experiment Design

Show steps

Define your research question
Choose your participants
Design your experiment
Collect your data
Analyze your results

Read 'Designing User Interfaces' by Bill Moggridge

Show steps

Expand your knowledge of user experience design by reading a classic book in the field.

View Melania on Amazon

Show steps

Read the book
Take notes on the key concepts
Apply the concepts to your own work

Develop a user experience prototype

Show steps

Create a user experience prototype to demonstrate your mastery of the design process and showcase your skills.

Browse courses on Design

Show steps

Brainstorm ideas for your prototype
Sketch out your design
Build a functional prototype
Test your prototype with users
Refine your design based on feedback

Contribute to an open-source user experience project

Show steps

Engage in open-source collaboration to contribute to real-world user experience projects.

Browse courses on Open Source

Show steps

Find an open-source user experience project
Review the project's documentation
Identify a way to contribute
Submit a pull request with your changes
Collaborate with other contributors

Career center

Learners who complete Designing, Running, and Analyzing Experiments will develop knowledge and skills that may be useful to these careers:

UX Designer

As a UX Designer, you will be responsible for designing and evaluating user experiences. This course will provide you with the skills you need to create user-friendly and effective user interfaces. You will learn how to conduct user research, design prototypes, and test your designs with users. This course will also help you to develop the analytical skills you need to evaluate user data and make informed design decisions.

See salaries and explore the career path for UX Designer

Product Manager

As a Product Manager, you will be responsible for managing the development and launch of new products. This course will provide you with the skills you need to define product requirements, create product roadmaps, and track product progress. You will also learn how to conduct user research and analyze user data to make informed product decisions.

See salaries and explore the career path for Product Manager

Data Analyst

As a Data Analyst, you will be responsible for collecting, analyzing, and interpreting data. This course will provide you with the skills you need to clean and prepare data, conduct statistical analysis, and create data visualizations. You will also learn how to communicate your findings to stakeholders in a clear and concise way.

See salaries and explore the career path for Data Analyst

Business Analyst

As a Business Analyst, you will be responsible for analyzing business processes and identifying opportunities for improvement. This course will provide you with the skills you need to gather requirements, create process maps, and develop solutions to business problems. You will also learn how to communicate your findings to stakeholders in a clear and concise way.

See salaries and explore the career path for Business Analyst

User Researcher

As a User Researcher, you will be responsible for conducting user research and providing insights to design teams. This course will provide you with the skills you need to design and conduct user studies, analyze user data, and create user personas. You will also learn how to communicate your findings to stakeholders in a clear and concise way.

See salaries and explore the career path for User Researcher

Software Engineer

As a Software Engineer, you will be responsible for designing, developing, and testing software applications. This course will provide you with the skills you need to write clean and efficient code, design user interfaces, and debug software defects. You will also learn how to work in a team environment and use version control systems.

See salaries and explore the career path for Software Engineer

Marketing Manager

As a Marketing Manager, you will be responsible for developing and executing marketing campaigns. This course will provide you with the skills you need to create marketing plans, manage budgets, and track campaign performance. You will also learn how to use social media, email marketing, and other marketing channels to reach your target audience.

See salaries and explore the career path for Marketing Manager

Sales Manager

As a Sales Manager, you will be responsible for managing a sales team and achieving sales goals. This course will provide you with the skills you need to develop sales strategies, motivate your team, and close deals. You will also learn how to use sales tools and techniques to maximize your sales potential.

See salaries and explore the career path for Sales Manager

Project Manager

As a Project Manager, you will be responsible for planning, executing, and closing projects. This course will provide you with the skills you need to develop project plans, manage budgets, and track project progress. You will also learn how to communicate with stakeholders and resolve project issues.

See salaries and explore the career path for Project Manager

Operations Manager

As an Operations Manager, you will be responsible for managing the day-to-day operations of a business. This course will provide you with the skills you need to develop and implement operational plans, manage staff, and ensure that the business runs smoothly. You will also learn how to use data to improve operational efficiency.

See salaries and explore the career path for Operations Manager

Quality Assurance Analyst

As a Quality Assurance Analyst, you will be responsible for testing software applications and ensuring that they meet quality standards. This course will provide you with the skills you need to design and execute test cases, analyze test results, and report defects. You will also learn how to use quality assurance tools and techniques to improve the quality of software applications.

See salaries and explore the career path for Quality Assurance Analyst

Information Architect

As an Information Architect, you will be responsible for designing and organizing the content of websites and other digital products. This course will provide you with the skills you need to understand user needs, create sitemaps, and design navigation systems. You will also learn how to use information architecture tools and techniques to improve the usability of digital products.

See salaries and explore the career path for Information Architect

Interaction Designer

As an Interaction Designer, you will be responsible for designing the user experience of websites and other digital products. This course will provide you with the skills you need to understand user needs, create prototypes, and design user interfaces. You will also learn how to use interaction design tools and techniques to improve the usability of digital products.

See salaries and explore the career path for Interaction Designer

Visual Designer

As a Visual Designer, you will be responsible for creating the visual design of websites and other digital products. This course will provide you with the skills you need to understand visual design principles, create mockups, and design user interfaces. You will also learn how to use visual design tools and techniques to improve the usability of digital products.

See salaries and explore the career path for Visual Designer

Technical Writer

As a Technical Writer, you will be responsible for creating documentation for software products and other technical topics. This course will provide you with the skills you need to understand technical concepts, write clear and concise documentation, and use documentation tools and techniques.

See salaries and explore the career path for Technical Writer