Human-Computer Interaction for CSAI

by Albert Major

Evaluation in HCI

The lecture focuses on the final stage of the UCD cycle: Evaluation. After finding needs and creating prototypes, you must evaluate the design to ensure it works for the user. The core message is that "you are not your user," so you cannot rely on your own intuition; you must gather objective data.

1. The Philosophy of Data and Evaluation

• The Danger of Manipulated Data: The lecture opens with a quote by Nobel Laureate Ronald Coase: "If you torture data enough, it will confess to anything."
  • This means that data can be presented in misleading ways to support a false narrative (often called "bullshit" in research).
  • Example: A graph showed a correlation between music genres and life expectancy (e.g., Rap artists die young, Jazz artists die old). This was a "visual hijack." It flawed logic because it ignored that Rap is a newer genre with naturally younger artists, while Jazz has been around longer. It confuses correlation with causality.
• Why Evaluate?
  • To fix problems early. It is expensive to fix a product after it has been released to the market.
  • To prevent market failure (e.g., the Jibo robot failed despite high funding because it wasn't evaluated properly against market needs).

2. The Evaluation Process

To evaluate a product scientifically, you generally follow these steps:

• Define a Task: What do you want the user to do? (e.g., "Find a book," "Send an email").
• Define Performance Measures: How do you measure success? (e.g., number of clicks, time to complete the task, error rate).
• Recruit Participants: Find people who represent your actual user base.
• Conduct Experiment: Run the test in a controlled or natural setting.
• Analyze Data: Use the results to improve the design or validate the product.

3. Evaluation Settings

• In the Wild: Observing users in their natural environment (e.g., watching athletes use a smartwatch at the gym). This provides realistic behavioral data.
• Lab Studies: Controlled experiments inside a lab. You can manipulate specific variables, but it lacks the "natural" feel.
• Remote Studies: Web-based testing. You send a link, and users complete tasks on their own devices.
• Expert Review: Instead of testing with regular users, you ask experts (e.g., retired pilots reviewing a new cockpit). This is useful for specialized, high-risk domains where general users cannot provide feedback.