ISWS 2024 - User-centered Evaluation QUIZ ONLY

Warm-up question: in which Italian region are we now?

• Tuscany

• Emilia-Romagna

• Sicily

• Algarve

Who's your best friend for user evaluation analysis?

• An anthropologist

• A statistician

• A philosopher

• A rich sponsor

How many users do you need to involve in a user evaluation study?

• 30

• 300

• 3,000

• It depends

Ok, the number of required evaluators depends... on what?

• The number of people I know

• The complexity of the evaluation task

• The size of the target user population

• The money I can invest to pay evaluators

Now, say you need around 100 evaluators. Therefore you involve...

• 100 friends from the target population

• 100 representative users from the target population

• The first 100 people available

• Ignore statistics, only the people I can get

You failed to involve enough users in your evaluation. Now what?

• You cannot include your evaluation in a paper

• You have to keep going, even for months, until you reach the right number

• You are doomed, better give up your PhD

• You can still derive some insights from the data you collected

And how many users do you involve in a qualitative evaluation study?

• Exactly the same as quantitative evaluation

• 3 evaluators per each "category" of users in the target population

• At least 10% w.r.t. quantitative evaluation

• Nah, 4 or 5 users in total are more than enough

What do you do in a "thinking out loud" method?

• You clearly express your opinion on why the evaluated system is beautiful

• You let the evaluators scream to release tension before the evaluation

• You ask the evaluators to describe what they do and think during evaluation

• You shout out to evaluators to hurry up

What's best: qualitative or quantitative user evaluation?

• Always quantitative: numbers don't lie

• Always qualitative: humanity is not explained by numbers

• Both qualitative and quantitative would be the best!

• Neither of them: I hate people

When can you run an evaluation campaign, say of a software tool?

• A drawing on a napkin could be enough to evaluate

• Only when the user interface design is finished

• Only when the main features are implemented

• Only when all bugs are solved