Reach × Impact × Confidence ÷ Effort. Four letters, one division, one ranking. The day I discovered RICE, I loved it: finally, a way to prioritize without spending two hours in a meeting arguing about which feature goes first. And I had never questioned it.

Let's imagine two features, same effort. Feature A reaches 1,000 people, with massive impact (3 on the RICE scale) and 50% confidence. Feature B reaches 1,500 people, with medium impact (1) and 100% confidence. A's score: 1,500. B's score: 1,500.

Sorry, what? An uncertain bet that could change everything and a small improvement we're sure of… are worth exactly the same? So we flip a coin? Or we let Excel decide, since it seems to have an opinion?

I didn't find this by reading an article about RICE. I found it while building Métitra, a tool for prioritizing user research. While trying to score risks, I stumbled on the same flaw in methods that are much older, and much more serious. Spoiler: the flaw isn't in the numbers. It's in the operation.

1. FMEA, or the quiet charm of multiplication

Multiplying scores means accepting that serious harm can be bought back with lots of little nothings.

Métitra starts from a simple idea: user research is expensive, so put it where a mistake would be expensive and where evidence is missing. The founding case? On SoCute Imaging, a medtech project on liver MRI, an interview guide asked radiologists how long a liver MRI takes. That figure was already published. Rare, expensive medical time, spent having experts recite what we could have read in a paper. Classy.

To score the risk of each question, the natural reflex is FMEA: failure mode and effects analysis, used in industry and in medical devices. You rate severity, occurrence and detectability, you multiply, you rank. And there's some truth to it: breaking a risk down into factors already beats a wet finger in the wind. Severity and probability are also how risk is defined in ISO 14971, the risk management standard for medical devices.

Except multiplication compensates. A rare, catastrophic error can get the same score as a frequent, harmless one. On paper, a diagnosis based on a wrong measurement weighs as much as a typo fixed every day. The math isn't wrong. It does exactly what we ask it to… and that's precisely the problem. The automotive industry's FMEA handbook (AIAG-VDA, 2019) ended up replacing that score with a table of action priorities, where severity sets the ceiling.

2. Cox, or the day I reopened RICE

A score doesn't lie. It erases.

Digging into the criticism of FMEA, I came across Louis Anthony Cox Jr. and his paper "What's Wrong with Risk Matrices?", published in Risk Analysis in 2008. He took risk matrices seriously, far more seriously than we do, and put them through the wringer, by hand. The verdict: they rank poorly, they flatten the gaps between very different risks, and they rest on subjective judgments that then get presented as a measurement.

An example everyone has run into: accessibility. A checkout button you can't reach with a keyboard or a screen reader. Probability? "Low, it only affects a few users." Green cell, out of the sprint, everyone's happy. Except that for the person using a screen reader, the dead end doesn't happen rarely: it happens every single time. The matrix spreads across all users a harm that one specific group takes in full. This isn't an interface detail. It's an online store that breaks its promise to part of its customers. That's why Métitra accounts for how diverse the users are, and asks, for each question, whether an error would hit the core purpose of the service.

And then… RICE. Same family. Scores on arbitrary scales, multiplied together, then divided by an effort estimate. Confidence, for instance, doesn't measure uncertainty: it dissolves it into a coefficient. A 50% confidence should mean "we don't know, let's go and look." In RICE, it just means "divide by two." Everyone can go to lunch.

I had fun testing it elsewhere, on other cases. And it gets complicated. Not because RICE is dumb: it does what it was designed for, quickly ordering a list of comparable bets. The trouble starts when we ask it to compare bets that look nothing alike, and treat its verdict as a weighing. A framework isn't a measuring instrument. It's a discussion aid passing itself off as a scale.

3. ELECTRE: when criteria vote instead of adding up

Some harms can't be bought back by any gain.

So we looked elsewhere. Toward multi-criteria decision analysis, and especially ELECTRE, Bernard Roy's family of methods (with Denis Bouyssou's overviews). The principle? Criteria don't add up, they vote. And a veto threshold can block a conclusion, whatever the rest says. Exactly what FMEA was missing.

Struggle #1: on a 1-to-5 scale, the veto wasn't enough. There was no way to rate a rare but catastrophic risk as "unacceptable" without blurring every other case. The veto existed… it just voted shyly.

Struggle #2: replayed on the SoCute case, our first version put 5 themes out of 6 at priority 1. A model that says everything is urgent says nothing. It's the fire alarm evacuating the whole building over a burnt slice of toast. Digging in, we understood why: we were confusing how critical a question is with how urgently it needs research. A critical question that's already documented in the literature doesn't need a single hour of a radiologist's time.

Struggle #3: six criteria was too many. Task criticality overlapped with severity, and detectability was stepping on irreversibility's toes. We went down to four.

Hence a two-step calculation. First, a non-compensable opinion: if an error can cause serious, irreparable harm to someone else, nothing makes up for it. A company can choose to risk its own money. It can't trade the harm someone else suffers for its own gain. Only then does ELECTRE TRI rank the rest. And when the cautious and the optimistic calculations disagree, the tool doesn't decide for you: it flags the question "to discuss in a workshop." Finally, priority crosses that risk with the evidence already available. On SoCute, two questions out of seven can be settled without any medical time, and two more partly.

4. Imperfect thresholds, on purpose

An openly provisional threshold beats invented precision.

Next, we had to set thresholds. Two options: invent numbers that look serious, or build on what exists and say clearly where it stops. We took the second.

To break a solution down, we drew on Teresa Torres and Continuous Discovery Habits: desirability, viability, feasibility, usability, ethics. A little calculation that stings: a solution resting on four essential assumptions, each 80% likely to be true, has only about a 41% chance of standing. Intuition would have bet much higher (Tversky and Kahneman showed as much back in 1974).

For the number of interviews, we aligned with Guest, Bunce and Johnson (2006): most themes show up within 6 interviews, and saturation arrives around 12 in a homogeneous group. Hence our thresholds, per segment: fewer than 3, no validity; 3 to 4, weak; 5 to 9, partial; 10 or more, strong. Saturation proves a pattern exists. Not how many people share it. For a quantity, it's the survey and its margin of error that speak: 15 responses out of 500 employees is about ±25 points. You might as well consult a crystal ball (at least it doesn't claim to have a sample).

Everything else is labeled "provisional": the weights, the veto thresholds, the ELECTRE TRI profiles. They'll be put to the test with real testers. Because a tool that sells rigor while hiding its approximations would be doing exactly what it claims to fix. Same logic for the veto: if a large share of a program's questions end up "unacceptable," the tool flags it. A veto that becomes the norm no longer protects anything.

And one rule we hold ourselves to: questions are scored with what we know at framing time, never with the results obtained afterwards. Otherwise the model would always be right… in hindsight. Like everyone. On SoCute, we assumed a single type of user; there were two. That's why an assumed score becomes a range, and a question to check.

5. Prototyping every case

Our frameworks aren't wrong. They've just never been tested by the people who use them.

What do I take away from all this? We work with deeply imperfect frameworks: RICE, risk matrices, the FMEA score. And we don't question them. Not out of laziness, but because nobody has time to prototype every case. Cox did it by hand. Respect.

That's where AI changes the game, and not where we expect it to. Not by deciding for me: by widening what I can read, and what I can test. It brought me sources I would never have found on my own, from medical device risk management to multi-criteria decision analysis, and I read them. It let me replay each case, SoCute and then a quality inspection project in textiles, until I could see where the model breaks. AI didn't give me the right answer. It gave me the means to see where ready-made answers go wrong.

AI augments me. It doesn't steer me.

So next time a score settles your prioritization, try something dead simple: look for two very different options that get the same score. If you find them, the score isn't measuring what you think it is. And that's where the real conversation starts.

Métitra is a work in progress, and open source. If you'd like to test it, or break it, I'm all ears.