What do customers want - other than faster horses?

A cautionary tale of treating what customers say as gospel

Engineer: Hi Product Manager, the new UI we’re working on, it’s not clear how this average here is calculated, I think it will be a problem.

PM: What do you mean? What’s the problem?

Engineer: On the page here (point to the page), it says “average”, but it doesn’t say what items are included in this averaging calculation. In this context, it could either be the average of set A, or set B, or set C - they are all valid possibilities.

PM: I’d assume it’s the average of set A.

Engineer: But.. It might not be clear to customers?

PM: We ran a few validation sessions with customers as well as internal teams who are experts in this part of the system. They all said they understood and didn’t have any questions on how the average is calculated.

Engineer: The customers are not thinking about this logic deeply enough.

PM: Are you calling customers dumb? It’s a very dangerous thing to call customers dumb, because we’ve built many things we imagined are important but weren’t. We need to actually listen to what customers are saying.

Engineer: I actually agree with you in general. In this case though, I’m not saying customers are dumb.

I’m saying at this part of their workflow, when they are going through this page, they’re not focused on this specific logic, so they may genuinely think they understand.

But later on when they dig into the result of their set-up here, when they realize it doesn’t match their expectations, they wouldn’t have a way to find out exactly how the average is calculated. At that point, their only option is to ask our support team, which causes more work for us.

PM: How do you know this is actually going to be a problem for customers? Just because you see it as an engineer, doesn’t mean it’ll be a problem for customers.

Engineer: That’s true, but we have historical data and questions exactly like this have come up time and again, that took our engineering team a lot of time away from their planned work.

PM: But this problem didn’t come up during user testing and validation sessions. Are you saying we can’t trust these sessions?

Engineer: When you say this problem didn’t come up, did we ask the customers how they thought the average was calculated, and they all explicitly said it was for set A?

PM: Hmmm…

Engineer: Right. So.. I’m not saying we can’t trust customer validation sessions. They have great value. But it doesn’t make sense to take what customers say as gospel, and ignore problems just because it didn’t come up in the sessions.

We are the team building it, we’ve thought about the problem for longer and more thoroughly. When customers are taking a small part of their busy day to give us feedback on a solution, we can’t expect them to uncover every problem with it, let alone the problems that are not immediately in front of them.

Heck, half the time, the customer isn’t necessarily giving it their full attention, because in that moment, what customers want, other than faster horses, is to get back to their actual work.

Unfortunately, talking to customers isn’t a sufficient substitute for thinking deeply about what we do.