Episode highlights
00:01:36 – Noam’s background
00:07:07 – Introduction to NPS flaws
00:16:10 – Creation of npsistheworst.com
00:19:59 – Alternatives to NPS
00:23:26 – Focus on custom KPIs over NPS
00:31:33 – UX researchers’ role in metrics and ethics
00:34:30 – Noam’s closing thoughts
About our guest Noam Segal
Noam Segal is a UX leader with experience at companies like Airbnb, Intercom, Wealthfront, X, and Meta. He’s currently Senior Research Manager at Upwork and founder of The Disruptive Coach, offering career coaching and resources for tech professionals. He helps individuals and teams grow, and has taught at Stanford and Harvard, as well as spoken at major industry conferences like UXRConf and Quant UX Con.
NPS violates countless principles in survey science. It’s noisy, overly complicated, and doesn’t accurately reflect customer satisfaction.
Podcast transcript
[00:00:00] Tina Ličková:
Welcome to UX Research Geeks, where we geek out with researchers from all around the world on topics they are passionate about. I’m your host Tina Ličková, a researcher and a strategist, and this podcast is brought to you by UXtweak, an all-in-one UX research tool.
This is the 44th episode of and you are listening through the discussion with Noam, who really has a lot of experience in the UX research space. Working for Metta, Airbnb, Twitter. Now, working for Upwork. Falling numb LinkedIn, I came across his posts about NPS and why is it such a bad practice. This is where I reached out and told him about having an episode on explaining people and talking about why we think NPS is really bad. Very humbly, I think we explain why NPS is such a damaging practice. But more importantly, I think we made a point that there are so many other alternatives. I can bring a bigger value. to the businesses. That there is not even a reason to do NPS.So tune-in into very nice conversation.
I think you will enjoy it.
Tina: Noam, you are very well known in the business. But I would like to know what you want to say about yourself?
[00:01:36] Noam Segal: It’s a good question. I’ve shared about myself quite a few times at this point and I still struggle with it a little bit. I think what I would choose to share Is that especially these days in a world where I’m struggling to find optimism and things have been difficult for years now across several different domains.
I remain a techno optimist. I believe that technology has a critical role. in making our lives better, in making our world better, and I chose the profession I’m in because I think we all have a critical role in building technology in a thoughtful way, in a way that’s built based off of solid insight and knowledge into people and What they truly need and I take that responsibility very seriously as I think we, we should all and I just hope that we’re able to manifest through our work that promise that I see in technology and I’m very excited about everything that’s going on and at the same time. well aware that if we mess this up, and if we don’t do our jobs technology can be a force for evil, technology can be destructive. Potentially. And I think I chose to bring that up rather than a list of companies I’ve worked at, or things I’ve done. Especially because in this episode, we’ll be discussing something that’s been top of mind for me for a while.
And it has to do with our responsibility to the world, and the fact that we need to be doing Proper, rigorous, thoughtful research to inform how we build technology products. And if we don’t, that’s a very serious problem. So that’s what I’ll choose to share about myself.
Otherwise, in case people are curious, I live in Florida these days. I’ve lived on or in the. U. S. Midwest. I’ve lived on the West Coast and now I live on the East Coast. Been in the United States for well over a decade at this point. I’m originally from Israel, born and raised, and I’ve been working in the UX field for coming up on two decades now, in a couple of years.
And these days in my… Full time role, I’m a Senior Research Manager at Upwork, and beyond that I do a few other things like teach and coach. And I’m sure there’ll be some links in the show notes afterwards to all of those things if you want to check them out.
[00:04:37] Tina Ličková: Yeah, I will stay a little bit there because also your LinkedIn page is Promoting or showing it very clearly that you do coach people for their careers and now you’re working for up work, which is. also for career development. What kind of links do you have there between those things for you personally?
[00:04:56] Noam Segal: I think these past few years of the pandemic have been incredibly challenging for people in terms of employment and career. We’ve seen The employment reality changed so much, the move to remote work, the move back, I hope not, to office work, hybrid work, lots of confusion right now.
I’m a huge believer in remote work, and I think the data supports that in many cases, it’s a wonderful thing. But that’s still something that’s being highly debated in the tech community and outside of it. And then beyond that, we’ve seen mass layoffs and reorganizations within companies changes in what VCs and other investors are choosing to invest in.
And it’s been really hard in particular for people looking for. their next employment opportunity and to improve their lives through economic opportunities.
That’s exactly what we’re focused on at Upwork, at Scale. At Scale we’re trying to provide economic opportunities and help connect between people who need to get work done and people who want to do the work and it’s a marvelous marketplace.
That enables that, and for me, with my coach hat on, I try to enable that at the individual level by supporting people, in particular in tech, with their careers, with their career choices, with their career development, because the fact is, working in tech is a rollercoaster, it’s hard, lots of tough decisions, lots of tough situations.
that people aren’t always sure how to deal with, transitions between being an individual contributor and a manager, difficulties in finding your voice as a senior individual contributor, or as a manager and a leader and so much more. So that to me is the connection.
[00:07:07] Tina Ličková: Thank you. I just wanted to introduce it because I think you are a great name to know for people who are looking for that kind of coaching. And you already pointed it out.
We are going to talk about those three dirty letters. NPS, Net Promoter Score, and my first question would be I know you hate NPS, I can relate to that, but why should people be hating NPS as well? Let’s start there.
[00:07:33] Noam Segal: First of all, I’d rather save the word hate for very specific things. But I do think that the use of NPS should be stopped.
I think NPS should be eradicated, and I was pondering on NPS before we started this conversation. And one thing I thought to myself, which tells you a lot about NPS, is that when companies report on Net Promoter Score, they literally say, Our NPS is the highest in the industry. Our NPS is 57.
Our NPS is this. Our NPS is that. And… I think it’s common knowledge that companies claim to be human centered, customer obsessed, focused on their users. But notice that way that companies word what’s going on with their NPS. They’re not saying our customers are super satisfied, our customers are engaged with our product, our customers love what we’re doing and what we’re building.
No, no, no, no, no. They’re reporting NPS as a metric. And to me, that says something. I almost think that subconsciously, even the companies themselves don’t actually believe that there’s a real relationship between NPS and the things we actually care about. It’s just a vanity. metric. It’s just garbage. And all it’s saying is that my garbage is nicer than your garbage, or I’m on the way to get my garbage to a level where it’s better than your garbage.
But it’s garbage. It’s a garbage metric. I don’t appreciate the use of it, to say the least. And I think we need to do better moving forward.
[00:09:43] Tina Ličková: I like that when we look at the high level yours. Pointing it out very rightly that companies talk about their NPS and not about their customers or users. But what is it exactly about the method, methodology of it that drives you crazy for not liking it if we don’t want to use the hate word?
[00:10:05] Noam Segal: Yeah.
People can get PhD and dedicate their entire careers just to surveys. Survey science is incredibly complex. And I think a lot of the criticisms of surveys aren’t really criticisms of the methodology, they’re criticisms of the many cases where surveys are implemented in ways that go against survey science.
And perhaps one of the best examples of that in the history of our profession is indeed NPS. NPS really violates countless principles in survey science, and that’s what bothers me. one, for example, which we know just doesn’t work very well, is when you ask people about what they intend to do in some future, do you intend to recommend my product in the next two weeks, 30 days, maybe longer? to a friend or a colleague of yours. We know that when we ask people about their intentions, there’s a weak relationship, if any, between those intentions and the actual behavior. We can talk about the calculation of NPS, the NPS formula, which is completely wacky.
And it can mask significant improvements in the product. And it can mask what’s really going on with the people you’re trying to serve. It’s overly complicated, it’s cognitively taxing. I… would rarely recommend anyone use a 10 or 11 point scale for anything, really. Not to mention a scale that’s then randomly divided into three subcategories within that scale of promoters, passive people, and detractors.
Oy, I could go on and on, but the bottom line is that NPS just violates so many survey science principles, it’s so noisy as a metric and we can do a lot better and, sadly, the reason why it became so popular, in my humble opinion, and it produces this kind of magical number that sadly a lot of companies use so you can compare your magical number to other companies magical number.
And the complexity of it, which again is ridiculous and within it there are so many errors and issues. The complexity of it makes it feel sophisticated. Whereas if I just ask you something like, how satisfied are you with… This product or this part of the product from one to five, it’s like, ah, too simple, like it doesn’t make sense to just ask people how satisfied they are from one to five.
Let’s ask from zero to 10 and divide it into three and make a scale from minus this to plus that and call it something very sophisticated, make up an acronym and, why it’s used, and that’s why I think it shouldn’t be used.
[00:13:36] Tina Ličková: I’m just wondering, because one time in my life, I came across an MPS, and I was like, yeah, hell, yeah, I would recommend this. Knowing for sure, and I know it for my personal life, or day to day life, that, yeah, I am recommending this tool that they are asking me about.
But this particular tool that I used was something that I will be like, hell yeah. And I’m wondering if there are any cases coming from this experience of mine where you would recommend using NPS. I think I know the answer, I’m just, .
[00:14:15] Noam Segal: Yeah, my answer is a definitive no. And… And I’ll tell you what, I just don’t see why. I don’t see why. Let me tell you something. One of the methods in research is diary studies, right? And diary studies used to be conducted by sending people physical diaries, physical notebooks to their home and having them respond to questions. And then they would send it back and the whole process would take a month or several weeks or however long.
And these days, obviously, we have these amazing digital platforms that enable us to connect with people. in a much more sophisticated and immediate way. Now, why do I bring this up? I bring this up because we have all of the means, all of the technology, to conduct accurate measurements of people’s satisfaction, people’s loyalty, people’s engagement, the growth of our product, without using crappy tools like It’s not as if we’re lacking any abilities to conduct accurate measurement.
So if you ask me isn’t there any case where we just should use it? Again, my response is no, because I don’t really understand how… Any of us, any company, no matter what its size, from an early stage pre seed startup to a large public enterprise company, why any company would have to revert to, or resort to using NPS?
I just don’t get it.
[00:16:10] Tina Ličková: if I understand you are the author of the website NPS is the worst.
[00:16:15] Noam Segal: I am, yes. I am the creator of that website. And by the way, a fun fact I always love to share about that website is I’m also the owner of the domain npsisthebest.com and it’ll send you to npsistheworst.com. That’s how much I dislike NPS as a metric. I am forcing people who love it to also see the website npsistheworst.com
[00:16:42] Tina Ličková: Great. if you want to know all the reasons and the history of why NPS is the worst, there you have the website.
But you just mentioned people who love NPS. I, and I remember a few situations in my life, either at clients or when I was employed, those eyes with stars. Oh, our NPS got better. And you’re new at the company and yet you just don’t want to go into the first fight and you don’t want to start your relationship with that person about how evil the NPS is.
How do you actually make sure that stakeholders, or let’s say, with a better expression, colleagues understand, who are not researcher wise, it’s so bad?
[00:17:26] Noam Segal: I think I have two things to say about that. First of all, there are two groups who love NPS, so to speak. The first is the people who came up with NPS in the first place and benefit from it.
They have a good reason to love NPS. Fred, I’m looking at you. And the other group is people who are in particular building companies and looking for investments and they need to show venture capitalists or angel investors or whoever they’re getting their money from that their NPS is extremely high or higher than competition because yes.
NPS, sadly, is currently widely used in our industry and it’s a way for companies to signal that they are doing better than other companies, supposedly. So that’s the first part of my answer. The second part is that it’s very clear to me that we are not going to be able to eradicate NPS quickly or easily.
And there will need to be. A certain period of time, a transition, where we continue to measure NPS, because people are still looking to hear about it, and also conduct other measurements, whether they are a standardized measurement that everyone knows about, or maybe not, maybe it’s a measurement tailored to the needs of your company.
And then once companies can show that their success is much more strongly correlated with those other measures, slowly we can start paring down the use of NPS until it is in fact eradicated. But that’s going to take time, and I do understand that for the time being, NPS is one of these things that has become a a standard in our industry.
Unfortunately. And once something is established as a standard, it takes time to change it. This is gonna be a paradigm change, which I insist on being part of. paradigm changes aren’t easy.
[00:19:40] Tina Ličková: And this is where it takes me now, is looking at it more constructively, and what is, in your opinion, what are the methods that we can use instead, which would have the same, maybe, power or interpretation for us?
[00:19:59] Noam Segal: So again, maybe two parts to my answer. The first is that we do have… other standardized metrics for things that supposedly NPS might be measuring. Satisfaction being one of them, right? Supposedly, if I would refer someone to a product, it’s because I myself, I’m satisfied with it. I would probably not refer to someone if I’m not satisfied.
And so we do have metrics like CSAT, which is just the customer satisfaction score. And I do feel that CSAT is a lot more rigorous and a lot more in line with service science principles than NPS. There are other options, for example, Thomas Charon, back in the day another well known research leader who’s written books about research and led research across several well known companies, he suggested a measure called actual NPS, where It’s a gradual step to move away from NPS, but it improves on one major issue with NPS, which I already mentioned, which is that in the original version of NPS, you’re asking people to predict what they might do in the future.
Whereas with actual NPS, you ask people, did you? mention this product to a colleague or a friend in the past couple of weeks? Did you actually refer anyone to it? And that in my mind is certainly an improvement. I, again, I think it’s a long journey to eradicate NPS, so that’s not bad.
So we have other standardized measures. like CSAT. We have better versions of NPS as a, an intermediate step on the way to getting rid of it. But I would say beyond all of that, going back to the point around us having so many tools these days and so many abilities to measure pretty much whatever we want, I strongly feel that we need to focus a lot more not on the standardized measures Where we compare ourselves to other companies Yes, it’s maybe good to know but at the same time pretty much every successful Founder and CEO of a tech company will tell you, you don’t want to be focusing on your competitors.
You want to be focusing on your users, your customers, and building better products for them. And every company has different Northstar metrics and key metrics that they care about. And they want to track. And if that company has a research team, or if they’re contracting a researcher, or if they have a researcher as an advisor, if they’re too early in the process, I would suggest to them, come up with your own proprietary set of well established, rigorous metrics that directly address your needs, given your stage and what you’re doing.
Don’t focus so much on the standardized, generalized, general. metrics that don’t really give you a lot of insight into what’s going on and what you need to improve. It’s not beneficial.
[00:23:26] Tina Ličková: when you’re saying every company should come up with their own KPIs and they own metrics, the underlying question is what to care about,
Do you have any kind of advice on how to get to the core of what we should focus on?
[00:23:45] Noam Segal: Just a small correction. What I said is that companies already have those things. Typically. Not that they need to come up with them. Companies typically do have certain goals or OKRs if they use the kind of OKR framework.
Metrics, or North Star metrics, top line metrics, second level metrics, etc. I wouldn’t be surprised if any tech company isn’t focused on certain types of metrics. Most of us, if not all of us, care about how users… are onboarding and starting to use our products. Most companies care about how people are engaging with their products and how they’re using them.
Most companies care about retention and churn and those sorts of things. But again, there’s a lot of flavors for those kind of goals or… or OKRs, and I think NPS has led companies to focus on the standardized metric rather than metrics that are much more closely aligned with what they care about. And keep in mind that depending on the stage a company is at and the types of issues they’re seeing in their data and in their analytics, they might want to be more focused on a particular stage in the customer journey, in the user journey, right?
Maybe they don’t want to focus right now on how many people are referring them to other people, because they’re not in that kind of growth stage, or maybe they have bigger issues, for example, with the onboarding to the product. So I just feel like, given all of the abilities we have, companies should be much more laser focused on the important issues they’re facing given their size and their stage and how they’re organized and what their product looks like and feels like what the experience is. and to make sure that they’re gleaning valuable, deep insights about their customers.
And I just want to say, because I feel like I have to mention this, that one of the reasons I am so excited about the era of artificial intelligence in the field of research specifically, is that we are going to be able to conduct research at a scale and at a depth.
Never seen before and we’re going to be able to adapt our research to the individual customer even and do so much more for our customers and our users by understanding truly what their issues are and not just how likely they are to refer to someone else. That to me is incredibly exciting. Ideas like gathering What some people would call thick data, qualitative data, but at the scale of quantitative data.
Ideas like conducting a survey where you don’t have some built in static logic where people are sent from one question to another, no, the questions change and adjust and are personalized for each individual survey.
All of these things are going to make NPS look even more Silly, embarrassing and just substandard compared to what we’re going to be able to do in the near future.
[00:27:40] Tina Ličková: maybe I will rephrase the question although I love where you land with it? Because I’m really looking forward to the future as well,But I’m going from a place where some companies don’t have these KPIs and metrics or really have so many of them that they don’t have a priority. So my question, maybe going back to that, would be, to know what is the priority towards our product.
[00:28:07] Noam Segal: I was lucky enough to work with a bloke named Lenny. At Airbnb back in the day when I was at Airbnb, and Lenny is the same. Lenny who went on to start Lenny’s newsletter and Lenny’s podcast, now one of the most popular tech newsletters and podcasts in the world. I really admire him as a person in tech, as a content creator, and so many other things, and Lenny reminded me once of that.
In our field, everything starts with, you might say, the problem or pain, even. We’re talking about Zoom. Why does Zoom exist? Why did Zoom become successful? Because it was painful, and hard, and a crappy experience to talk on video before Zoom came about. And right next to me, outside of the frame, you can’t see it, is a Peloton.
Why does Peloton exist? Because it was hard and painful to leave your home, certainly during the pandemic, and exercise, even if it’s only five minutes. There was five minutes on the way to the gym! What stopped you from going to the gym? And you were just wasting your money not going to the gym i’m not saying this I go on my peloton all the time, by the way, But to be clear but every amazing startup and maybe even every amazing story by the way starts with this pain this problem that’s irking people and stopping them from living their best lives, and doing what they want to do, and thriving.
And so really, if you’re going to measure anything, I would suggest you reflect on that first. Not reflect on whether someone’s referring to you, or not referring to you, or whatever. But where is the pain? Where is the problem? What are the barriers that are stopping the people you’re trying to serve? From doing what they need to be doing, and once you land on that, I think you can do quite well.
And some people, including Thomas Charon, who I mentioned earlier put a more positive spin on this, and discuss how you measure happiness, the other side of pain and how you can come up with E UX metrics.
But again, I just can’t say this enough. I believe these key UX metrics should be Tailored to what you’re trying to do if your focus Is on being helpful, like Google’s focus once was with their search, like their North Star metric was, I think, helpfulness. Then measure helpfulness. Don’t measure whether you’re gonna send Google. com to a friend or a colleague. Measure helpfulness. Is your focus on trust? Okay. There are plenty of ways to, rigorous, well established. Scientific ways to measure trust. Pick one of them and measure trust.
[00:31:33] Tina Ličková: Maybe my last question, What is the user’s researcher responsibility in metrics and surveys and NPS?
[00:31:42] Noam Segal: I think first and foremost We all have to, first of all, educate ourselves. Again, survey science is incredibly complex.
So I would suggest our most important responsibility is to educate ourselves on the best practices within survey science. And how do we make sure we’re following them? I also think that, as researchers, we want to be more connected than we have been in the past to the fabric of the company we work for.
Companies like organisms, they have all of these different mechanisms within them. And I don’t think, traditionally, researchers have always paid enough attention and built up their understanding of business strategy, of financials, of metrics. I think maybe we’ve been, in certain cases, a little too insular.
So I think that beyond continuously educating ourselves on our own profession, it’s also our responsibility to educate ourselves about the companies we work for and those metrics that you mentioned and the analytics that the company has and the business goals and strategies and armed with those two types of knowledge, I think we can do a much better job at measuring what, what matters, and I think that beyond that, again, I’m a tech optimist, but I do feel that Few people have more of a responsibility to ensuring the outcomes of building technology are positive than researchers. As researchers, we are somewhat more objective than lots of the other people building products.
We have this passion and ability to understand people, understand the situations they’re in, and also maybe identify some of the pitfalls and the dangers and the issues with what we’re creating.
And so I think that’s also our responsibility, to bring up those things, to make sure that we are building artificial intelligence products and other products in a responsible way, in an ethical way, that we are balancing pace with ethics. Those to me are the key responsibilities of researchers in the kind of modern age of research, which is going through quite a few shifts these days, I’d suggest.
[00:34:30] Tina Ličková: Beautiful closure, beautiful closing. Thank you very much for your words and for sharing the wisdom of yours.
[00:34:38] Noam Segal: Of course thank you for having me. It was a real pleasure. I will say that I’m always delighted to connect with people in tech, in UX research. And I invite anyone who’s interested to connect with me on LinkedIn.
if you are… Interested in changing something about your career, getting some coaching, getting some advice.
I am absolutely available for that and excited for that. A few things give me more joy these days than connecting with fellow tech employees, UX professionals, everyone really. And to help people think through some of the pain points and the problems in their own career, not just the products we’re building.
So, thanks so much for having me. It was a real pleasure and I really appreciate the chance to talk about these things and to continue my battle against NPS.
[00:35:35] Tina Ličková:
Thank you for listening to UXR Geeks. If you enjoyed this episode, please follow our podcast and share it with your friends or colleagues. Your support is really what keeps us going. If you have any tips on fantastic speakers from across the globe, feedback, or any questions, we’d love to hear from you too. Reach out to UXR Geeks podcast at UXtweak. Thanks for tuning in.
💡 This podcast was brought to you by UXtweak, an all-in-one UX research tool.
