Lecture 16 - How to Run a User Interview (Emmett Shear)

Emmett Sheer, CEO of Twitch, discusses the importance of conducting effective user interviews for startups. He emphasizes learning from past experiences, such as the limited success of Kiko Calendar due to a lack of user engagement and how Justin TV's pivot to Twitch involved extensive user interviews to identify key market needs. Sheer highlights the significance of understanding user problems without imposing preconceived solutions, focusing on broadcasters' feedback, and addressing non-users' barriers to expand market reach. He advises startups to avoid common pitfalls like relying on easy-to-access users and emphasizes the need for ongoing user engagement as the market evolves.

• Emmett Sheer, CEO of Twitch, shares insights from his experience with startups, emphasizing the importance of user engagement.
• His first startup, Kiko Calendar, failed due to a lack of user engagement and understanding of the target market.
• The second startup, Justin TV, was initially successful because the founders were the primary users, but it faced limitations due to a narrow user base.

"We didn't know anything about calendars; neither of us were users of calendars nor did we during the period of time we worked on Kiko talk to anyone who actually did use a calendar."

• The failure of Kiko Calendar was attributed to the lack of user interaction and market research.

"We were our own consumer. We had this idea for a television show, Justin TV, a reality show, but Justin Con's life and we built a whole set of technology and website around the reality show we wanted to run."

• Justin TV initially succeeded because the founders built a product they personally wanted, but it struggled to expand beyond its initial concept.

• The pivot from Justin TV to Twitch involved identifying broader use cases and engaging with a wider audience.
• Emmett led the gaming initiative, recognizing the potential market for broadcasting video games.
• Conducting user interviews was crucial in understanding the needs of broadcasters and shaping Twitch's product features.

"We ran a very large number of user interviews. We talked to a lot of people and brought that data back and that formed the core of all the decision-making for the next 3 years of product features on Twitch."

• User interviews provided essential insights that guided Twitch's product development and strategic direction.

"We determined that the broadcasters were the most important people. The reason we determined that was when we went and looked into the market, we looked into what determined why people watched a certain streamer."

• Identifying broadcasters as key users was a strategic decision that influenced Twitch's growth and success.

• Emphasizes the importance of identifying the right target users for a product.
• Suggests that who you talk to is as crucial as the questions you ask during user interviews.
• Advises using judgment and critical thinking to determine the target user base.

"Who you talk to is as important as what questions you ask and what you pull away from it."

• Understanding the right audience is essential for gathering useful feedback and developing a successful product.

"There isn't a recipe. It comes down to think really hard and use your judgment to figure out who you're really building this for."

• Determining the target user requires thoughtful analysis and understanding of the market.

• Emmett conducts an exercise to help the audience practice identifying potential users for a lecture-focused note-taking app.
• Encourages participants to think about different types of users and their needs.
• Highlights the importance of considering various user groups beyond the obvious ones, such as college students.

"Think about who you would talk to. Who's going to give you that feedback that's going to tell you whether this is good or not?"

• Identifying potential users involves considering various perspectives and needs.

"One of the problems with selling things to college students is that college students don't actually spend very much money."

• Understanding the purchasing behavior of potential users is crucial for targeting the right audience.

• Encourages exploring a wide range of potential users and stakeholders to ensure comprehensive user engagement.
• Suggests considering non-obvious user groups, such as school administrators or parents, who might influence the product's success.

"You want to have that broadest group you possibly can. You don't just want to talk to one type of person and learn that."

• Engaging with a diverse group of users can provide valuable insights and improve the product's market fit.

"You might talk to parents... There are a lot of groups that are potential that aren't necessarily the obvious user but who are critical to your app's success potentially."

• Considering various stakeholders can help identify additional opportunities and challenges for a product.

• The focus of the user interview is to understand the note-taking habits of a college student to inform the development of a note-taking app.
• The interview aims to gather insights into the tools and methods used for note-taking, collaboration preferences, and reviewing habits.

"I take notes in a variety of ways. I like to, um, now because of speed and efficiency and just to come back to it later. It's easy for me to just take notes on my laptop."

• The interviewee uses a combination of digital and traditional methods for note-taking, preferring digital for speed and efficiency.

"Evernote is easy if I'm trying to just collect it for myself. I think um and yes you can share but I think Google Docs for me is um easier to share."

• Different tools are used based on the purpose: Evernote for personal use and Google Docs for collaboration.

• The interviewee elaborates on the use of different tools and methods for note-taking, including the use of pen and paper for complex diagrams.
• The preference for digital notes is due to ease of access, sharing, and collaboration with classmates.

"It's mostly mine because I'm pretty picky about the way I like things organized. Um, like designwise or formatting."

• Personalization and organization of notes are critical to the interviewee's study habits.

"If it's notes that I've taken for like talks like these, for example, or if it's like interview prep that I'm doing, um, I tend to go back because it's things that I like to kind of keep fresh in my mind."

• Notes are reviewed more frequently if they are for personal development or preparation for future events.

• The goal of initial user interviews is to understand user problems rather than focus on specific features.
• It's important to avoid asking users directly about features to prevent the "horseless carriage effect."

"The main thing you're trying to do when you're running these first set of interviews is not necessarily get like questions about like user flows and like optimizing that or questions about uh like the specifics of um of of any of that stuff."

• Initial interviews should focus on understanding user problems, not on gathering feature requests.

"If you start asking people about features, so you want to stay as far away from features as possible because the things they tell you wind up almost feeling overwhelmingly real."

• Direct feature requests from users can be misleading and may not address the actual problem.

• Consider developing features that enhance existing tools rather than building a completely new product.
• Validate new feature ideas by understanding if they solve a real problem for the user.

"If you're going to build just one feature on top of Google Docs, uh, what would that feature be, right?"

• Focus on enhancing existing experiences with incremental improvements that are compelling to the user.

"What if we had a uh Google Docs that had the collaborative aspects and the group aspects of that, but where you uh you could pull in more little one-off notes and it was it was designed more around note-taking."

• Consider features that integrate the benefits of different tools, such as combining the collaborative nature of Google Docs with the note-taking flexibility of Evernote.

• Validate feature ideas by creating prototypes or minimal viable products (MVP) rather than relying solely on user feedback.
• Avoid asking users directly if they like a feature; instead, observe if they would switch to using it.

"The one question you can't ask is is this feature actually good or not?"

• Directly asking users if they like a feature can lead to biased positive feedback that doesn't translate into actual usage.

"Usually the best thing you can do is uh is is really just hack something together."

• Consider creating quick, minimal solutions to test if the feature truly addresses user needs before investing in full development.

• Testing a product's appeal by getting users to commit financially is a strong validation of interest.
• Charging money for a product, even a small amount, can clarify whether users are genuinely excited about it.

"If you can't actually put it in front of people, it's really, really hard to find that out for bigger things where you're actually trying to get people to spend money."

• Demonstrates the challenge of gauging interest without direct user interaction.

"If you go out there and you if you can get people to say, 'Hey, I'm going to give you money.' The money test is amazing."

• Highlights the importance of financial commitment as a true indicator of user interest.

• Existing users often provide detailed feedback on specific features they use regularly.
• Feedback can include requests for improvements or new features based on their current usage.

"They come back to you with actually very detailed things about features because they actually get mired in the feature."

• Indicates that users deeply engaged with the product can offer specific, actionable feedback.

"This is what the Justin TV broadcasters wanted us to build and this is what they felt pain using the product."

• Shows that user feedback often highlights pain points and desired improvements.

• Competitor users can provide insights into what might be deterring them from switching to your product.
• Feedback from this group often focuses on critical issues like stability and monetization options.

"They complained about us not having a revshare program, where they talked a lot about how they were trying to make a living."

• Emphasizes the importance of monetization features for attracting users from competitors.

"The things that people who didn't use our service said about what they cared about was completely different than the things that people who were using the service cared about."

• Highlights the difference in priorities between existing users and potential users from competitors.

• Non-users provide crucial insights into barriers preventing market expansion.
• Addressing non-user concerns can lead to significant growth opportunities.

"In the case of gaming broadcasting, almost everyone's a non-user, right? The majority, and this is true for most new products, the majority of people you're competing with are non-users."

• Stresses the importance of understanding non-users to expand market reach.

"We bought people computers. We worked really closely with gaming broadcast software companies to help the people who made the broadcasting software to make that better."

• Demonstrates proactive steps taken to address non-user barriers.

• Prioritizing core issues over specific feature requests can lead to more impactful improvements.
• Understanding the underlying goals of users helps in addressing the real problems.

"People wanted money. People wanted stability and quality. People wanted universal access for viewers all around the world to be able to watch them."

• Identifies the fundamental needs that were prioritized over specific feature requests.

"We dumped almost all of our resources into things that none of no one ever mentioned in an interview."

• Suggests that addressing core issues rather than specific requests can lead to successful outcomes.

• Engaging with users through research can lead to product improvements that align with user needs.
• User research helps in identifying representative user groups and addressing their concerns.

"The people we converted first to our product were the people that we talked to about user research."

• Indicates the effectiveness of user research in converting users to the product.

"We picked people who were representative and we picked big broadcasters, small ones, medium ones and we made sure we were addressing their concerns."

• Engaging with the right users is crucial for obtaining valuable feedback.
• Avoid the mistake of talking to easily accessible users instead of those whose insights are necessary.
• Efforts should be made to identify and reach out to the right individuals, even if it requires extensive networking.

"The other big mistake people make is talking to who's available rather than talking to who they need to talk to."

• Engaging with easily accessible users can lead to suboptimal data collection.

"We spent like weeks digging for identity information and figuring out who these people were. Uh, so we could contact them, so we could talk to them."

• Significant effort may be required to identify and reach out to the right users, which is necessary for obtaining valuable insights.

• Securing buy-in from the company requires demonstrating the value of user insights.
• Recording user interviews can effectively convey the importance of user feedback to other team members.

"There's something magic about showing them the interview though. So, I really recommend you record interviews."

• Recording interviews allows for compelling presentation of user data to stakeholders, fostering trust and alignment.

• Conducting interviews via interactive platforms like Skype is preferable to email.
• Interactive interviews allow for deeper exploration of unexpected insights through follow-up questions.

"You don't want to do interviews over email if you can avoid it."

• Email interviews lack the interactivity needed to explore unexpected insights.

"The most interesting things you learn in interviews come from the interesting tell me more."

• Interactive interviews facilitate deeper insight extraction by encouraging users to elaborate on unexpected points.

• Language barriers complicate user research in non-English speaking markets.
• Efforts to overcome language barriers may not yield fully representative samples.

"Twitch works way better in English speaking countries than it does in non-English-speaking countries."

• Language barriers impact the effectiveness of user research in non-English speaking markets.

"The very fact that they are a fluent English speaker means they're not representative of all the people who don't speak fluent English."

• Hiring bilingual individuals or translators may not fully address the challenge of obtaining representative samples.

• Various channels can be used to engage users, such as on-site messaging and email.
• Compensation is generally not necessary if users are genuinely interested in the problem being addressed.

"The channels we used to reach out to them were on-site messaging systems."

• On-site messaging and email are effective channels for engaging users.

"We tended not to compensate people."

• Genuine user interest often negates the need for compensation in user interviews.

• On-site user feedback tools are valuable for refining product features but not for determining what to build.
• Such tools identify usability issues but do not address fundamental user needs.

"That kind of work is super super important and it can tell you lots of things about where you went wrong building something before you launch it."

• On-site feedback tools are crucial for identifying and resolving usability issues pre-launch.

"It doesn't tell you what to build."

• These tools are not suitable for determining the core product features or solving user problems.

• Startups with limited resources should prioritize engaging users of competing products.
• This approach leverages existing user behavior and facilitates quicker adoption.

"We focused on the competing people using competing products because we knew that they already were interested in the behavior that we needed."

• Prioritizing users of competing products can lead to faster adoption and growth.

"Building something that some people want generally generalizes."

• Focusing on a specific user group can lay the foundation for broader market appeal.

• User engagement strategies need to evolve as the company grows and the market changes.
• The initial user base may differ significantly from the user base after several years.

"The pool of people you care about is going to shift over time."

• Ongoing user engagement is necessary to adapt to changing market dynamics.

"The people who get you started like the crucial people to get your product started for the first 6 months are not who will be using it three years later."

• Acknowledging and adapting to changes in the user base is essential for sustained success.

• Users can contribute valuable feedback by openly sharing their thoughts and experiences.
• Detailed and candid user feedback helps companies understand user needs and context.

"I want a user to tell me about, uh, what they like what they're really thinking, right? And what what what their problems really are and to just sort of ramble."

• Encouraging users to share openly provides richer insights into their needs and challenges.

"The more you learn about them as a person and sort of them the their what's going on in the context of what they're doing uh the easier it is to understand why they want the things they want."

• Understanding user context is critical for developing products that meet their needs.