20 Product The Ultimate Guide to Product Reviews What Makes the Best vs the Worst How Often Should They Be Who Should Be Invited Who Sets The Agenda How to do FollowUps Post Product Reviews

In the latest episode of "20 Product" with Harry Stebbings, the focus centers on the art of conducting effective product reviews. Scott Belsky, CPO at Adobe, emphasizes the importance of prototyping and contextual understanding of customer experience, advocating for a prototype-driven approach to align teams. Tony Fidel, creator of the iPod, iPhone, and Nest, discusses targeting the right customer and optimizing the product experience. Alpana Chenna Pragada, CPO at Robin Hood, stresses the need for minimum viable process and stage-appropriate reviews, while Lenny Rujitsky highlights the importance of tying reviews back to the core problem being solved. Lastly, Kyvan Bakepool, former head of consumer product at Twitter, shares insights on structuring reviews and the dual role they play for both the team and leadership. Across the board, the experts advocate for clarity, conviction, and customer-centric prioritization in product reviews, particularly in the context of remote work and the abundance of feedback in the digital age.

• "20 Product" is a monthly show featuring leading product minds discussing tips, tactics, and strategies for building high-performing product teams.
• Product reviews are a common topic of interest, with listeners seeking advice on their structure and execution.

This is 20 product with me, Harry Stebings. Now, over the last six months, the most common question we've had from thousands of product people listening to the show is on product reviews.

This quote introduces the show "20 Product" and highlights the interest in product reviews among the audience.

• Prototypes are essential for aligning teams and avoiding unnecessary disagreements.
• Reviews should consider the broader customer experience, tracing the customer journey to the point under review.
• Questions to ask during reviews include customer awareness of their position, what to do, and what comes next.

The first thing is I deeply, deeply believe in the fact that a prototype is worth 100 meetings.

Scott Belsky emphasizes the value of prototypes in facilitating productive discussions and alignment within teams.

• Analyze if the customer knows their journey, their current action, and their next steps.
• Discuss the object model and changes needed to reduce cognitive load for the customer.
• Ensure the right people, such as designers and product leaders, are in control of the discussions.

Do they know how they got here? Do they know what to do? Do they know what to do next? If not discuss the object model and what needs to change?

Scott Belsky outlines the critical questions that should be addressed during product reviews to ensure customer clarity and ease of use.

• The design and product team should initially review the product without engineers to focus on what's best for the customer.
• Engineers join later to provide input on feasibility and cost.
• The process emulates Walt Disney's three-room concept for idea generation, critique, and reconciliation.

I used to be a fan of being very thoughtful about who's in that meeting.

Scott Belsky describes the selective approach to inviting team members to product reviews, ensuring each phase is focused and productive.

• Align around a prototype to ensure everyone sees the same future.
• Use project or program managers to capture actionable items during conversations.
• Ensure immediate documentation of feedback to prevent loss of critical details.

Number one is aligning around the right prototype experience and iterating it until everyone sort of sees the same future.

Scott Belsky discusses the importance of alignment on a prototype for successful prioritization and development.

• Focus on optimizing for desirable problems that arise from customer engagement.
• Prioritize solving issues that prevent customers from finding value, rather than adding enhancements.
• Aim to get customers through the first mile of the funnel for initial success.

My trick that I ask myself all the times or not trick, but core question is always around how to optimize for the problems you want to have.

Scott Belsky shares his approach to prioritizing which features to include in a minimum viable product (MVP) and what to defer until later.

• Intuition guides what products to create, while data helps refine and improve them.
• Customers may not initially recognize the need for innovative products.
• Data becomes crucial after the product concept is validated by initial customer engagement.

Intuition is what brings you to a mountain to climb. Data is what helps you climb it.

Scott Belsky provides an analogy to illustrate the respective roles of intuition and data in product development and decision-making.

• Customers often have unarticulated needs that are uncovered by asking about their struggles rather than pitching solutions.
• A detailed understanding of customer struggles can lead to the creation of a product that intuitively meets their needs.
• This approach is exemplified by the development of a portfolio platform that deconstructs portfolios into projects with relevant data.

"Tell us what you're struggling with. And they're like, oh, well, that's easy. I never get attribution for my work. My portfolio is always out of date."

This quote highlights the importance of directly asking customers about their struggles to uncover their true needs, which can inform the development of a more effective product.

• Defining the target audience and desired experience is critical in product design.
• It's important to have a clear priority path for the product and optimize everything around it.
• Drawing from past experiences with other products can help shape the design of a new product.
• Collaboration between teams can reduce unnecessary steps in the product experience.

"Who is the audience and what kind of experience are we trying to go for?"

This quote emphasizes the necessity of understanding the target audience and the experience the product is meant to offer as a starting point for product design.

• Product designers must train themselves to switch between focusing on details and maintaining a broader perspective.
• It's important to consider the customer's viewpoint, especially for those new to the space.
• Asking simple questions like "why" can lead to a deeper understanding of the product's user experience.

"How are they going to think about thinking like they might think and put your role playing out of like, why did this come up on the screen? I don't understand."

This quote underscores the practice of adopting the customer's perspective to identify potential issues and improvements in the product's design.

• Product reviews are essential, and they should be tailored to the stage of the product—solve, scale, or sustain.
• Early-stage reviews should involve a small group with a divergent mindset, focusing on problem-solving.
• Later-stage reviews should be data-driven and have a clear framework for decision-making.
• It's important to include key cross-functional team members in reviews, especially research and design leads.

"You want a small group and you want the problem solvers in the group, regardless of the organizational hierarchy."

This quote illustrates the need for a focused group of problem solvers in early-stage product reviews, highlighting that organizational hierarchy should not dictate participation.

• Senior leaders must be careful with their communication, as their suggestions can be interpreted as directives.
• It is crucial to clarify whether feedback is a strong conviction or merely a suggestion for consideration.
• Separating feedback into directives and suggestions can prevent unnecessary work and keep the team focused on priorities.

"Your whisper is heard as a scream, right? You just say, hey, you guys should think about it. And the next thing you know, they've spun up like gigantic effort to say, and like it's a telephone game."

This quote conveys the weight of a senior leader's words and the need for clear communication to ensure that feedback is understood and acted upon appropriately.

• Product reviews generate a lot of feedback and ideas, which can be overwhelming.
• Prioritization is key to managing the influx of feedback.
• Good note-taking is an essential skill for product managers to capture and organize feedback effectively.

"One of the best things I learned from Google, wide eyed, pushy rail, PM taking good meeting notes."

This quote points to the value of meticulous note-taking in managing the information gathered during product reviews and ensuring that important feedback is addressed.

• Meetings should have a clear structure with specific action items assigned to specific people.
• Discussions should be summarized with key decisions made being highlighted.
• Not all action items have the same level of priority; some require immediate attention while others are considerations.
• The lead responsible for the discussion should own the prioritization and publish it, indicating the urgency of each action item.
• Trust in the product area lead's judgment is essential for effective prioritization.

"You had very clear action items. You had specific people, specific things that they had to follow. Discussion, summary, key decisions made. And then on top of it, the twist I put on top of it. Not all action items are made equal to your point."

This quote emphasizes the importance of clarity in meeting outcomes, assigning specific tasks to specific people, and recognizing that not all tasks have the same level of importance. The speaker also mentions adding their own twist by prioritizing the action items.

• Conducting reviews that are not stage-appropriate, either by scaling too early or having disorganized reviews with unclear ownership and agendas.
• There should be a balance between focusing on the potential of early ideas and not rushing into scalability concerns.
• Reviews should have a clear agenda, with specific people taking charge and setting the tone to avoid a cacophony of voices and confusion.

"One is certainly this stage appropriate review as I call it. Right. Like they do it both ways."

The speaker identifies the mistake of not aligning the review process with the appropriate stage of product development, either by rushing to scalability or lacking a clear agenda.

• The shift to remote meetings has led to experiments with different formats, such as having a core group of participants and allowing observers to learn.
• Observers can gain peripheral vision into the product review process, which can be beneficial for junior members.
• Remote meetings have normalized expectations around product reviews.

"We tried a couple of experiments here. One is have two classes of folks."

This quote discusses the adaptation of product review meetings to accommodate remote participation, creating opportunities for learning and observation despite the challenges of not being physically present.

• Teams should always connect their work back to the problem they are solving.
• Presentations should start with a clear statement of the problem being addressed.
• Leaders should consistently remind the team of the problem to ensure that the product features align with the intended goals.
• Clarity on the type of feedback desired in a product review is crucial to avoid wasting time on irrelevant details.

"A lot of times you come to a product review, you're like, oh, check out these great new designs."

The speaker points out a common issue where teams get caught up in the aesthetics of a design rather than focusing on whether it solves the intended problem.

• The number of participants in a product review should be kept to the minimum necessary.
• Only those directly involved or who need to hear feedback should be in the room.
• With remote work, there may be more flexibility for people to join without actively participating.
• The goal is to keep meetings efficient by limiting the number of participants.

"Generally, the rule of thumb is it should be as small as possible."

The speaker suggests that smaller meetings are more effective and that the participants should be carefully chosen based on their involvement in the project.

• Product reviews should have a regular cadence, with different types of reviews for different stages of product development.
• Reviews serve two customers: the team seeking feedback and the leadership team evaluating progress and team dynamics.
• Product reviews can reveal issues such as slow velocity or lack of progress, which require intervention.

"So I'll first give you the what and then I'll give you the why."

The speaker outlines the structure of product reviews, explaining their frequency, format, and the different types of reviews conducted, as well as the dual purpose they serve for both the team and leadership.

• Indicators of ineffective reviews include slow progress, lack of clear direction, and teams that are not making decisions or moving forward.
• When reviews are not working well, it's essential to identify the causes and take action to address them.

"Signs that it's not working well, slow velocity. If the team is just treading water and spinning their wheels."

This quote highlights the symptoms of a product review process that isn't functioning as intended, specifically pointing out slow progress as a key sign of ineffectiveness.

• Teams often mistakenly convey what they think leadership wants to hear instead of their strong point of view.
• It's important for teams to have and express a strong conviction in their ideas.
• Leadership should push teams on their reasoning and standpoints to ensure they have a solid understanding and belief in their approach.
• Conviction from teams is crucial, especially when they can defend their perspective against leadership's initial doubts.

"Another sign is if the team doesn't convey strong conviction. Like oftentimes, teams make the mistake of assuming that they just need to share what they think leadership wants to hear."

This quote emphasizes the problem of teams not showing true conviction in their ideas, instead trying to please leadership, which can hinder progress.

"What I care about is the team has a very strong point of view and it's very easy to kind of smell test whether they do or they don't."

Scott Belsky expresses the importance of a team having and presenting a strong point of view, which is critical for effective product development.

"The way you really tease that out is really pushing the team. Why do you feel this way? Why is this the hypothesis?"

This quote highlights the method of challenging the team to ensure they have thought through their ideas thoroughly and are not just trying to appease leadership.

• Leaders must differentiate between providing feedback and giving directives.
• Leaders need to communicate clearly that most of their input is meant as feedback, not an order.
• Regular communication is necessary to remind teams of the purpose of feedback, especially in a growing company.
• Leaders should be careful not to give the impression of directives when they are merely giving feedback.

"I tend to be very passionate and say things with conviction. With 90% of the time I'm saying things with conviction, not because I'm telling the team to do it, I'm just teasing out an idea."

Scott Belsky talks about the personal challenge of ensuring his passionate delivery is not misconstrued as a directive when it's intended as feedback.

"Just repeating to the point of utter repetition and seemingly like, I feel like I'm a broken record here. But just to be clear, this is me not conveying a decision, this is me just asking the question."

This quote shows the lengths to which leadership must go to make sure teams understand the nature of their feedback.

• Product managers and teams must organize and prioritize amidst a barrage of feedback.
• Not all feedback should be acted upon; some should be considered later, and some ignored.
• It's crucial to focus on what's important for the customer and the business.
• Leadership should provide clear and concise feedback post-product reviews to avoid overwhelming teams.

"Your job as a team and as a product manager is to take all that input in and make organization and prioritization out of chaos."

Scott Belsky describes the critical role of product managers in managing feedback and prioritizing tasks effectively.

"We have a huge responsibility when we do product reviews to make sure we are giving clear and not distracting feedback."

This quote points out the responsibility of leadership to give feedback that is helpful and not confusing or contradictory.

• Product reviews are a platform for generating valuable feedback and ideas.
• Having an abundance of feedback is a benefit that should be appreciated.
• Clear communication from leadership is essential for effective product review outcomes.
• Properly handling feedback is a sign of a mature product team and leadership.

"We work on a product that we could launch the most trivial feature and 50% of our customers will be outraged and 50% of our customers will be like, this is amazing."

Scott Belsky illustrates the challenge of dealing with diverse feedback on a widely used product like Twitter.

"After we have the live product review, we have a debrief as a leadership team, 15 minutes debrief where we try and amongst ourselves say, all right, we just talked about a lot of stuff. What's the most useful feedback we can clarify and write down?"

The quote describes a practical approach to distilling feedback into actionable items for the team post-product review.

• Speaking to several world-leading experts can provide deep insights into a subject.
• Finding commonalities in expert approaches can be a powerful learning tool.
• Continuous improvement and adaptation are key to maintaining effective product review processes.

"I've often read that the best way to learn about a subject is to speak to five or six world leading experts and really unpack how they think about it and find the commonalities in how they do it."

Harry Stebings suggests that learning from multiple experts can yield a comprehensive understanding of a subject.

• UXCam is an analytics platform designed for mobile product teams.
• It offers tools like session replay, heat maps, funnels, crash analytics, and captures user interactions.
• UXCam provides insights into user behavior to help improve app experiences.

"Designed specifically for mobile product teams, UX Cam helps build better apps by making user motivations and frustrations loud and clear."

Harry Stebings promotes UXCam as a valuable tool for mobile product teams to understand user needs and improve app functionality.