Lenny'sLens
← All topics

Experimentation

4 claims5 moments3 on the cutting room floor
Product ManagementLeadershipStartup BuildingGrowthCareerAi ToolsGo To MarketMetricsProductivityProduct DesignEngineeringB2bAi ProductHiringRetentionFeedbackManagementSalesVibe CodingPrototypingProduct StrategyEvalsCareer GrowthMarketplaceCompensationProduct Market FitHealthStartup StrategyCommunicationGrowth StrategyPricingNamingCareer DevelopmentStartupsDecision MakingOrganizational DesignMarketplacesCompany CultureFocusProduct Led GrowthInnovationProduct DevelopmentStorytellingCompetitive AdvantagePrioritizationStrategyFrameworksB2b GrowthLearningExperimentationUser ExperienceExecutionPaid MarketingSelf AwarenessAiProduct BuildingGamificationAngel InvestingUser ResearchDistributionDesignViralityAirbnbTeam ManagementWord Of MouthGrowth TeamsProduct LeadershipCareer AdviceInterviewingProduct QualityFundraisingCompetitive StrategyCreator EconomyContent StrategyEarly StageTime ManagementConversion OptimizationCareer Planning

Lenny's Written Position

Early-stage startups should not try to be data-driven because they simply do not have enough data for statistical significance; a 10% improvement at 20% conversion requires over 5,000 users.

Synthesisrecommendation2 connections
1 extend1 contradict

You should skip running an A/B experiment when it will take too long to get actionable results, when the downside risk is low and setup effort is high, or when the change only has long-term effects.

Originalframework0 connections

SEO experiments must be bucketed at the page level rather than the user level because Google recrawls pages as different 'users' each time, and the only metric to measure is incoming organic traffic.

Originalframework0 connections

Title tag experiments led to a 15-20% aggregate increase in organic traffic at Airbnb, making them the most impactful type of SEO experiment to run.

Originalobservation0 connections

Podcast Moments

Sam Schillace00:00:00
“If that's messing around and playing around with cool ideas, do the hell out of that.”

How to be more innovative | Sam Schillace (Microsoft deputy CTO, creator of Google Docs) · Sam Schillace

Itamar Gilad00:00:00
“You fake it, you do a fake door test, you do a smoke test, Wizard of Oz tests. We used a lot of those in the tabbed inbox by the way, one of the first early versions was actually we showed the tabbed inbox working to people. But it wasn't really Gmail, it was just a facade of HTML.”

Becoming evidence-guided | Itamar Gilad (Gmail, YouTube, Microsoft) · Itamar Gilad

Ronny Kohavi00:00:00
“I'm very clear that I'm a big fan of test everything, which is any code change that you make, any feature that you introduce has to be in some experiment. Because again, I've observed this sort of surprising result that even small bug fixes, even small changes can sometimes have surprising, unexpected impact.”

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon) · Ronny Kohavi

Ronny Kohavi00:00:22
“You have to allocate sometimes to these high risk, high reward ideas. We're going to try something that's most likely to fail. But if it does win, it's going to be a home run. And you have to be ready to understand and agree that most will fail.”

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon) · Ronny Kohavi

Yuriy Timen00:00:00
“The only thing that's worse than a channel or a tactic that you tried not working. The only thing that's worse now is when you didn't give it the appropriate shot, right? And you prematurely were erroneously concluded that it doesn't work.”

How to grow a subscription business | Yuriy Timen (Grammarly, Canva, Airtable) · Yuriy Timen

Cutting Room Floor

Guest insights on this topic that Lenny hasn't (yet) written about in his newsletters. Potential material for future posts.

Sam SchillaceUnsynthesized
“If that's messing around and playing around with cool ideas, do the hell out of that.”

How to be more innovative | Sam Schillace (Microsoft deputy CTO, creator of Google Docs) · Sam Schillace

Ronny KohaviUnsynthesized
“You have to allocate sometimes to these high risk, high reward ideas. We're going to try something that's most likely to fail. But if it does win, it's going to be a home run. And you have to be ready to understand and agree that most will fail.”

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon) · Ronny Kohavi

Yuriy TimenUnsynthesized
“The only thing that's worse than a channel or a tactic that you tried not working. The only thing that's worse now is when you didn't give it the appropriate shot, right? And you prematurely were erroneously concluded that it doesn't work.”

How to grow a subscription business | Yuriy Timen (Grammarly, Canva, Airtable) · Yuriy Timen