VC's Framework for Evaluating an AI Startup's Tech Stack. | VC Jobs
Coinbase's Decision Making Framework & Two Law of Startup Physics- "Capital compounds both positive and negative formulas."
👋 Hey Sahil here! Welcome to this bi-weekly venture curator newsletter. Each week, I tackle questions about building products, startups, growth, and venture capital! In today’s newsletter, we dive into -
Deep Dive: VC's Framework for Evaluating an AI Startup's Tech Stack.
Quick Dive:
Coinbase’s Decision-Making Framework.
The Two Laws Of Startup Physics By Eric Paley.
Hierarchy of User Engagement By Sarah Tavel.
The CUP Method - A Prioritization Framework From Idea To PMF.
Major News: Musk's X.AI Building $9B AI Supercomputer In Memphis, NVIDIA Becomes $3.012 Trillion Value Company, NVIDIA's $450 Million Investment In AI Startup & Cisco Launched $1 Billion AI Startup Fund.
Best Tweet Of This Week On Startups, VC & AI.
VC Jobs & Internships: From Scout to Partner.
IN PARTNERSHIP WITH DRAPER UNIVERSITY
Build your Web3 startup on Stellar & receive $20K in grant funding.
Draper University has partnered with the Stellar Development Foundation to grow 30 Web3 startups on Stellar.
We are the #1 early-stage backers of decentralization and Web3. Our alumni include Lemon Cash, Highstreet, QTUM and more.
Join our 3-week Silicon Valley residency, the Stellar Astro Hacker House. Learn how we support Web3 startups to fundraise, hire elite talent, and grow their user base to the moon.
PARTNERSHIP WITH US
Want to get your brand in front of thousands of founders, investors, executives, and startup operators? For details on our sponsorships, fill up this quick form and we’ll get in touch.
TODAY’S DEEP DIVE
VC's Framework for Evaluating an AI Startup's Tech Stack
From fraud detection to agricultural crop monitoring, a new wave of tech startups has emerged, all armed with the conviction that their use of AI will address the challenges presented by the modern world.
However, as the AI landscape matures, a growing concern comes to light: The heart of many AI companies, their models, are rapidly becoming commodities. A noticeable lack of substantial differentiation among these models is beginning to raise questions about the sustainability of their competitive advantage.
Instead, while AI models continue to be pivotal components of these companies, a paradigm shift is underway. The true value proposition of AI companies now lies not just within the models, but also predominantly in the underpinning datasets. It is the quality, breadth, and depth of these datasets that enable models to outshine their competitors.
Many founders want to understand the framework that VCs use to evaluate the AI startup tech stack. I am sharing a framework that a General Partner shared with me.
From inconsistent datasets to noisy inputs, what could go wrong?
Before jumping into the frameworks, let’s first assess the basic factors that come into play when assessing data quality. And, crucially, what could go wrong if the data’s not up to scratch?
Relevance
First, let’s consider the datasets’ relevance. Data must intricately align with the problem that an AI model is trying to solve. For instance, an AI model developed to predict housing prices necessitates data encompassing economic indicators, interest rates, real income, and demographic shifts.
Similarly, in the context of drug discovery, experimental data must exhibit the highest possible predictiveness for the effects on patients, requiring expert thought about the most relevant assays, cell lines, model organisms, and more.
Accuracy
Second, the data must be accurate. Even a small amount of inaccurate data can have a significant impact on the performance of an AI model. This is especially poignant in medical diagnoses, where a small error in the data could lead to a misdiagnosis and potentially affect lives.
Coverage
Third, coverage of data is also essential. If the data is missing important information, then the AI model will not be able to learn as effectively. For example, if an AI model is being used to translate a particular language, the data must include a variety of different dialects.
For language models, this is referred to as a “low resource” versus “high resource” language dataset. This also requires having a complete understanding of the confounding factors that affect the outcome, which typically requires the collection of metadata.
Bias
Finally, data bias also warrants rigorous consideration. Data should be captured in an unbiased way to avoid human prejudice or bias in the model. For instance, image recognition data should minimize stereotypes. In drug discovery, datasets should encompass both successful and unsuccessful molecules to avoid skewed outcomes. In both cases, the data would be considered biased and likely lose its ability to make novel predictions.
The repercussions of subpar data shouldn’t be underestimated. At best, they result in a model that underperforms, and at worst, they render the model entirely ineffective. This can lead to financial losses, missed opportunities, and even physical harm.
Similarly, if the data is biased, the models will produce biased results, which can foster discrimination and unjust practices. This has been a particular concern with large language models, which have come under recent scrutiny for perpetuating stereotypes.
Compromised data quality also has the potential to erode effective decision-making, which can ultimately result in poor business performance.
Framework 1: Tech stack pyramid for data generation
To avoid investment in ineffectual AI startups, there is a need to first evaluate the processes behind the data. Picturing a company’s tech stack as a pyramid is a good place to start, where the foundational tiers tend to have the biggest impact on the predictive outcome. Without this solid base, even the best data analysis and machine learning models face significant constraints.
Here are some basic questions that a VC might initially ask to figure out if a startup’s data generation process can create usable results for AI:
Is data capture automated to enable scale-up?
Is the data stored in secure cloud environments with automated backups?
How is access to infrastructure and relevant compute resources managed and guaranteed?
Are data processing pipelines fully automated, with rigorous data quality controls implemented to limit pollution from contaminated data points?
Is the data readily accessible across the company to empower ML model-building and data-driven decisions?
How is data governance implemented?
Is there a data management strategy in place?
Are data and ML model versions tracked and accessible, ensuring ML models are always working on the latest data version?
Receiving robust answers to these questions can help determine a company’s grasp of the underpinning principles of its data pipelines. This understanding, in turn, will help gauge the quality of the model’s output.
Framework 2: The five V’s of data quality
Once a company’s tech stack has been deemed suitable for AI, there is also a need to carefully consider the quality of the resulting data being used to train its models. A common framework used to capture the classification of data quality is the five V’s of data quality. They represent five key dimensions of data quality that VCs should consider when evaluating AI startups:
Veracity: The data must be accurate and truthful.
Variety: The data must be diverse and representative of the real world.
Volume: The data must be large enough to train the AI model effectively.
Velocity: The data must be updated frequently to reflect changes in the world.
Value: The data must be useful for the AI model to learn from.
Here are some introductory questions to help evaluate a company’s data for the five V’s:
Does the startup have a good hypothesis about which data they need to create to build a differentiated capability or useful model?
What data do they collect?
Do they also collect any relevant metadata?
How do they ensure the correctness and consistency of the data they collect?
How does the startup plan to deal with data bias?
Do they collect multiple examples for the same question or experiment?
How useful is this data for the product they are building?
What’s the rationale behind collecting this data?
Do they have evidence that their predictions improve by collecting and using this data? If yes, how does the data amount correlate with prediction improvement?
How easy is it for a competitor to collect the same data? How long would it take and how much would it cost for them to do so?
Specifically for a biotech, how well does the proxy they are predicting correlate with a clinically relevant endpoint? Is there evidence for this?
What is the startup’s plan for ensuring the quality of its data over time?
How does the startup plan to protect its data from unauthorized access?
How does the startup plan to comply with data privacy regulations?
By carefully considering the five V’s of data quality, VCs can make sure they are investing in AI startups that have the data they need to succeed. If the startup can answer the above questions convincingly and their data scores highly in the five dimensions, it is a good sign that they are serious about data quality and are properly equipped to apply their AI models.
Finally, VCs should assess the startup’s commitment to data security. This includes things like their data governance policies, their data quality assurance procedures, and their data breach response plans.
Interrogate the hype to find the winners
Amid the resounding buzz surrounding AI in recent months, the allure of substantial investments has attracted startup founders willing to exaggerate their infrastructure and inflate capabilities in the search for capital.
Successful VCs are asking the right questions to interrogate these companies thoroughly and filtering out the potential winners built on a solid foundation from those with a hollow shell that are ultimately destined to fail.
QUICK DIVES
1. Coinbase’s Decision Making Framework.
Low-risk decisions should be made pretty quickly. For high-risk decisions, a decision-making framework can be helpful.
I’ve seen a lot of decision-making frameworks. But I like this one from Brian Armstrong, the founder of Coinbase because it works both for quick 15-minute live meetings and over multi-week strategic decisions.
This framework can be used to decide:
Whether to hire a candidate
What to prioritize next in a product roadmap
Whether to buy or sell a company
A new name for a product or team
And more…
Here’s the template to implement this framework and here’s how to use it.
You might think this template looks overly simplistic, but what I like about it is that it’s just meant to record the context around the decision and provide a clear voting framework.
It forces the actual decision-making process to happen through live conversation. And no matter what you’ll end up with a clear, numerical score for the option that you collectively favour.
2. The Two Laws Of Startup Physics By Eric Paley
With 15 years of experience as a venture capitalist under his belt, Eric Paley, a Partner at Founder Collective, has identified two fundamental and inviolable "laws of startup physics" that all founders must internalize to build a successful, sustainable company.
Capital compounds both positive and negative formulas. All positive formulas compound at diminishing rates of return.
Capital compounds positive and negative formulas
We are fond of saying, “Capital has no insights.” It doesn’t have the answers to your problems and can only fund two things for a startup:
A) Experimentation - which rarely is expensive.
B) Scale - which compounds whatever is already happening at the startup, whether compounding toward greater intrinsic value or compounding the startup’s value to zero.
Capital can scale intrinsic value rapidly if you have an engine that allows it to turn $1 into $5 of value. If it has an engine that turns $5 into $1 (or even $1 into $0.99), capital will ultimately compound the negative value formula to zero.
All positive systems compound at diminishing rates of return
Sooner or later, the return on each dollar invested will shift to negative. If you’re unaware of the point at which compounding goes into the red, you start compounding negative value.
Paradoxically, the desire for growth often prematurely drives startups into negative compounding, ultimately leading to failure.
In my experience, startups that internalize these rules have done incredibly well.
Failure to respect the rules of startup physics - capital compounds good and bad, and all positive compounding eventually diminishes - has been the cause of just about every startup failure I’ve seen that can’t, simplistically, be written off as “no one wanted the product.”
Other things to consider from this tweet
Startups should experiment with small amounts of capital until they find a formula that generates positive intrinsic value on each dollar invested, before attempting to scale.
Many startups skip this crucial experimentation phase and attempt to scale prematurely, leading to compounding losses over time.
As startups attempt to grow, various levers like adding features, pursuing low-quality customers, and scaling sales/marketing become subject to diminishing rates of return.
Venture capitalists often prioritize short-term growth over long-term intrinsic value creation, incentivizing startups to spend poorly on vanity metrics.
Founders should resist this pressure and focus on low-cost experiments until they discover how to create positive value formulas that justify increased spending.
Discipline is critical – when something stops working, startups must immediately stop scaling that area.
Successful approaches include not skipping the experimentation stage, keeping experiments small, running only a few at a time, and asking better questions about ROI versus growth.
3. Hierarchy of Engagement By Sarah Tavel.
Sarah Tavel, General Partner at Benchmark, developed a way to understand user engagement called the Hierarchy of Engagement.
Level 1: Develop a “core action” you want users to take. Most often the core action is the action most correlated with retention. Facebook was growing your friend list.
Level 2: Enable network effects. Make sure that more people = more value on the network. Using our FB example, the more friends you have on FB the harder it is to leave.
Level 3: Flywheels and more flywheels. Once you have engaged users, make sure the time they’re engaged creates more engagement from other users — create an infinite engagement glitch in the matrix. For FB, having more friends on the platform meant you would invite more friends to join the platform. (Link)
4. The CUP Method - A Prioritization Framework From Idea To PMF.
The CUP method is a prioritization framework that helps you go from idea to product-market fit. It’s most useful in the 0 to 1 stage before you’d consider something like “jobs to be done.”
The CUP Method is designed:
To help entrepreneurs address user pains, not build vanity features.
To help entrepreneurs build a unique solution, not build another copycat.
To help entrepreneurs set and achieve short-term goals in service of reaching their ultimate vision, not get lost dreaming about what happens 3,5, or 10 years from now.
It’s broken down into three steps:
Current Primary Goal → Set a single goal you can reach in the short term (think a few weeks to a few months).
Unique Value Proposition → What makes you different from all the alternatives?
Pain Points → What major problems are your users experiencing?
A must-read framework for founders.
Join 25000+ Founders, Investors and Startup Enthusiasts Getting Tactics To Build, Learn and Implement About Startups and Venture Capital.
THIS WEEK’S NEWS RECAP
Major News In VC, Startup Funding & Tech
Musk's AI startup X.AI Corp. plans to build a massive AI supercomputer in Memphis with an estimated cost of $9 Billion. More Here
Nvidia (NVDA.O) became the world's second most valuable company at $3.012 trillion, surpassing Apple Inc. (AAPL.O) at $3.003 trillion. More Here
Cisco has launched a $1 billion fund to invest in AI startups and made a $200 Million Investment in Mistral AI, Cohere and scale AI. More Here
Tesla is likely to spend $3-4 billion on Nvidia hardware for AI training this year. More Here
Despite raising $100 million, LoanSnap, an AI mortgage startup, is facing lawsuits from creditors and has been evicted from its headquarters. More Here
→ Get the most important startup funding, venture capital & tech news. Join 20,000+ early adopters staying ahead of the curve, for free. Subscribed to Venture Daily Digest Newsletter.
TWEET OF THIS WEEK
Best Tweet I Saw This Week
“If someone on the team claims that the entire app needs to be polished and snappy, I would push back unless you’re entering into a very competitive market with existing substitutes.
Even then, there are probably areas of uncertainty that should be eliminated first that don’t require the building of a Porsche as V1.” - Nikita Bier.
How Can I Help You?
Build Your Pitch Deck: We write, design and model your pitch deck into a storyteller book within 4-5 days.
Get Your MVP In 15 Days: Have an Idea? Turn your idea into Reality. Move fast before your competitors.
TODAY’S JOB OPPORTUNITIES
Venture Capital Jobs & Internships
Director, Investments - Stand together venture lab | USA - Apply Here
Head of Legal and Compliance - Signature Venture | Germany - Apply Here
Investor Relations and MarCom - E4E Africa | South Africa - Apply Here
Associate Advantage founders | India - Apply Here
Venture Capital Internship - 14peak capital | Swiss - Apply Here
Executive Assistant/ Office Manager - SE Venture - Apply Here
Associate - Yournest VC | India - Apply Here
Senior Associate - Yournest VC | India - Apply Here
Head of Startup Ops - Stand together | USA - Apply Here
Visiting Analyst - Seedcamp | UK - Apply Here
Investment Analyst - We fund | Germany - Apply Here
Senior Manager - Entegris Venture | USA - Apply Here
Fellowship - Mighty Capital | Remote - Apply Here
Chief of Staff - Pillar VC | USA - Apply Here
Senior Associate - GEHA Investing | USA - Apply Here
🧐 Some Tips To Break Into VC:
What does the life of a VC look like?
“The life of a VC: where you spend your days trying to make money, but mostly just trying to make sense of the absurdity that is the startup landscape. It's like trying to find a needle in a haystack, but the haystack is on fire and the needle is a unicorn. And if you're lucky, you might even get to wear a plaid shirt and say something soulfully profound like, "Get to know an expert in your chosen field—and ignore everything he has to say..."
There are plenty of articles out there about what a 'day in the life' of a VC looks like and also about what struggles VCs have to sometimes go through. Here's a couple that I found interesting:
What is the day in life like for a Venture Capitalist? - Quora (and part 2)
This tweet and the original quoted tweet give a funny & fairly realistic picture of it too
...and many more
Looking To Break Into Venture Capital?
Join our VC Crafter community and get access to VC learning resources, daily VC job updates, daily discussion sessions, 1:1 call access, worked as a scout for various VC firms, a CV/Interview preparation guide and more. Don’t miss this opportunity….
This is good.
Thanks for sharing the Sahil.
The Brian Armstrong framework is a gem.