20VC Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher

In this episode of 20vc, Harry Stebbings interviews Richard Socher, a pioneering AI researcher and the founder and CEO of U.com, discussing the impact of AI on job automation and the economy. Socher, formerly the chief scientist at Salesforce and CEO of Metamind, highlights the uneven distribution of AI's benefits and the challenge of jobs without digital data to automate. He also addresses the complexities of AI development, the importance of retrieval augmentation for language models, and the potential overestimation of AGI's near-future impact. Additionally, the conversation touches on the role of AI in society, the balance between innovation and distribution, and the future of U.com as a search platform. Promotional segments interspersed throughout the podcast feature Coda's AI-powered work assistant, Navan's travel credit rewards, and Public.com's treasury accounts.

• The future has already arrived but is not distributed equally.
• Automation is more likely in jobs with abundant data.
• Jobs without data collection are becoming the new bottleneck.
• Physical tasks are getting more expensive, potentially slowing GDP growth.
• AI's impact on GDP may be less than anticipated due to these bottlenecks.

"The future is already here, it's just not equally distributed and I think it will meaningfully change so many jobs. The more data there is about a job, the more that job is likely to be automated."

This quote highlights the disparity in the adoption of technology across different jobs and sectors, emphasizing the role of data availability in automation.

"What that means is that the tasks that are physical are getting more and more expensive, and they're going to become the new bottleneck, and then they're going to slow down overall progress so that the GDP can't like 100 x because of AI."

The speaker is explaining that physical tasks, which are harder to automate, are becoming costlier and could limit the overall economic growth expected from AI advancements.

• Richard Socher is recognized for his contributions to neural networks and NLP.
• He brought word vectors, contextual vectors, and prompt engineering into NLP.
• Richard is the founder and CEO of U.com, and has a significant number of citations.
• His past roles include Chief Scientist and EVP at Salesforce and CEO and CTO of Metamind.

"I'm so excited to welcome Richard Socher, founder and CEO of U.com."

Harry Stebbings introduces Richard Socher, highlighting his achievements and contributions to the AI field.

• Coda offers an AI-powered work assistant to streamline tasks.
• The assistant helps product teams with customer feedback and PRDs, sales teams with data integration and lead scores, and marketing teams with user insights and press releases.
• Coda's goal is to transform the way teams collaborate and make progress.

"Well, Coda is here with their new AI-powered work assistant that helps you and your team not just finish tasks, but make progress."

Harry Stebbings discusses the benefits of using Coda's AI-powered work assistant for improving team productivity and task management.

• Navan offers cost savings on travel and expense management.
• Employees can earn personal travel credit for saving company money.
• Navan is confident in their service, offering a $250 travel credit for a demo.

"Navan rewards your employees with personal travel credit every time they save their company money when booking business travel under company policy."

The speaker promotes Navan's unique approach to incentivizing cost-effective travel booking by rewarding employees, suggesting potential cost savings for companies.

• Public.com offers a simple way to earn yield on cash through Treasury bills.
• Treasury accounts with Public.com allow for automatic rollover and compounding yield.
• There is flexibility like a bank account, with the full backing of the US government.

"As of the 29 June, you can get 5.5% yield on your cash with 26 week treasury bills."

The speaker informs listeners about the opportunity to earn a high yield on cash using Public.com's service for purchasing Treasury bills.

• Richard started in linguistic computer science at Leipzig University in 2003.
• He switched to computer vision due to a lack of math in early NLP.
• During his PhD, he applied deep learning and neural networks to NLP.
• Richard has worked towards a single model for all of NLP tasks.

"Boy, that goes back to 2003 is when I started linguistic computer science at Leipzig University."

Richard Socher gives a brief history of his journey into the ML and NLP field, dating back to his university days.

• AI is experiencing exponential improvements.
• There are inflated expectations regarding AI's capabilities.
• Not all tasks are suited for AI, such as chatbots for everything.
• The level of AI capabilities is rising, but there are waves of inflated expectations.

"We are at the beginning of an exponential improvement in a lot of different capabilities. At the same time, some people think the exponential will just keep on going."

Richard Socher discusses the current state of AI, noting both rapid advancements and the tendency for people to overestimate the continuous pace of breakthroughs.

• The idea of a single pretrained model for all NLP tasks is relatively new.
• Previously, each NLP task had its own model.
• Richard advocates for a single, continually improving model.
• A single model is more efficient and beneficial for collaborative advancement.

"So there is an existence proof for a single model for all of NLP."

Richard Socher argues for the feasibility of a single model capable of handling all NLP tasks, drawing an analogy to the human brain's versatility.

• Large models are necessary for a single model approach to NLP tasks.
• Small models cannot handle the complexity and variety of tasks.
• The size of the model and the amount of training data are both crucial.

"You just cannot train a single model for all of these different tasks with a small model."

Richard Socher emphasizes the importance of model size in creating a versatile, single model for various NLP tasks.

• Language models learn world knowledge through extensive training across various internet sources.
• Geography of the northeastern United States is used as an example of specific knowledge that can be learned.
• The process requires a large number of parameters to capture and learn the breadth of knowledge available on the internet.

"You need to learn everything about the geography of the northeastern United States. And now if you do this billions of times across the Internet, on all the chemistry articles and biology articles and so on, you learn world knowledge."

The quote emphasizes the scope and scale of learning required for a language model to develop a comprehensive understanding of world knowledge, using geography as a specific example.

• There is a debate about whether access to data is democratized or if incumbents have an advantage.
• Unsupervised data is accessible, but private databases hold significant data that is not publicly available.
• Incumbents with access to this data can train AI more effectively for customer service and other specific needs.
• Large language models and foundational models have made it easier for small startups to build solutions with a general understanding of natural language.

"But there are still a lot of data sets out there that are not out there. They're actually stored in private databases."

This quote highlights the dichotomy between publicly available data and the vast amounts of data stored in private databases, which can be a competitive advantage for incumbents.

• There is skepticism in the venture community about the value of AI startups that build on foundational models.
• The complexity of building a successful company is often underestimated.
• Instagram is used as an example of a company with a moat that is not based on AI or backend technology.
• Large language models require additional components, such as a search backend, to provide accurate and up-to-date information.

"So there's a lot. And then there are also areas where the default large language model will not do as well."

The quote suggests that while foundational models are powerful, the real value and differentiation come from additional layers and components that companies build on top of these models.

• Hallucinations in language models are seen as both a feature and a problem, depending on the context.
• The balance of what models can and cannot say needs careful consideration.
• Ethical concerns about content generation by AI are analogous to concerns about creative works by human authors.

"But hallucinations aren't a feature and are a problem for search engine."

This quote acknowledges that while creative deviations can be entertaining, accuracy is crucial when it comes to providing factual information in a search engine context.

• Language models are updated frequently, but the general architecture is expected to remain in use.
• The constant evolution of models is part of the AI landscape, and companies must adapt to keep up.

"We're going to update all these models like we update our model every other week."

The quote indicates the rapid pace at which language models are updated, reflecting the dynamic nature of AI development.

• The ability to transition between different models is seen as a potential competitive advantage for companies.
• The importance of how models are used, tuned, and integrated with other systems is emphasized over the specific model in use.

"It won't matter that much which lm you use, but it matters what you do with it, how you tune it, what kind of training data you add onto it to fine tune it."

This quote suggests that the strategic use and customization of language models are more critical than the choice of a specific model.

• OpenAI is recognized as a current leader in foundational models.
• Predictions are made about the emergence of open-source models equivalent to GPT-4.
• Open-source models are expected to become more prevalent and commoditized, with foundational companies like Anthropic trying to catch up.

"I predicted that we'll have a GBD four equivalent model before the end of the year. That's open source."

The quote reflects an expectation that open-source models will continue to improve and become more accessible, challenging the dominance of proprietary models.

• There is a debate about whether open or closed ecosystems will prevail.
• The lack of alignment in open ecosystems could be a disadvantage compared to closed ecosystems with clear goals and timelines.

"Due on the show from contextual said that actually it was the lack of alignment between open ecosystems which would lead to closed winning."

The quote presents an argument that the organizational structure and focus of closed ecosystems might give them an edge over open ecosystems in the AI industry.

• Open models like llama two have shown significant progress over past models.
• The dream is to have a single model that the whole world can collaborate on, similar to Wikipedia.
• Open models require coordination, trust-building, and overcoming bureaucratic hurdles.

"But that would be my dream, is that we actually have a single model. And just like Wikipedia, the whole world collaborates and adds to it."

This quote underscores the aspiration for a collaborative AI model akin to Wikipedia's open editing system, emphasizing the potential for global enhancement and accessibility.

• Coordination and funding are major challenges in developing open models.
• Governments can help but face difficulties in funding uncertain research due to potential taxpayer complaints and bureaucratic necessities.
• Despite challenges, governments are encouraged to fund open science.

"The problem is it's very hard for governments to fund uncertain research and science projects because when it doesn't work out, some taxpayers will for sure complain."

This quote highlights the difficulty governments face in funding research with uncertain outcomes due to the risk of public backlash and the need for bureaucratic oversight.

• Search is a high-impact area of NLP technology due to its massive market size and daily use in learning.
• There is potential for significant improvement over the current state of search technology.

"But more importantly, intuitively, we ask search engines questions every day to learn something."

This quote emphasizes the daily relevance and significant impact of search engines on learning, suggesting the importance of advancements in this area of NLP technology.

• Search engines need to evolve into open platforms where content providers can contribute and monetize their offerings.
• Subscription models could be integrated within the search engine.
• The uptake of open platforms has been slow due to a lack of user base.

"We need to have that search engine that we've built be an open platform where you can actually contribute your apps to and your content to."

This quote suggests a vision for future search engines, where content providers can directly contribute and potentially profit from their contributions, fostering a more collaborative ecosystem.

• The battle between startups gaining distribution and incumbents acquiring innovation is ongoing.
• Startups face challenges in distribution and must continually seek partnerships and clever strategies to compete.
• Incumbents have the advantage of established user bases but may lack agility.

"The truth is distribution won't be ever fully solved and it's a constant uphill battle."

This quote acknowledges the persistent challenge startups face in achieving distribution and the need for continuous innovation and strategic partnerships to compete with established players.

• The next wave of AI is expected to create value for both incumbents and startups.
• Incumbents like Salesforce have been quick to incorporate AI into their products.
• Startups have prompted incumbents to innovate more actively in the last few years.

"I think it'll be a mix."

This quote suggests that both startups and established companies will derive value from advancements in AI, with neither group having a definitive advantage.

• People often extrapolate current progress to predict future advancements, leading to overoptimism.
• Progress in technology is not always linear or exponential, as seen in the history of human flight.
• In AI, some areas like image generation are reaching a plateau, challenging the notion of continuous exponential growth.

"But progress is not as linear or as exponential as a lot of people think."

This quote cautions against assuming that technological progress will continue at a consistent rate, using historical examples to illustrate that advancements can slow or plateau.

• Technological progress is influenced by science, nature, and human factors such as skill and funding.
• AI's progress may be overestimated, with certain domains already nearing their potential.
• Superhuman capabilities in AI are limited by human constructs and comprehension.

"You think, like, we have this keep having exponential growth. But I'll give you an example where we already see the flattening out of that exponential growth into an s curve, and that is an image generation."

This quote points out that the perceived exponential growth in AI may be overstated, with some areas like image generation already approaching their limits, signaling a possible plateau in progress.

• Superhuman AI is constrained by human comprehension and consumption rates.
• AI can produce content quickly and in large quantities, but humans can only consume at a certain pace.

It doesn't make sense to have superhuman language because humans couldn't understand it anymore to some degree.

The quote emphasizes that AI advancements in language must remain comprehensible to humans, as surpassing human understanding would render the AI's abilities ineffective for practical use.

• There is a significant gap between the advancements in technology and public awareness of these technologies.
• Most people outside tech-centric areas are unaware of cutting-edge developments like mid-journey image generation.

It's weird that we've reached plateauing technological development with zero awareness of it in 6.9 billion people.

Harry Stebbings points out the odd situation where technological advancements have reached a high level, yet the vast majority of the global population remains unaware of these developments.

• The future's advancements are not evenly distributed among populations.
• The quote by William Gibson is invoked to describe this uneven distribution.

The future is already here. It's just not equally distributed.

This quote, cited by the speaker, illustrates that while the future with all its advancements exists, it is not accessible or known to everyone equally.

• AI will significantly change jobs that are digital and data-rich, leading to automation and efficiency improvements.
• Jobs without data collection or in unconstrained environments will become more expensive and could slow down overall economic progress.
• Physical tasks are becoming the new bottlenecks in the economy due to AI advancements.

The more digital they are, the more data there is about a job, the more that job is likely to going to be automated and improved massively in its efficiency.

The speaker suggests that jobs with more digital presence and data availability are more likely to be automated, improving their efficiency significantly.

• Industrial and technological revolutions have historically caused job displacement, often over decades.
• The current pace of AI-induced changes might lead to a much faster transition period, raising concerns about job displacement.

Are you concerned by the job displacement question and just the awareness that when you look at industrial revolutions and technological revolutions, there's always decade long, actually transition periods, whereas here it seems like the transition period is years.

Harry Stebbings is asking about the concern regarding the rapid pace of job displacement due to AI compared to slower transitions in past industrial revolutions.

• There is a need for empathy towards individuals whose jobs are changed by AI.
• Social systems should support people in learning new skills and adapting to new technologies.
• While some people resist technological changes, most appreciate the benefits once they are accustomed to the new status quo.

And I feel empathy with those people. And it would be nice to have social systems that catch some of those people and help them learn new kinds of skills, help them incorporate those new technologies into their workflows.

The speaker expresses empathy for those affected by AI's impact on jobs and advocates for social systems that could help these individuals adapt and learn new skills.

• There are still many research breakthroughs needed to achieve Artificial General Intelligence (AGI).
• Predicting the timing of these breakthroughs is challenging.
• AGI would require AI to have its own goals and motivations, which is not currently a focus of research due to economic and practical reasons.

We do have a lot of research breakthroughs we still need to make in order to achieve AGI.

The speaker acknowledges that significant research progress is required to reach AGI, indicating that it is a complex and ongoing challenge.

• Knowledge distribution is uneven, and retrieval augmentation for large language models (LLMs) is undervalued.
• Being in Silicon Valley helps AI founders but is not essential.
• The AI community should focus on real risks and not be overly influenced by science fiction scenarios.
• AI will play an even larger role in society in ten years.
• Pausing the development of AI models is not supported by the speaker, as continuous improvement is beneficial.
• U.com aims to be the default on many devices, helping people find better answers.

Knowledge is similar to the future is already here, but not equally distributed. How important retrieval augmentation is for llms. A lot of people underestimate that.

The speaker believes that the significance of retrieval augmentation in the context of large language models is underestimated by many people, mirroring the uneven distribution of future advancements.

• Coder AI offers an AI-powered work assistant to help teams finish tasks and make progress.
• Navan provides a travel and expense system that rewards employees for saving company money.
• Public offers a simple way to earn high yield on cash through treasury accounts.

Well, Coda is here with their new aipowered work assistant that helps you and your team not just finish tasks, but make progress so your product team can bring a feature to market faster.

This promotion highlights Coda's AI-powered work assistant, which is designed to help teams be more efficient and productive in their tasks.