by Aparna Sinha (originally published on Pear’s website , here).
We are often asked by founders what type of AI company to start, and how they can start a long-lasting business.
Pear’s AI and ML thesis is based on the belief that these fields will be game-changing in the same way as the internet was in the late ’90s. We believe that AI and ML are going to revolutionize enterprise software as well as consumer applications. We are particularly interested in generative AI. We believe that it has the potential to boost productivity by 5-10 times across multiple verticals. Pear has invested in AI/ML and generative AI for years. There’s still a great deal of noise in the AI space. This is why we hosted a technical fireside “Perspectives In AI” series in order to cut through all the hype and connect research with product.
Researchers at Google made a breakthrough in 2017 when they invented the transformer. This was the beginning of a lot of progress in Generative AI. The combination of this innovation and the availability at scale of GPUs on public clouds allowed large language models to be trained using massive datasets. These models exhibit emergent behaviors and perform tasks that appear intelligent when they reach 6 billion parameters. These models can be generalized by training them on mixed-domain data, such as the pile dataset. They are capable of a variety of tasks, including code generation, summary, and text-based functions. They are statistical models that have non-zero rates of error and hallucination, but they still represent a breakthrough for the development of intelligent output.
A related advance is the ability of these large foundational model to be customized through transfer learning for specific tasks. There are many techniques used, but the one most relevant for commercial applications and efficient is fine tuning with Low Ranking Adaptation. LoRA allows the creation of smaller, fine-tuned models that are optimized for a specific purpose or character and work in conjunction with a larger model to produce a more efficient and effective output. RLHF, RLAIF and RLHF-S are two of the latest innovations that have allowed LLMs to be widely released. These models can be aligned to company values or specific use-case needs. These breakthroughs collectively have catalyzed today’s capabilities, indicating a rapid acceleration.
Text is the most common domain for AI models. However, there have been significant advances in other areas, such as video, image, vision and biological systems. In particular, this year marks significant advancements in generative AI including speech and multimodal model. It is important to note the interplay between commercial closed models and open-source models. Open source models are becoming just as good as their commercial counterparts and the training costs of these models is declining.
Our AI thesis is divided into three parts. 1. Applications along with foundation/fine tuned models. 2. Data, tooling, and orchestration. 3. Infrastructure which includes cloud software and hardware. We believe that the applications that will succeed in the generative AI eco-system are those that use ensembles of task-specific models, which have been fine-tuned with proprietary data (specific for each vertical, user experience, and use case), as well as retrieval augmentation. OrbyAI was an early innovator in the area of AI-driven workflow automation. It’s extremely useful and relevant for enterprises. tooling for integratingmodel-based applications, orchestrating them, testing and evaluating them, as well as securing, deploying, and continuously deploying these applications, is also a separate category of investment. Nightfall is well aware of this issue and has focused its efforts on tools for data privacy, security and composite AI applications. We see a lot of opportunity for infrastructure advancements at the software layer, hardware layer, and cloud services level to enable efficient training and inference across multiple device forms. Infrastructure is a vast area that includes everything from high-speed networking to specialized AI chips, to novel model architectures. Quadric, a Pear Portfolio Company working in this area.
Entrepreneurs who are successful will use a combination of specialized models, fine-tuned using proprietary or individual data, along with retrieval enhancement and prompt engineering techniques, to build intelligent, reliable applications that automate cumbersome processes. In most enterprise cases, the models are augmented with a retrieval system that ensures a fact base and explains results. In this context, we discuss open-source models because they are more accessible and can be used to access proprietary data in private environments. They are also often available in a variety of sizes, allowing for applications to be developed with localized and edge-based form factors. With new releases like Llama2, open source models have become more powerful and cost-effective.
We think that when we speak about moats it is important for founders to have a compelling insight into the problem they are solving, and have experience in going-to-market. It’s important for any startup, but for AI it is even more crucial to have access to proprietary data as well as skilled human expertise. According to our thesis, building a moat requires the use of proprietary data and expertise in fine-tuning models for specific uses cases. Startups who solve open AI problems such as data privacy, improving accuracy, safety and compliance, or providing scalable mechanisms to integrate data can have a moat.
The “Anatomy of modern AI applications” or high-level architecture often involves preprocessing, chunking and using an embedding models, then putting these embeddings in a database, then creating an index, or multiple indexes, and at runtime creating embeddings from the input, and then searching the index, with appropriate curation, and ranking of results. AI applications can pull in data and information from other sources using databases and APIs. This is useful for real-time or point-in time information or facts that are referenceable. RAG, or retrieval-augmented generation, is the term used to describe this. Many applications need prompt engineering, including to format the model input/output and add specific instructions, templates, or provide examples to the LLM. The retrieved data combined with prompting engineering is fed into an LLM, or a group of LLMs/mixture of large language models. The synthesized output then is sent back to the end user. LLMs are equipped with input and output validation, rate-limiting, and other privacy and safety mechanisms. The Embedding Model and the LLMs are bolded because they benefit from fine tuning.
There are many applications where generative AI can disrupt the market. The idea of “AI Assistants”, which are personalized for all consumers, will probably be the biggest change in computing in the near future. In the short term, we can expect “assistants”, which are specific to major functional areas. The engineering function and software development in particular will be the first to adopt AI assistants, from code development through to application troubleshooting. While some jobs (e.g. SWE, Test/QA and SRE) are already using generativeAI, there’s still a lot more potential. Generative AI simplifies data and analytics, which is a second opportunity area. All CRM systems, including marketing, sales and support, recruiting, learning/education, and HR, are rich sources of generative AI application development. Sellscale, one of our newest portfolio companies, accelerates sales and marketing using generative AI. We believe that it is crucial for startups in all these areas to create deep moats by using proprietary data, and fine-tuning domain specific models.
We can also see AI applications in the healthcare, legal and manufacturing verticals, as well as in finance, insurance, biotech, and pharma. All of these have workflows with a lot of text, numbers, or images. Artificial intelligence is a great way to improve those workflows. Federato, a Pear Portfolio company, is using AI for risk optimization in the insurance industry. VizAI utilizes AI to improve clinical care pathways and connect care teams sooner. These verticals also have higher standards for accuracy, privacy, and explainability. This provides great opportunities for differentiation. Separately media, retail, and gaming verticals are emerging with generative AI opportunities that cater to a more consumer-centric market. This type of vertical’s scale and monetization may differ from verticals that are highly regulated. Longer term, we also see applications for Climate, Energy and Robotics.
Pear believes that the infrastructure and tools layers will be the ones to benefit the most from generative AI. Startups that solve problems with systems to make inference, training, and data integration more efficient, push the envelope on context lengths, enable data integration, model alignement, privacy, safety, and build platforms for model evaluation and iteration should see a rapid growth market.
We are excited to work with entrepreneurs who are creating the future of these workflows. AI’s recent advancements offer a new capability which will force us to rethink how we work, and what can be done intelligently. We are excited to see the pain points that you will be addressing!