Distributional is a modern platform for enterprise AI testing and evaluation to make AI safe, reliable, and secure. Pulse 2.0 interviewed Distributional co-founder and CEO Scott Clark to learn more about the company.
Scott Clark’s Background
Could you tell me more about your background? Clark said:
“I founded Distributional in September 2023, alongside an 11-person founding team, all of whom bring a wealth of experience in testing complex AI systems from companies like Bloomberg, Google, Meta, Intel, SigOpt, Slack, Stripe, Uber, and Yelp. My academic background includes bachelor’s degrees in mathematics, physics, and computational physics from Oregon State University, as well as a PhD in applied mathematics and MS in Computer Science from Cornell University. At Distributional, we leverage our deep technical expertise by utilizing over 35 technical degrees and 5 PhDs among our 25 employees.”
“Before founding Distributional, I experienced firsthand the challenges of testing AI systems at scale while leading a 200-person AI and HPC software engineering team focused on applications running on Intel-based supercomputers. My journey in the AI space began when I co-founded SigOpt, a pioneering AI startup specializing in scalable model experimentation and optimization. SigOpt was founded in 2014, and received funding from Andreessen Horowitz, DCVC, Intel and others, raising over $17M before being acquired by Intel in 2020.”
“At SigOpt, we encountered three key insights that shaped my vision for Distributional. First, we helped customers develop bespoke objective metrics, or evals, and discovered that using multiple metrics—even if they were individually weaker—provided a better prediction of an AI system’s generalized success than a single, narrowly constructed composite metric. Second, we found that many of our customers used our optimization solutions for robustness and stress testing of AI applications, despite it not being specifically designed for that purpose. Lastly, we built a robust testing and evaluation system for our own algorithms, ensuring that new code updates wouldn’t disrupt our stochastic system. These experiences underscored the critical importance of testing in AI systems and the necessity of adopting a broad perspective to capture application behavior effectively.”
Formation Of The Company
How did the idea for the company come together? Clark shared:
“The idea for Distributional emerged from my extensive experience in the AI field, particularly during my time at SigOpt. As AI applications become increasingly integral to businesses, the operational risks associated with them have also grown, making effective testing essential. I saw the critical need for a robust solution to address the unique challenges of testing AI systems.”
“Our platform is designed to bridge the gap in AI testing, providing enterprise teams with the tools they need to proactively detect and address risks throughout the AI lifecycle. This means ensuring that applications behave consistently from development through production, which is no small feat given AI’s complexity and non-deterministic nature.”
“In my role as CEO, my primary responsibilities include guiding our strategic vision, overseeing product development, and ensuring that we are meeting the diverse needs of our enterprise customers. I focus on fostering a culture of innovation within our team, leveraging our deep technical expertise to continuously improve our platform. This involves not just enhancing our automated testing capabilities but also ensuring that our solutions remain customizable and standardized to address the unique requirements of enterprise AI applications. Ultimately, my goal is to help organizations deploy their AI solutions with confidence, reducing risks before they manifest in production.”
Favorite Memory
What has been your favorite memory working for the company so far? Clark reflected:
“The most recent memory that really encapsulates things for me is a recent customer pilot. The customer sent us data with one known issue to see if our system could catch it. Our system automatically created tests that caught four issues instead of one – three of which the customer was unaware of. Our product made it such a simple workflow and it was such a powerful realization of our vision in the product. And, by implication, it was a reflection of all the hard work of the entire team. I can’t wait to get it in the hands of more customers.”
Core Products
What are the company’s core products and features? Clark explained:
“At Distributional, our core product is a modern platform specifically designed for enterprise AI testing. This platform enables organizations to gain confidence in their AI applications throughout their entire lifecycle. Our key features include:
— Depth: We address the complexities of AI by allowing teams to conduct statistical tests on distributions of data properties. This depth of analysis provides visibility into the consistency and performance of all components within AI applications, helping teams understand and quantify behavior effectively.
— Automation: Our platform streamlines the testing process through automation. This includes the collection of application data, the derivation of testable properties, and the creation of statistical tests. By automating these tasks, we empower teams to derive insights quickly and recalibrate tests based on real-time feedback, enhancing the efficiency of their testing processes.
— Standardization: We provide a standardized solution that brings visibility and consistency across all AI applications within an organization. This feature ensures that teams can approach AI testing in a cohesive manner, enabling governance and oversight while also allowing for effective auditing of risk mitigation strategies. Our tools facilitate a repeatable testing process, making it easier for teams to share templates, configurations, and insights.
Overall, our platform fills a critical gap in the AI stack, enabling teams to conduct continuous and adaptive testing. This helps identify potential issues before they impact production, ensuring a more reliable deployment of AI applications.”
Challenges Faced
What challenges have Clark and the team faced in building the company? Clark acknowledged:
“As CEO, there are daily challenges from every direction. But most of them have been good challenges recently. As one example, we needed more engineers to deliver our roadmap faster, so we came together and mined our networks and doubled the team in about a month. I’m fortunate that most of the challenges I need to prioritize today focus on team growth or product development. These are the fun days!”
Evolution Of The Company’s Technology
How has the company’s technology evolved since launching? Clark noted:
“We have always maintained a relatively consistent product vision around a platform that would integrate with a customer’s existing systems, and provide a framework, dashboard and intelligence to automate AI testing as a net new component of their AI software lifecycle.”
“But we underestimated three components of our platform. First, we underestimated what proportion of the market needed a self-managed version of our product deployed in their VPC, so re-oriented to prioritize this deployment strategy and are doing on prem installs for customers today. Second, we underestimated the need for intelligence in our minimum viable product, which, for us, means introducing more automation around deriving properties, creating tests and calibrating these tests. This type of functionality is in our team’s wheelhouse, so it is now a standard way that customers interact with our platform – which makes it really easy for them to scale us up across all AI applications in production very quickly from scratch. Third, our product is valuable at each step of the AI software development lifecycle, but we underestimated how critical continuous testing was in the production phase. One of the things that separates testing for AI from testing for traditional software is that testing for AI needs to be both statistically deep and continuous in production. We’ve married the depth of the evaluation step with the continuous nature of monitoring in one solution to create a robust testing approach that runs continuously alongside AI apps in production. “
Significant Milestones
What have been some of the company’s most significant milestones? Clark cited:
“At Distributional, we’ve celebrated several significant milestones since our founding. First and foremost, we successfully raised our seed round, which provided the foundation for our early growth and development.”
“Following that, we validated the enterprise need for confidence in AI applications through extensive conversations with over 100 large enterprises across the financial, industrial, and technology sectors. This research was instrumental in shaping our understanding of the market needs and refining our product offering accordingly.”
“We also grew our team substantially, bringing together a group of experts in AI testing, engineering, and product development. This diverse talent pool has been crucial in driving our platform’s development. Today, we are a 24-person team. We have 19 technical team members (engineering or research) with 35 technical degrees and 5 PhDs between them.”
“Another key milestone has been the deployment of our enterprise testing platform with our enterprise design partners. This collaborative effort has allowed us to refine our solution in real-world settings, ensuring it meets the needs of our customers effectively.”
“Most recently, we secured a $19 million Series A funding round ($30 million total raised within the first year of founding) led by Two Sigma Ventures, with participation from notable investors like Andreessen Horowitz and dozens of angel investors. This funding will help us accelerate our growth, expand our team, and enhance our platform further.”
“Overall, these milestones reflect our commitment to addressing the critical need for robust AI testing solutions and our progress in building a platform that empowers organizations to deploy AI with confidence.”
Customer Success Stories
When asking Clark about customer success stories, he highlighted:
“We can’t share any specific customer names yet, but we are at various stages of working with several F500 firms in a variety of industries.”
Funding
When asking Clark about the company’s funding details, he revealed:
“Distributional recently secured $19 million in Series A funding, led by Two Sigma Ventures, with participation from notable investors including Andreessen Horowitz and Operator Collective. This round brings our total capital raised to $30 million within less than a year since our incorporation.”
“This funding milestone coincides with the initial enterprise deployments of our AI testing platform, which is designed to help AI engineering and product teams gain confidence in the reliability of their applications while reducing operational risks. Our platform addresses the unique challenges of AI testing, which requires consistent and adaptive methodologies over time due to the inherent complexities of AI systems.”
“Revenue metrics will evolve as we scale our deployments, refine our platform, and continue building strong partnerships. Our goal is to enable organizations to deploy AI solutions with greater confidence, maximizing the impact of AI across their use cases. We’re excited about the path ahead and the opportunities for growth as we continue to meet the needs of our enterprise customers.”
Total Addressable Market
What total addressable market (TAM) size is the company pursuing? Clark assessed:
“Our platform is horizontal – it works the same regardless of industry or AI use case. So our TAM is inclusive of all companies pursuing AI.”
“More realistically, however, we are prioritizing working with larger companies (F500 or G2000) who have real operational or reputational risk tied to their AI applications. They are willing to invest resources in the AI testing problem because they have a significant confidence gap in their AI applications that makes it hard for them to get these apps in production today – and holds implications for the downside of these apps once they are in production.”
“These companies are spending many tens of billions of dollars on AI across all efforts and we think we can capture a meaningful part of that by providing confidence through testing. We also think this will grow as AI becomes more powerful and prevalent in all businesses.”
Differentiation From The Competition
What differentiates the company from its competition? Clark affirmed:
“Confidence in software is primarily achieved through testing, ensuring the application behaves as expected with respect to performance and consistency. But, traditional software testing cannot be readily applied to AI because it focuses on asserting known, expected results on static, modular components to catch errors at specific stages of application development. Distributional focuses on distributional change in properties of dynamic, interconnected components to adaptively catch behavioral change throughout an application’s lifecycle as the application and its usage change over time.”
“Without access to AI testing today, many teams try to gain confidence through siloed tools focused on the development process or production monitoring. These tools often focus on a specific part of the AI stack and a specific type of application, like the prototyping stage of creating LLM applications, or the production monitoring of non-LLM, ML applications. Distributional can provide deeper insights and catch more issues by working across the application lifecycle from development to deployment to production and back as the application is iterated on. Furthermore, Distributional can be standardized throughout an organization as it is agnostic to the underlying application, allowing the platform to test all current AI applications as well as any future ones.”
“With existing development tools, teams are forced to hack together siloed research processes focused on benchmarking and evals. Teams seek to mitigate risk through means of spot checking and intuition by building the applications themselves, but this manual approach leads to a narrow view of risk focused on a handful of bespoke metrics and fixed datasets and is often unstandardized between projects. This leads to repeated work, inconsistent levels of confidence, and a lack of visibility into the process. Distributional can provide broader understanding of application behavior by looking at many properties of the application and data. Furthermore it can standardize this understanding through a repeatable, consistent, and visible process.”
“With production observability and monitoring tools, teams are forced to passively observe a small number metrics, catching issues after the fact, effectively treating their users as tests. Furthermore, when issues are discovered, observability and monitoring systems don’t provide enough contextual information to assess root cause and resolve the underlying issue because they only focus on specific metrics computed from recent lookback windows of data. Distributional can proactively address risk through testing earlier in the lifecycle and tying production testing failures back to historical context and allow for iterative testing as solutions are developed.”
“Distributional’s testing platform removes the operational burden on enterprises to build and maintain their own solutions or cobble together incomplete solutions with other tools. By proactively addressing these testing problems with Distributional, AI teams can deploy with more confidence and proactively catch issues with AI applications before they cause significant damage in production.“
Future Company Goals
What are some of the company’s future goals? Clark emphasized:
“Our future goals at Distributional are ambitious, and I’m incredibly proud of what we’ve achieved so far. However, we recognize that there’s much more work to be done.”
“First and foremost, our primary goal is to enhance our AI testing platform to ensure it effectively addresses the complexities and challenges of AI systems at scale. We want to be the go-to solution for enterprises looking to build and deploy reliable AI applications.”
“With the fresh capital from our recent funding round, we plan to accelerate our development efforts, expand our team, and refine our platform’s features to better serve our enterprise customers. This includes enhancing our intelligent automation capabilities and ensuring our platform remains adaptable to the evolving landscape of AI technologies.”
“Additionally, we aim to deepen our partnerships with enterprises across various sectors, leveraging insights from our design partners to continuously improve our offering. Our goal is to empower teams to proactively manage AI risk, thereby maximizing the impact of AI within their organizations.”
“Ultimately, we are committed to solving the AI testing problem at scale and ensuring that as the power and pervasiveness of AI grow, our platform provides the reliability that enterprises need to confidently deploy their applications.”
Additional Thoughts
Any other topics you would like to discuss? Clark concluded:
“Our product is designed to enable teams to test their AI applications in a variety of ways. They can test for performance against metrics they determine to represent successful usage of their application. They can test for bias, fairness, toxicity and other standards around whether the AI application will unfairly or negatively impact certain populations. It can also be used to test for robustness or generalizability of these AI applications. By continuously running tests with our product across this array of metrics, teams build and maintain confidence in their AI application.”
“This is a particularly critical capability for teams today because the usage of AI applications is growing much faster due to the rise of LLMs. AI is opening up a lot of opportunities for enterprises, but this also comes with challenges. Teams who have deployed GenAI struggle to understand how these applications behave, resulting in hallucinations, incorrectness or an unreliable customer experience, among other issues. Other teams have a long backlog of these applications they want to deploy, but struggle with confidence in their performance or capacity to satisfy governance needs—so these use cases are withering on the vine. Additionally, some teams are getting these applications in production, but spending way too much time on the cumbersome existing methods.”
“Better testing is therefore critical to unlocking this potential of AI for enterprises – and for helping them avoid ending up on the front page of a publication for a rogue AI application, hallucination or evidence of bias.”
“Even more challenging, these AI applications are moving targets. An AI application may perform well on a use case today and degrade tomorrow, necessitating an adaptive AI testing platform that evolves with the user’s understanding of these applications. This includes algorithmic features that automatically learn user preferences for how tests are calibrated over time – a version of reinforcement learning with human feedback.”
