Portkey is building the critical tooling every AI team needs to ship reliable, cost-efficient, and performant LLM apps. Pulse 2.0 interviewed Portkey co-founder Rohit Agarwal to learn more about the company.
Rohit Agarwal’s Background
What is Rohit Agarwal’s background? Agarwal said:
“I’ve been in the AI applications space for the past decade. At Freshworks, we built customer support bots using BERT & GPT-2, and later, at my next company, we built one of the world’s first content generation apps on top of GPT-3 in early 2021. Through these experiences, I personally encountered many of the challenges that developers face when building AI apps. This firsthand knowledge inspired us to create Portkey, aiming to provide developers with superior tooling and infrastructure to address these production challenges.”
Formation Of Portkey
How did the idea for the company come together? Agarwal shared:
“When we were building a content generation app based on GPT-3, we encountered a unique set of challenges that were fundamentally different from the traditional “cloud” world – the inherent nature of large language models (LLMs) is non-deterministic. They require a lot of handholding, and building apps on top of them required us to build a very different set of tooling than what we were familiar with.
This involved everything – from simply being able to observe the requests so that we can debug them, to testing new capabilities and deploying them to production at scale, to handling inexplicable LLM errors, and to building an AI service that the whole company could use.
This experience highlighted the need for specialized tools tailored to LLM-based development.
With the release of ChatGPT, we had our lightbulb moment. We realized that the tooling we’ve been building is exactly the tooling all the other engineers building on top of LLMs will also need. This insight led us to found Portkey.ai and bring our experience and product-solution fit to the world. It has been a very fulfilling journey so far!”
Favorite Memory
What has been your favorite memory working for the company so far? Agarwal reflected:
“Just recently, we celebrated our 1st anniversary of starting Portkey.ai with the entire team and their family. Interestingly, right when we were about to cut the cake, I got a ping on my phone saying that our first enterprise wire had gone through. Felt like a surreal moment!”
Core Products
What are the company’s core products and features? Agarwal explained:
“At Portkey.ai, we are building a Control Panel for Gen AI apps. We do 5 things to help enterprises control their in-production app:
- Observability – Setup instant monitoring, tracing, feedback loops, and alerts for all your LLM calls – easily debug issues, find usage patterns, and areas of improvement
- AI Gateway – Use not just OpenAI but 250+ other LLMs (open source & closed) without changing production code – to help reduce dependency on one provider and experiment faster with new LLMs. Set up fallbacks, load balancing, retries, caching, request timeouts, rate limiting, and more for LLMs – to make apps more reliable, robust, and cost-efficient
- AI Guardrails – Redact PII from user inputs, and also enforce LLM behavior in real-time so that your customized content policies are always being followed.
- Governance – Empower your org with a central dashboard to manage AI services effectively. Track service adoption across your company, implement granular permissions and access controls, and easily provision new LLMs. With comprehensive activity auditing, maintain oversight and ensure responsible AI use throughout your organization, all from a single, intuitive interface.
- Prompt Management – Manage all your system prompts, prompt variables, and their various versions in one place with advanced publishing controls – so that you can iterate very fast and ship to production without having to make any changes to your production code every time.
Portkey is built from the ground up to be enterprise-ready. It complies with global standards of data privacy and security with SOC2, ISO, HIPAA, and GDPR certifications, and undergoes regular audits.
Challenges Faced
What challenges have Agarwal and the team faced in building the company? Agarwal acknowledged:
“Being a startup, there are existential challenges almost every week. But that’s part of the job. On any given week, a big name company is launching a killer competitor product, a key customer that was about to convert is going cold, growth is stalling, a key team member is leaving. What doesn’t change is the constant customer demand for what we are solving and our path towards our vision getting clearer – so that’s the fuel.”
Evolution Of Portkey’s Technology
How has the company’s technology evolved since launching? Agarwal noted:
“Tremendously. We started from zero, and just within a year, we now process tens of billions of tokens through our Gateway every single day. Not only that, we support hundreds of production apps and deployments – all thanks to our early bets on architecture and infrastructure that are now paying off. With time, we have started incorporating LLMs and no-code tools more and more in our workflows. Everything from building quick dashboards, to full landing pages, to building LLM-powered features that bring user delight.”
Significant Milestones
What have been some of the company’s most significant milestones? Agarwal cited:
“Crossing our first billion LLM tokens daily, crossing 200 LLMs supported on the platform, 100 production deployments, 2 Billion total requests processed – are the most significant milestones.”
Customer Success Stories
When asking Agarwal about customer success stories, he highlighted:
“Qoala, a leading insurtech in Southeast Asia, manages 30 million insurance policies by automating 25+ use cases with AI. To manage prompts and costs for these use cases, Qoala adopted Portkey’s unified platform, which provided a centralized system for managing prompts, tracking costs, and maintaining operational visibility. Portkey’s features, such as prompt management, feedback loops, and AI guardrails, have streamlined Qoala’s processes, allowing them to scale efficiently. By adopting Portkey, Qoala has optimized operations, reduced manual oversight, and strengthened its AI-driven capabilities.”
Funding
When asking Agarwal about the company’s funding, he revealed:
“We’ve raised a $3 million seed from Lightspeed Venture Partners and angels from OpenAI, CloudFlare, Amazon, Asana, and more. We are generating revenue from both our self-serve product as well as from our enterprise offering, and growing the revenue very fast. We are not looking for any further funding at the moment.”
Differentiation From The Competition
What differentiates the company from its competition? Agarwal affirmed:
“Majority of our competition in the LLM tooling space is focused on helping developers “build” LLM-based apps – which involves a specific set of workflows and frameworks. We believe, when you take these same apps to production, the challenges you encounter are very different – a 3% API failure set suddenly starts to hit, random 502 errors need urgent attention, being able to see exactly how much you are spending on each customers in LLM costs becomes paramount – and all of this, while maintaining development velocity, and fast time-to-market. Portkey.ai is tailor-made to optimize ‘in-production’ workflows and helps reduce time-to-market and experiment-to-production time, and significantly boosts reliability numbers.”
Future Company Goals
What are some of the company’s future company goals? Agarwal concluded:
“The next areas that we are personally very excited about are – online evaluations, integration into existing big platforms for data store and more, Gateway upgrades to bring in more providers, and overall, increasing contributions to the open source community and to our partners.”
