Fisent Technologies, a pioneer in Applied GenAI Process Automation solutions, announced a new framework designed to help its Fisent BizAI customers objectively evaluate the performance of various GenAI models for specific business process automation use cases. Fisent’s GenAI Efficacy Framework (GEF) enables enterprises to measure, compare, and select the most effective GenAI models based on key metrics like accuracy, speed, cost, and consistency.
Fisent Technologies, a pioneer in Applied GenAI Process Automation solutions, announced a new framework designed to help its Fisent BizAI customers objectively evaluate the performance of various GenAI models for specific business process automation use cases. Fisent’s GenAI Efficacy Framework (GEF) enables enterprises to measure, compare, and select the most effective GenAI models based on key metrics like accuracy, speed, cost, and consistency.
Fisent’s GEF includes a configurator that customers can use to evaluate the tradeoffs inherent in comparing their LLM options. And the GEF configurator intelligently scores given requirements against available LLMs to produce a ranked list of the models expected to perform best for a specific situation along with numerous visual comparison charts for speedy analysis. For example, using the configurator to increase the requirement for accuracy will adjust the rank-order of LLMs under consideration.
Plus, other variables evaluated by the configurator adjust accordingly. If there is a need for both speed and high accuracy, the cost variable associated with the best-fit LLMs is likely to increase. And GEF is useful to Fisent customers when they initially implement Fisent BizAI against a specific process automation and again when evaluating new models or model upgrades.
The key benefits of Fisent’s GEF include:
- Comprehensive evaluation – Assess GenAI models across multiple variables, such as accuracy, speed, cost, and consistency.
- Data-driven insights – Provide actionable recommendations based on objective metrics and statistical analysis.
- Continuous optimization – Enable enterprises to monitor and improve model performance over time.
- Ease of use – Streamline the process of evaluating and selecting GenAI models, even for non-technical users.
KEY QUOTES:
“The idea for GEF sparked as typical AI model evaluation methods, like those that measure Massive Multitask Language Understanding (MMLU), failed to balance the nuanced requirements of our customers’ real-world automation decisions. GEF offers a more pragmatic approach by evaluating the most important factors to any given application decision: accuracy, speed, cost, and consistency. Understanding these metrics allows enterprises to make more informed decisions about which LLM to employ for each of their process automation challenges.”
“Fisent is committed to driving innovation in the Applied GenAI Process Automation space and empowering enterprises to harness the full potential of this transformative technology. Providing a robust and reliable framework for evaluating meaningful GenAI model performance is just one way that Fisent is delivering on this vision.”
-Adrian Murray, Founder and CEO of Fisent
