Learn extra at:
This newest launch additionally contains cost-based load balancing, enabling Kong to route requests based mostly on token utilization and pricing. For instance, low-complexity prompts can go to cheaper fashions, whereas high-value duties path to premium suppliers. That is particularly useful for corporations utilizing a number of LLMs for various use circumstances, permitting them to optimize for each efficiency and finances.
Kong
This visible outlines the breadth of Kong AI Gateway options, together with LLM orchestration, load balancing, immediate administration, and extra.
Moreover, Kong now helps pgvector, extending semantic capabilities like routing, caching, and guardrails to Postgres-based databases. This offers platform groups extra flexibility when designing AI pipelines inside current cloud-native environments like AWS Relational Database Service or Azure Cosmos DB.
