From Hacker News · · 2m
Respan: Open-source Rust AI Gateway with Plugin Architecture
Respan's open-source Rust AI gateway lets teams build custom AI infrastructure using modular plugins.
In brief
Respan's open-source Rust AI gateway lets teams build custom AI infrastructure using modular plugins. Respan provides an open-source Rust AI gateway where teams assemble AI infrastructure through plugins without modifying the core. Originally reported by Hacker News.
Respan Launch
Respan is an open-source Rust AI gateway built around plugins for teams to assemble their AI infrastructure.
“Hey HN, this is Andy from Respan (https://www.respan.ai). We are building an open-source Rust AI gateway.”
Plugin Architecture
Everything in Respan is a plugin including providers, protocols, modalities, and gateway features like PII redaction.
“Respan provides a small core and public interfaces that teams can use to assemble the ai infrastructure their application needs.”
Problem with Existing Gateways
Existing AI gateways struggle to add new features without breaking existing functionality, causing teams to build custom gateways.
“Adding new features can become prohibitively difficult so developers can end up building another gateway to solve a specific problem.”
LiteLLM Experience
Respan started with LiteLLM but faced issues with 100,000+ lines of wrapper code and connection errors after a LiteLLM upgrade.
“We've been building our gateway based on LiteLLM since 2.024 and now it's providing services for thousands of orgs.”
Redesign Motivation
Respan redesigned the gateway to have a small core responsible for event dispatching with plugins handling everything else.
“Inspired by Nginx, we think an ai gateway should have a really small core only responsible for dispatching events.”
Plugin Examples
Plugins include providers like OpenAI, Anthropic, Gemini, protocols like chat and voice, and modalities like images and videos.
“A provider is a short file, Mistral's is 22 lines, and each quirk gets its own test.”
Request Handling
Respan handles requests by translating between client and provider formats through shared types for messages and tool calls.
“The core runs plugins in registration order allowing them to choose a provider filter or wrap either stream.”
Open Source Availability
Respan is open source under MIT or Apache-2.0 licenses with GitHub repository for direct implementation.
“The gateway is open source at https://github.com/respanai/respan-gateway under the mit or apache two point zero license.”
Self-Hosted Option
Users can run Respan directly on their local machine without using the hosted platform.
“You don't need to use respan's hosted platform and can run it directly on your local machine.”
Hosted Platform
Respan's hosted platform provides native observability, evaluations, and unified billing for users.
“We make money from our hosted platform (https://www.respan.ai/pricing) which runs the gateway for you with native observability and evals.”
Plugin Development
Developers can create plugins by following PLUGINS.md in the repository with links to existing plugins.
“PLUGINS.md in the repo explains how to write one and links to the plugins we've created so far.”
Ecosystem Roadmap
Respan's roadmap includes building an ecosystem of plugins for routing, tools, filtering, observability, and security.
“Turning that into an ecosystem is basically our roadmap and we welcome you to contribute your own plugins.”
Rust Implementation
Respan uses natively compiled Rust crates for plugins, not interpreted or dynamically loaded code.
“The plugins are natively compiled rust crates not interpreted or dynamically loaded.”
Usage Reporting
Respan handles usage reporting quirks like Groq's token usage in its own field and Mistral's lack of reporting.
“Groq only gives you usage on streamed responses if you ask for it with stream_options otherwise you're guessing.”
Community Feedback
The community has raised concerns about usage reporting, plugin security, and the complexity of Rust implementation.
“The point about plugin boundaries and safety is spot on in rust choosing between compile-time static dispatch vs runtime dynamic loading is always the hardest architectural trade-off.”
Next Steps
Respan aims to build a plugin ecosystem for routing, tools, filtering, observability, and security features.
“We'd love to answer any questions and hear your feedback!”
The takeaway
Respan provides an open-source Rust AI gateway where teams assemble AI infrastructure through plugins without modifying the core.
