# Respan: Open-source Rust AI Gateway with Plugin Architecture

> Respan's open-source Rust AI gateway lets teams build custom AI infrastructure using modular plugins.
- Title: Respan: Open-source Rust AI Gateway with Plugin Architecture
- Summary: Respan's open-source Rust AI gateway lets teams build custom AI infrastructure using modular plugins. Respan provides an open-source Rust AI gateway where…
- Keywords: respan, rust, ai gateway, plugins, open source, lite llm, mcp, providers, gateway features, pii redaction, smart routing, fallbacks
- Source: Hacker News — https://news.ycombinator.com/item?id=50167583
- Read time: 2 min
- Topics: respan, rust, ai gateway, plugins, open source, lite llm, mcp, providers, gateway features, pii redaction, smart routing, fallbacks
## Respan Launch
Respan is an open-source Rust AI gateway built around plugins for teams to assemble their AI infrastructure.

> Hey HN, this is Andy from Respan (https://www.respan.ai). We are building an open-source Rust AI gateway.
## Plugin Architecture
Everything in Respan is a plugin including providers, protocols, modalities, and gateway features like PII redaction.

> Respan provides a small core and public interfaces that teams can use to assemble the ai infrastructure their application needs.
## Problem with Existing Gateways
Existing AI gateways struggle to add new features without breaking existing functionality, causing teams to build custom gateways.

> Adding new features can become prohibitively difficult so developers can end up building another gateway to solve a specific problem.
## LiteLLM Experience
Respan started with LiteLLM but faced issues with 100,000+ lines of wrapper code and connection errors after a LiteLLM upgrade.

> We've been building our gateway based on LiteLLM since 2.024 and now it's providing services for thousands of orgs.
## Redesign Motivation
Respan redesigned the gateway to have a small core responsible for event dispatching with plugins handling everything else.

> Inspired by Nginx, we think an ai gateway should have a really small core only responsible for dispatching events.
## Plugin Examples
Plugins include providers like OpenAI, Anthropic, Gemini, protocols like chat and voice, and modalities like images and videos.

> A provider is a short file, Mistral's is 22 lines, and each quirk gets its own test.
## Request Handling
Respan handles requests by translating between client and provider formats through shared types for messages and tool calls.

> The core runs plugins in registration order allowing them to choose a provider filter or wrap either stream.
## Open Source Availability
Respan is open source under MIT or Apache-2.0 licenses with GitHub repository for direct implementation.

> The gateway is open source at https://github.com/respanai/respan-gateway under the mit or apache two point zero license.
## Self-Hosted Option
Users can run Respan directly on their local machine without using the hosted platform.

> You don't need to use respan's hosted platform and can run it directly on your local machine.
## Hosted Platform
Respan's hosted platform provides native observability, evaluations, and unified billing for users.

> We make money from our hosted platform (https://www.respan.ai/pricing) which runs the gateway for you with native observability and evals.
## Plugin Development
Developers can create plugins by following PLUGINS.md in the repository with links to existing plugins.

> PLUGINS.md in the repo explains how to write one and links to the plugins we've created so far.
## Ecosystem Roadmap
Respan's roadmap includes building an ecosystem of plugins for routing, tools, filtering, observability, and security.

> Turning that into an ecosystem is basically our roadmap and we welcome you to contribute your own plugins.
## Rust Implementation
Respan uses natively compiled Rust crates for plugins, not interpreted or dynamically loaded code.

> The plugins are natively compiled rust crates not interpreted or dynamically loaded.
## Usage Reporting
Respan handles usage reporting quirks like Groq's token usage in its own field and Mistral's lack of reporting.

> Groq only gives you usage on streamed responses if you ask for it with stream_options otherwise you're guessing.
## Community Feedback
The community has raised concerns about usage reporting, plugin security, and the complexity of Rust implementation.

> The point about plugin boundaries and safety is spot on in rust choosing between compile-time static dispatch vs runtime dynamic loading is always the hardest architectural trade-off.
## Next Steps
Respan aims to build a plugin ecosystem for routing, tools, filtering, observability, and security features.

> We'd love to answer any questions and hear your feedback!
## Key takeaway

Respan provides an open-source Rust AI gateway where teams assemble AI infrastructure through plugins without modifying the core.