The Uber of LLM Inference: Looking Into OpenRouter

This article was automatically translated from Japanese by AI. Read the original Japanese version →

Table of Contents

In recent years, large language models (LLMs) have become deeply embedded in our digital lives, and the pace of their evolution has been remarkable. However, as diverse models and inference providers continue to emerge, companies and developers face a complex challenge: which model should they use, from which provider, and under what conditions?

For example, consider a company building a customer support chatbot.

  • Which model should you choose? Do you need fast responsiveness to answer customer questions quickly, or high reasoning capability to handle complex inquiries? Is a lightweight model like GPT-3.5 enough, or do you need a more capable model like GPT-4 or Claude 3? There may also be cases where a custom model specialized for a particular industry is required.

  • Which provider should you choose? Should you rely on the stable services of major cloud providers (AWS, OpenAI, Anthropic, Google Cloud, etc.), or are you drawn to the cost efficiency and specific features of more niche specialized providers (such as providers focusing on particular open-source models)? Each provider's regions, SLA (Service Level Agreement), and support structure are also important selection criteria.

  • Under what terms can you use it? Beyond cost (token pricing, pay-as-you-go vs. flat-rate), latency (response speed), throughput (processing capacity), and rate limits (API call limits), you also have to consider data privacy protection, security, and contractual or legal compliance requirements (GDPR, CCPA, Japan's Act on the Protection of Personal Information, etc.).

Finding the optimal combination among these wide-ranging options that best fits a project's requirements takes specialized knowledge and ongoing research. Amid all this, there is a service occupying a very unique position—what you might call the "Uber of LLM inference infrastructure." That service is OpenRouter.

OpenRouterThe unified interface for every model. Find the best models & prices for your promptsOpenRouter · openrouter.ai

What Is OpenRouter?

In short, OpenRouter is a proxy service for LLM inference. The basic concept is that as long as an API is provided in an OpenAI-API-compatible format, OpenRouter relays requests to any LLM model or inference provider. This lets developers access multiple different LLMs and their providers through a unified interface. It also lets them centrally manage billing across multiple providers and consolidate payments.

For details on OpenRouter's terms of use from a provider's perspective, the official documentation provides thorough information, which is very helpful for understanding the nature of the service.

At the core of this service are the following features.

Fallbacks

One of OpenRouter's most appealing features is model and inference provider fallback. For example, you can set GPT-4 as your primary model, and if the GPT-4 API stops responding or becomes unavailable for some reason, it can automatically switch to an alternative model such as Claude 3. This is a useful feature for ensuring stable service operations and reducing the risk of downtime.

However, when using this fallback feature to switch models rather than just providers, it assumes there is a certain degree of compatibility between the models being switched. Even with the exact same prompt, response accuracy and tendencies can differ greatly between different models. Therefore, when choosing fallback target models, it is important to consider not only technical compatibility, but also the impact on business requirements and user experience, and to evaluate whether the response quality and style fall within an acceptable range.

Flexibility Through Decoupling Models and Providers

In OpenRouter's world, models and providers exist separately. For example, models with publicly released weights in open source (open-weight models) can be used through a variety of inference providers registered on OpenRouter. This brings the freedom and stability of being able to use your desired model via whichever provider is best at any given moment, without being locked in to a specific provider.

Provider Routing

OpenRouter's real strength lies in its intelligent provider routing. OpenRouter tracks real-time data across many providers, including per-token pricing, throughput, latency, and uptime/outage status. Based on this information, it dynamically selects and routes user requests to the highest-performing and most cost-effective provider.

Problems OpenRouter Solves and Use Cases

You might naturally wonder, "Do you really need to switch models that often?" Still, OpenRouter solves several concrete problems:

  • Fallback against rate limits: Even when you hit the rate limit of a specific provider or model, automatically switching to another provider or model helps keep your application running smoothly.

  • Pursuing optimal price and throughput: By comparing the price and throughput of models offered by various providers in real time, you can use them under the best available conditions at that moment.

  • Simple code-based model benchmarking: When testing multiple models or providers, you don't need to keep changing your code—you can easily run benchmarks through a common interface. This is very useful for evaluating new LLMs and finding the best model for a specific task.

Challenges and Concerns with OpenRouter

While OpenRouter looks like a convenient service, it also comes with several challenges and concerns.

Concerns Around Regions and Confidential Information

OpenRouter's infrastructure is likely located in the United States. As a result, highly sensitive customer data, trade secrets, or other information subject to strict data sovereignty or compliance requirements may currently be difficult to pass to LLMs via OpenRouter. Ideally, OpenRouter would have infrastructure within Japan and also offer a feature to narrow provider routing down to a specific region (such as within Japan). Since the provider application form has a field for the location (country) of the inference infrastructure, it should be technically possible for OpenRouter to offer such a feature. That said, recently even major cloud vendors have tended to hold off on offering the latest models in Japan regions, so maybe it's not something worth worrying too much about.

Concerns Around Provider Trustworthiness

One of the deepest concerns with using OpenRouter is that there is no way to verify whether the provider OpenRouter routed your request to actually ran inference using the specified model. The same problem exists with major cloud vendors, of course, but there you ultimately rely on trust in that company's brand and track record.

Of course, OpenRouter does provide a feature to whitelist specific providers, but OpenRouter's true value only shines if you are willing to send prompts to a decent number of providers. If a provider you've never heard of claims, "We're genuinely using the specified model, we don't log prompts, and we don't use them for training," how are you supposed to trust that? It's also an issue that the criteria OpenRouter uses to vet providers are unclear.

Therefore, if you want to use it somewhat securely, you end up having to do your own credit/trust vetting of providers. Anyone considering OpenRouter is confronted with a fundamental question about data processing safety and reliability: "Is it really okay to casually swap out the company executing your processing all the time?"

Trade-offs with Discount Plans

Major cloud providers (such as AWS) offer plans for high-volume enterprise users—such as Provisioned Throughput or Reserved Instances—that heavily discount unit prices in exchange for committing to a certain duration or throughput. However, in an operating model that dynamically switches among multiple providers through a proxy service like OpenRouter, it's hard to take advantage of these discount plans. Because usage isn't concentrated on a single provider, it becomes difficult to secure enough volume in one place to qualify for discounts.

At present, OpenRouter itself does not offer usage-based volume discounts or long-term commitment discounts. As a result, you also need to factor in the possibility that total costs could end up higher compared to contracting directly with individual providers.

The Philosophy Underlying OpenRouter

OpenRouter's service design is built on two assumptions.

The first assumption is that depending on circumstances at any given time, even for the same model, pricing and service levels (such as rate limits) can vary by provider. You could also view this as a belief that stable supply of LLM inference won't be achieved anytime soon.

The second assumption is that LLMs will increasingly be offered by various companies under diverse terms, and that there should be high "interchangeability" among those models.

At the core of OpenRouter is the idea that in such an environment, rather than modifying application code every time you want to switch models or providers, you need a mechanism that can easily and automatically route models and providers through a unified interface.

However, it is quite conceivable that major cloud vendors will gradually achieve stable supply and competitive pricing. Nor can we rule out the possibility that those major vendors will start offering similar proxy/routing features as part of their own services. If that happens, it remains an open question whether OpenRouter can maintain a long-term competitive advantage.

Summary

OpenRouter offers a very compelling solution for improving the stability and optimization of LLM inference. In particular, its model fallback and intelligent routing features carry significant value for boosting the robustness and efficiency of LLM-powered applications.

On the other hand, concerns about provider transparency and trustworthiness remain a major hurdle, especially for companies handling highly confidential data. For the majority of companies and developers, the inference infrastructure provided by major cloud vendors may well be enough.

I hope this article helps you in selecting your services.