How to Build a Local Semantic Router in Python to Split Traffic Between Claude and Local LLMs

Stop Burning Your API Budget on Simple Tasks

Let's be honest: using Claude 3.5 Sonnet to classify a user's intent as "yes" or "no" is like hiring a Michelin-star chef to toast a slice of white bread. It is overkill, and it is costing you a small fortune in API credits.

While frontier models are spectacular at complex reasoning, multi-step planning, and writing elegant code, a huge chunk of your application's traffic consists of mundane tasks. Sentiment analysis, basic entity extraction, and straightforward QA can easily be handled by small, open-weight models running locally.

The trick is knowing when to delegate. In this guide, we are going to build a local semantic router using Python. This router will inspect incoming user prompts and determine whether they can be resolved by a lightweight local model running via Ollama, or if they need to be escalated to Claude.

By the end of this tutorial, you will have a production-ready routing layer that keeps your cloud API bills down without sacrificing the intelligence of your application. Let's see what makes your AI stack tick.

The Architecture: How Semantic Routing Works

Instead of asking a cheap LLM to classify every incoming prompt (which adds latency and API costs of its own), we will use local vector embeddings.

Here is how the pipeline works: 1. Define our Routes: We establish two primary paths: local (for simple tasks like classification, parsing, and basic queries) and cloud (for complex logic, coding, and nuanced reasoning). 2. Generate Route Vectors: We write a few exemplar prompts for each category and embed them using a fast, local embedding model via sentence-transformers. 3. Calculate Similarity: When a user prompt comes in, we embed it and compare its vector representation against our route exemplars using cosine similarity. 4. Dispatch: The router automatically forwards the prompt to either our local Ollama instance or the Anthropic API.

Prerequisites

Before writing any code, make sure you have the required libraries installed and Ollama running on your local machine.

First, install the Python dependencies:

`bash pip install sentence-transformers numpy anthropic requests `

Next, ensure you have Ollama installed and the Llama 3 model pulled locally:

`bash ollama pull llama3 `

Step 1: Setting Up the Local Embedder and Route Definitions

We will use the all-MiniLM-L6-v2 model from Hugging Face. It is incredibly small (around 90MB), runs blazing fast on standard CPU hardware, and is highly accurate for semantic similarity tasks.

Create a new file named router.py and set up your route exemplars:

`python import numpy as np from sentence_transformers import SentenceTransformer

Initialize our super-fast, local embedding model embedder = SentenceTransformer('all-MiniLM-L6-v2')

Define exemplar prompts that typify each route LOCAL_EXEMPLARS = [ "Is this review positive or negative?", "Extract the phone number and email address from this text.", "Summarize this 200-word paragraph in bullet points.", "Translate the following phrase into Spanish.", "Correct the spelling and grammar in this sentence.", "Convert this CSV data into a JSON format." ]

CLOUD_EXEMPLARS = [ "Write a custom Python script to scrape a JavaScript-heavy website using Playwright.", "Explain the architectural differences between vector search and relational databases.", "Refactor this legacy React component to use TypeScript and modern hooks.", "Analyze this financial report and identify potential risks in the balance sheet.", "Help me debug this recursive function that is hitting a stack overflow.", "Draft a polite but firm response to a client refusing to pay their invoice." ]

Pre-compute the embeddings for our exemplars local_embeddings = embedder.encode(LOCAL_EXEMPLARS) cloud_embeddings = embedder.encode(CLOUD_EXEMPLARS) ```

Step 2: Building the Routing Logic

Now, we need a helper function to compute the cosine similarity between the incoming user prompt and our pre-computed exemplars. We will average the top similarity scores for each category to make a routing decision.

`python def calculate_similarity(vector_a, vector_b_list): # Helper to calculate cosine similarity against a list of vectors similarities = [] for vector_b in vector_b_list: dot_product = np.dot(vector_a, vector_b) norm_a = np.linalg.norm(vector_a) norm_b = np.linalg.norm(vector_b) similarity = dot_product / (norm_a * norm_b) similarities.append(similarity) return max(similarities) # We care about the single closest match

def route_prompt(user_prompt): # Embed the incoming prompt prompt_embedding = embedder.encode(user_prompt) # Get maximum similarity for both local and cloud routes local_score = calculate_similarity(prompt_embedding, local_embeddings) cloud_score = calculate_similarity(prompt_embedding, cloud_embeddings) print(f"\n[Router] Local Match Score: {local_score:.3f} | Cloud Match Score: {cloud_score:.3f}") if local_score >= cloud_score: return "local" else: return "cloud" `

Step 3: Integrating the Model Clients

With our routing decision in hand, we now need to actually talk to the models. We will configure a client for our local Ollama instance and another for the /platforms/claude API using the official SDK.

Make sure your environment variables are set up before running this code (e.g., export ANTHROPIC_API_KEY="your-key").

`python import os import requests from anthropic import Anthropic

Initialize the Anthropic client anthropic_client = Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))

def call_ollama(prompt): print("[Dispatcher] Routing to Local Ollama (Llama 3)... ") url = "http://localhost:11434/api/generate" payload = { "model": "llama3", "prompt": prompt, "stream": False } try: response = requests.post(url, json=payload) return response.json().get("response", "Error communicating with Ollama.") except Exception as e: return f"Ollama connection failed: {str(e)}"

def call_claude(prompt): print("[Dispatcher] Routing to Anthropic Claude (Sonnet 3.5)... ") try: message = anthropic_client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1000, temperature=0, messages=[ {"role": "user", "content": prompt} ] ) return message.content[0].text except Exception as e: return f"Claude API call failed: {str(e)}" `

If you run into issues authenticating with Anthropic's API or experience network dropouts, you can refer to the official troubleshooting steps at https://claude-support.com to verify your API status.

Step 4: Putting It All Together

Let’s build a clean, unified interface to test our router. We will feed it a variety of prompts—some simple, some highly complex—to see how accurately it classifies and processes them.

`python def execute_query(user_prompt): destination = route_prompt(user_prompt) if destination == "local": response = call_ollama(user_prompt) else: response = call_claude(user_prompt) print(f"[Result] Response:\n{response}\n{'-'*60}")

Test cases if __name__ == "__main__": # This should go to Ollama execute_query("Please extract all the email addresses from this string: 'test@example.com, support@company.org'") # This should go to Claude execute_query("Write a secure, production-ready Python script to encrypt password hashes using bcrypt. Explain the security trade-offs of salt rounds.") ```

Fine-Tuning Your Router

While the code above works beautifully right out of the box, you can make it even smarter: Adjust thresholds:* You can add a minimum confidence barrier. For example, if both similarity scores are below 0.3, default to Claude as a safe fallback. Keep updating your `/prompts` bank:* As your application evolves, capture real user interactions that were routed incorrectly and add them to your LOCAL_EXEMPLARS or CLOUD_EXEMPLARS lists. Expand the taxonomy:* You don't have to limit yourself to a binary decision. You can route to intermediate models (like Gemini Flash for cheap, long-context queries) by following a similar approach via /platforms/gemini.

Building a local semantic router ensures you only pay for premium intelligence when you actually need it. Your local machine handles the heavy lifting of mundane classification, saving your wallet for the hard problems.