How to build a lightweight prompt evaluation rig in Python

Let’s face it: prompt engineering is mostly vibes. You write a prompt, run it three times in the web playground, decide the output looks “good enough,” and ship it to production. Then, two days later, a user inputs a slightly unusual query and the whole thing falls apart.

So, you tweak the prompt to fix that specific edge case. But how do you know your tweak didn’t quietly ruin the outputs for the other ninety-nine cases?

You don’t. Not unless you are systematically testing your prompts against a solid evaluation dataset.

You do not need to buy an enterprise-grade, eye-wateringly expensive Prompt-Ops platform to solve this. You can build a lightweight, local evaluation rig in under fifty lines of Python. This guide will show you how to set up an automated testing harness that runs your prompt variations across multiple test cases, calls both /platforms/openai and /platforms/claude, and uses an “LLM-as-a-judge’ to objectively score the results.

The Architecture of a Local Eval Rig

Our lightweight rig consists of three simple components: 1. The Test Suite (`tests.json`): A collection of inputs and the ideal criteria we expect in the output. 2. The Runner (`eval.py`): A script that sends the inputs and prompts to your chosen models. 3. The Judge: A robust model (like GPT-4o or Claude 3.5 Sonnet) programmed to grade the outputs on a scale from 1 to 5 based on your criteria.

This setup allows you to test your prompts against dozens of edge cases in seconds, giving you hard data on whether your changes actually improved the system or just shifted the failure points somewhere else.

Step 1: Defining Your Test Cases

Create a file named tests.json. Do not make your test cases too easy. You want to include standard queries, but also the weird, borderline-unreasonable inputs that usually make LLMs hallucinate or break formatting.

`json [ { "id": "case_01", "input": "Suggest three vegetarian high-protein dinners that can be made in under 15 minutes.", "criteria": "The response must list exactly three meals, all must be vegetarian, all must take under 15 minutes, and each must specify the estimated protein count." }, { "id": "case_02", "input": "What are the main causes of the French Revolution? Keep it to a single concise paragraph of under 80 words.", "criteria": "The response must be a single paragraph, strictly under 80 words, and mention economic hardship or social inequality." } ] `

Step 2: Writing the Evaluation Runner

Now, let’s write the Python script to run these tests. We will use the official SDKs. If you run into API key configuration issues, you can check the troubleshooting guides at https://claude-support.com or https://www.openai-support.com.

First, make sure you have your environment variables set and libraries installed:

`bash pip install openai anthropic export OPENAI_API_KEY="your_key_here" export ANTHROPIC_API_KEY="your_key_here" `

Create your eval.py script:

`python import json import os from openai import OpenAI from anthropic import Anthropic

Initialize clients openai_client = OpenAI() anthropic_client = Anthropic()

Load the system prompt we want to test SYSTEM_PROMPT_V1 = """ You are a precise, helpful assistant. When asked to limit your response length or format, you must follow the constraint perfectly. """

def get_openai_response(system_prompt, user_input): response = openai_client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_input} ], temperature=0.0 # Keep temperature at 0 for deterministic testing ) return response.choices[0].message.content

def get_claude_response(system_prompt, user_input): response = anthropic_client.messages.create( model="claude-3-5-haiku-20241022", max_tokens=1000, system=system_prompt, messages=[{"role": "user", "content": user_input}], temperature=0.0 ) return response.content[0].text `

Step 3: Implementing the LLM Judge

To make this rig work without human bottlenecking, we need an automated judge. We will use a separate, high-tier model call to evaluate the outputs. This is where we see what truly makes our prompts tick.

Add this helper function to your eval.py:

`python def judge_output(user_input, model_output, criteria): judge_prompt = f""" You are an unbiased, highly critical quality control agent. Your job is to rate an AI assistant's response based on a set of criteria. User Input: {user_input} Assistant Response: {model_output} Evaluation Criteria: {criteria} Provide your response in the following format: Score: [1 to 5, where 5 is perfect compliance and 1 is a total failure to meet the criteria] Reasoning: [A one-sentence explanation of why you gave this score] """ response = openai_client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": judge_prompt}], temperature=0.0 ) return response.choices[0].message.content `

If you want to construct more complex grading routines, you can use our /prompts to build custom system instructions for your LLM judge.

Step 4: Putting it All Together

Now, write the main loop to run through your tests, execute the prompts on your target models, send the results to the judge, and print a clean scorecard.

`python def main(): with open("tests.json", "r") as f: test_cases = json.load(f) print("--- Starting Prompt Evaluation ---\n") for case in test_cases: print(f"Running Case {case['id']}...") # Test GPT-4o-mini gpt_output = get_openai_response(SYSTEM_PROMPT_V1, case['input']) gpt_eval = judge_output(case['input'], gpt_output, case['criteria']) # Test Claude Haiku claude_output = get_claude_response(SYSTEM_PROMPT_V1, case['input']) claude_eval = judge_output(case['input'], claude_output, case['criteria']) print(f"\nInput: {case['input']}") print(f"--- GPT-4o-mini ---\nOutput: {gpt_output}\nEvaluation: {gpt_eval}\n") print(f"--- Claude Haiku ---\nOutput: {claude_output}\nEvaluation: {claude_eval}\n") print("="*40 + "\n")

if __name__ == "__main__": main() `

Why This Beats manual "Vibe Checks"

Running this script gives you an instant, structured output of how well your prompts performed under pressure. If you want to update your system prompt (say, to add a clause about tone or safety), you can simply change SYSTEM_PROMPT_V1 to your new version and rerun the script.

If your average score across your ten test cases drops from 4.8 to 3.2, you know immediately that your “fix” caused regression failures elsewhere. You can tweak your system instructions until you hit a perfect five-star sweep across the board.

No bloated SaaS platforms, no unnecessary complexity. Just clean code, fast feedback loops, and prompts that you can actually trust in production.