How to Build an Automated Prompt Evaluation Pipeline with Gemini 1.5 Flash and Python

Why Eyeballing Prompt Updates is Slowing You Down

You edit a system prompt in your application to fix one specific edge case, run it once in your local terminal, and it works. Satisfied, you push it straight to production. Two hours later, your users are reporting that the entire system is hallucinating, and your structured JSON output has completely broken.

We have all been there. Prompt engineering without validation is just guessing.

To move fast, you need an evaluation workflow that systematically tests your prompts against a defined test suite whenever you make a change. By choosing Gemini 1.5 Flash, you get incredibly low latency and tiny API bills, making it the perfect engine to run high-volume evaluation suites on every commit.

In this guide, we will build a command-line evaluation pipeline in Python to run assertions over your prompts, ensuring your updates always tick all the right boxes.

Setting Up Your Testing Environment

We will use Python along with the official Google GenAI library. If you need some help getting your initial API keys, jump over to our developer quickstart at /platforms/gemini.

First, install the required packages:

`bash pip install google-generativeai pydantic `

Make sure your Gemini API key is configured in your terminal environment:

`bash export GEMINI_API_KEY="your-gemini-api-key-here" `

Step 1: Defining Your Test Cases

An evaluation framework requires three components: the input you want to test, the current prompt you are evaluating, and the criteria for success.

Let's model this using Pydantic to ensure our test cases remain robustly structured. Create a file named run_evals.py and start with the following models:

`python from pydantic import BaseModel from typing import List, Dict import json import os import google.generativeai as genai

class TestCase(BaseModel): id: str user_input: str expected_keywords: List[str] forbidden_keywords: List[str] minimum_length: int = 10 `

Now, let's create a small collection of sample test cases. In a real-world project, you would store these in a JSON file inside your test suite folder.

`python TEST_SUITE = [ TestCase( id="tc_01", user_input="Can you explain how refunding an order works?", expected_keywords=["refund", "policy", "days", "original payment"], forbidden_keywords=["crypto", "bitcoin", "no refunds"], minimum_length=30 ), TestCase( id="tc_02", user_input="Where is my tracking number?", expected_keywords=["tracking", "email", "shipping", "carrier"], forbidden_keywords=["refund", "credit card"], minimum_length=20 ) ] `

Step 2: Designing Your Candidate Prompt

This is the prompt we want to test. Suppose we are building a customer service bot, and we have updated its instructions. We want to test whether this candidate prompt adheres to our quality standards across all test inputs.

`python CANDIDATE_SYSTEM_PROMPT = """ You are a helpful customer support agent for our e-commerce platform. Always reply politely and professionally. Provide instructions on refunds or shipping tracking based on standard policy (30-day refunds back to the original payment method, and tracking links are sent automatically via email within 24 hours of dispatch). Do not mention cryptocurrency or unsupported payment processors. """ `

If you want inspiration for structuring candidate prompts, read our in-depth prompt engineering guides on /prompts.

Step 3: Writing the Evaluation Runner

Our runner needs to take each test case, run it through Gemini 1.5 Flash using our system prompt, and then evaluate the response. We will write explicit assertion checks for length, expected keywords, and forbidden words.

`python def run_llm_inference(system_prompt: str, user_input: str) -> str: # Initialise the Google Generative AI client genai.configure(api_key=os.environ.get("GEMINI_API_KEY")) # Use the ultra-fast gemini-1.5-flash model model = genai.GenerativeModel( model_name="gemini-1.5-flash", generation_config={"temperature": 0.1}, system_instruction=system_prompt ) response = model.generate_content(user_input) return response.text.strip()

def evaluate_test_case(test: TestCase, response_text: str) -> Dict: failed_rules = [] # Check length if len(response_text) < test.minimum_length: failed_rules.append(f"Length too short ({len(response_text)} chars vs expected >={test.minimum_length})") # Check expected keywords for keyword in test.expected_keywords: if keyword.lower() not in response_text.lower(): failed_rules.append(f"Missing expected keyword: '{keyword}'") # Check forbidden keywords for keyword in test.forbidden_keywords: if keyword.lower() in response_text.lower(): failed_rules.append(f"Found forbidden keyword: '{keyword}'") return { "id": test.id, "passed": len(failed_rules) == 0, "errors": failed_rules, "output": response_text } `

Step 4: Executing the Suite and Outputting Results

Let's write a main execution block that processes the tests, prints formatting to our terminal, and signals a non-zero exit code if any test fails. This is crucial if you want to plug this pipeline directly into automated testing environments like GitHub Actions.

`python def main(): if not os.environ.get("GEMINI_API_KEY"): print("Error: GEMINI_API_KEY environment variable is missing.") exit(1) print(f"Starting evaluation run against {len(TEST_SUITE)} test cases...\n") all_passed = True for test in TEST_SUITE: print(f"Running test {test.id}...") response = run_llm_inference(CANDIDATE_SYSTEM_PROMPT, test.user_input) result = evaluate_test_case(test, response) if result["passed"]: print(f"[✅] {test.id} PASSED") else: print(f"[❌] {test.id} FAILED") for error in result["errors"]: print(f" - {error}") print(f" Actual Output: {result['output'][:100]}...") all_passed = False print("-" * 40) if all_passed: print("All tests passed successfully!") exit(0) else: print("Some evaluation tests failed. Check your candidate prompt constraints.") exit(1)

if __name__ == "__main__": main() `

Running the Evaluator

Save the script and run it via your terminal:

`bash python run_evals.py `

Because Gemini 1.5 Flash is highly optimized for raw speed, your test suite will execute in just a couple of seconds. You will see a clean visual summary of which test assertions succeeded and which failed.

If you find your requests are timing out, or if you need to integrate structured JSON schema validation into your evals, explore our diagnostic resources over at /platforms/gemini/articles for advanced tips on SDK configurations. Stop eyeballing and start validating.