How to Build an Automated API Contract Tester with Gemini 1.5 Flash and GitHub Actions

Why API Changes Break Quietly

We have all committed code that we thought was harmless, only to receive a message from the frontend team complaining that an API response format has changed. Maybe you changed a snake_case key to camelCase, or turned an integer ID into a string.

Standard integration tests and unit tests are great for catching logic failures, but they are surprisingly bad at highlighting semantic contract drift. Unless you are running heavy, slow integration environments or writing tedious OpenAPI contract schemas by hand for every minor update, these interface mismatches will slip straight into production.

In this guide, we are going to build a lightweight, continuous integration tool that runs on every pull request. It uses Gemini 1.5 Flash because of its incredibly low cost and fast inference times. The tool will parse the existing "gold standard" schema file, compare it against your newly proposed codebase changes, and fail the CI run if it detects a breaking API contract change.

The Logic of Semantic API Inspection

Unlike traditional schema validation tools that require strict JSON Schema formats, an LLM-powered inspector can read natural changes. It understands that changing a field name from created_at to createdAt is a structural break, but adding a brand new optional field user_avatar_url is perfectly backward-compatible. This level of smart interpretation saves developer hours otherwise wasted on rigid schema definition maintenance.

Step 1: Writing the Parser Script

First, we need a small script that fetches the API schema definitions. In our example, we will assume your codebase has a static or auto-generated schema_v1.json (the production benchmark) and a newly generated schema_draft.json during build time. Alternatively, you can just feed two files to our checking script.

Let us initialize a Node.js project to handle this analysis.

`bash mkdir api-contract-tester && cd api-contract-tester npm init -y npm install @google/genai dotenv `

Create a file named check_contract.js:

`javascript import { GoogleGenAI, Type } from '@google/genai'; import * as fs from 'fs'; import dotenv from 'dotenv';

dotenv.config();

// We initialise the SDK using our Gemini credentials const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });

const SYSTEM_INSTRUCTION = ` You are a strict QA automated tester. Your role is to perform contract testing by comparing a target Production API schema with a proposed Draft schema. Identify breaking changes. A breaking change includes: - Deleting an existing field. - Changing a field's data type (e.g., String to Integer, or Array to Object). - Marking a previously optional field as required. - Renaming keys.

Non-breaking changes include adding new fields, or making existing required fields optional. Be objective. No fluff. `;

// Define a strict schema response so our CI engine can easily read the outcome const contractResponseSchema = { type: Type.OBJECT, properties: { is_breaking: { type: Type.BOOLEAN, description: "Set to true if there is any backward-incompatible schema change. Otherwise, false." }, violations: { type: Type.ARRAY, items: { type: Type.STRING }, description: "A detailed list of the broken contracts and field mismatches found." } }, required: ["is_breaking", "violations"] };

async function runContractCheck() { const productionSchema = fs.readFileSync('schemas/production.json', 'utf-8'); const draftSchema = fs.readFileSync('schemas/draft.json', 'utf-8');

const prompt = ` Analyze the differences between these two API schemas:

=== PRODUCTION SCHEMA === ${productionSchema}

=== PROPOSED DRAFT SCHEMA === ${draftSchema} `;

try { console.log("Sending API contracts to Gemini 1.5 Flash for evaluation..."); const response = await ai.models.generateContent({ model: 'gemini-1.5-flash', contents: prompt, config: { systemInstruction: SYSTEM_INSTRUCTION, responseMimeType: 'application/json', responseSchema: contractResponseSchema, temperature: 0.1, // Keep it deterministic } });

const result = JSON.parse(response.text);

if (result.is_breaking) { console.error("❌ API Contract Violation Detected!"); result.violations.forEach(v => console.error( - ${v})); process.exit(1); // Exit code 1 fails the GitHub Action runner } else { console.log("✅ No breaking API contract changes detected. Schema is backward-compatible."); process.exit(0); } } catch (error) { console.error("An error occurred while evaluating the contract:", error); process.exit(2); } }

runContractCheck(); `

Step 2: Designing the GitHub Action

To make this run automatically on every pull request, we need to declare a GitHub Action pipeline. Create a file structure inside your directory: .github/workflows/api-contract-check.yml.

This runner will generate your dynamic schema from your development branch, compare it with the production schema stored on your main branch, and use our Gemini script to evaluate compatibility.

`yaml name: API Contract Security Check

on: pull_request: branches: [ main ]

jobs: contract-check: runs-on: ubuntu-latest steps: - name: Checkout pull request code uses: actions/checkout@v3 with: path: draft-branch

- name: Checkout production benchmark code uses: actions/checkout@v3 with: ref: main path: prod-branch

- name: Setup Node.js uses: actions/setup-node@v3 node-version: '18'

- name: Install dependencies run: | cd draft-branch npm install

- name: Arrange schemas for analysis run: | mkdir -p draft-branch/schemas # Mock step: Copy actual schemas or run your schema generator tool cp prod-branch/schemas/production.json draft-branch/schemas/production.json cp draft-branch/schemas/draft.json draft-branch/schemas/draft.json

- name: Execute Gemini Contract Analysis run: | cd draft-branch node check_contract.js env: GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }} `

Ensure that you add your GEMINI_API_KEY to your repository's GitHub Secrets settings before testing. If you run into authorization issues or want to compare configuration options, browse our Gemini workspace articles.

Step 3: Verifying the System

Let us mock a breaking change scenario. In your /schemas/production.json file, define a simple endpoint structure:

`json { "/api/v1/users": { "get": { "response": { "user_id": "integer", "username": "string", "email": "string" } } } } `

Now, simulate a mistake in /schemas/draft.json by renaming user_id to userId and dropping the email field entirely:

`json { "/api/v1/users": { "get": { "response": { "userId": "string", "username": "string" } } } } `

When your pipeline runs or when you run the node script locally via node check_contract.js, Gemini 1.5 Flash will instantly parse these definitions and output a clean JSON failure listing that user_id was dropped, its replacement uses an incompatible type, and the email payload field is missing. The process will exit with code 1, safely halting your build pipeline before anything is deployed.

For more complex schema environments or to view real-world examples of system parameters, you can inspect their API details directly on Google's official platforms.