How to Build a Lightweight Automated Pull Request Auditor Using GitHub Actions and OpenAI

The Pull Request Bottleneck

Let’s be honest: code reviews in most engineering teams are broken. Instead of focused discussions on API design, architectural patterns, or hidden security loopholes, PR threads frequently devolve into bikeshedding. We waste hours arguing about stylistic preferences, or worse, we miss critical logical vulnerabilities because we got distracted by syntax.

While traditional linters handle basic formatting issues, they cannot understand intent. They cannot tell you if your colleague is accidentally exposing database transactions to potential race conditions, or if they are bypassing standard authorisation protocols in a newly introduced controller.

We can automate this process using an AI agent built directly into your continuous integration workflow. In this tutorial, we will build a custom GitHub Action running a lightweight Python script that queries OpenAI's API (using the cost-efficient GPT-4o, which you can read about in our [/platforms/openai] guide). It will analyse git diffs, run a context-aware architectural audit, and write feedback comments directly back into the pull request.

The Architecture

Our automated PR auditor works like this: 1. A developer opens or updates a Pull Request. 2. A GitHub Action is triggered, extracting the git diff of the branch. 3. A Python script processes the diff, removing noisy artifacts (like lockfile changes) to keep context clear. 4. The script formats the code changes and sends them to OpenAI, querying for security vulnerabilities, architectural consistency, and structural bugs. 5. The auditor posts its feedback cleanly as a markdown comment on the pull request.

To understand terms like context windows and system prompt structure before we write the code, read through our [/glossary] definition of prompt parameters.

Step 1: Writing the Python Audit Script

We will write a script called audit_diff.py. This script will use the official Python OpenAI client and interact with GitHub’s REST API using basic environment variables provided by the GitHub Action environment.

Create a new file called audit_diff.py in your repository:

`python import os import sys import urllib.request import json from openai import OpenAI

Ensure environment variables are loaded github_token = os.getenv("GITHUB_TOKEN") repo = os.getenv("GITHUB_REPOSITORY") pr_number = os.getenv("PR_NUMBER") openai_api_key = os.getenv("OPENAI_API_KEY")

if not all([github_token, repo, pr_number, openai_api_key]): print("Error: Missing environment variables.") sys.exit(1)

Initialise OpenAI client client = OpenAI(api_key=openai_api_key)

Fetch the Git Diff from GitHub's REST API def fetch_pr_diff(repo, pr_number, token): url = f"https://api.github.com/repos/{repo}/pulls/{pr_number}" headers = { "Authorization": f"token {token}", "Accept": "application/vnd.github.v3.diff" } req = urllib.request.Request(url, headers=headers) try: with urllib.request.urlopen(req) as response: return response.read().decode('utf-8') except Exception as e: print(f"Failed to fetch diff: {e}") sys.exit(1) ```

Step 2: Filtering the Diff and Structuring the Prompt

Git diffs can get massive. If someone upgrades their packages, the diff of package-lock.json or poetry.lock will quickly blow past model token limits and waste API spend on useless lockfile metadata. We must filter out non-essential changes before sending our payload to the LLM.

Add this helper function to your audit_diff.py file:

`python def filter_diff(raw_diff: str) -> str: lines = raw_diff.splitlines() filtered_lines = [] skipping = False for line in lines: if line.startswith("diff --git"): # Skip lock files, assets, and config maps if any(x in line for x in ["package-lock.json", "yarn.lock", "poetry.lock", ".svg", ".png"]): skipping = True else: skipping = False if not skipping: filtered_lines.append(line) return "\n".join(filtered_lines)[:15000] # Hard character cap to control token spend `

Next, let’s construct our evaluation logic. We will explicitly tell GPT-4o to act as a senior software architect focused strictly on architectural patterns, security issues (like credentials in code or missing sanitisation), and performance pitfalls. We instruct it to be brief and constructive.

`python def audit_code_changes(diff_content: str) -> str: system_prompt = """ You are an expert software architect and security auditor. Analyse the provided git diff and provide a highly targeted code review. Focus solely on: 1. Critical security vulnerabilities (credentials, SQL injections, open endpoints). 2. Structural architectural violations (breaking modular design patterns, raw DB queries in controller files). 3. Blatant performance regressions. Do not comment on style rules, tabs vs spaces, variable names, or minor nits. If the changes look great, reply with a simple thumbs up or confirm there are no critical issues. Keep formatting clean, using bold headers and markdown bullet points. """ response = client.chat.completions.create( model="gpt-4o", temperature=0.2, # Kept low for highly analytical, deterministic output messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": f"Please audit the following codebase diff:\n\n{diff_content}"} ] ) return response.choices[0].message.content `

If you find your reviews are failing or the payload triggers rate-limiting errors from the model endpoints, check our troubleshooting documentation at [/platforms/openai/articles] to adjust your threshold settings.

Step 3: Posting Feedback Back to GitHub

Once the AI has generated its audit summary, the Python script needs to post this evaluation directly onto the PR. We will do this using a basic HTTP POST request against GitHub's issue comments API endpoint.

`python def post_github_comment(repo, pr_number, comment, token): url = f"https://api.github.com/repos/{repo}/issues/{pr_number}/comments" headers = { "Authorization": f"token {token}", "Content-Type": "application/json", "Accept": "application/vnd.github.v3+json" } body = json.dumps({"body": f"### 🤖 Automated Pull Request Auditor\n\n{comment}"}).encode('utf-8') req = urllib.request.Request(url, data=body, headers=headers, method="POST") try: with urllib.request.urlopen(req) as response: print("Audit comment posted successfully.") except Exception as e: print(f"Failed to post comment: {e}")

if __name__ == "__main__": print("Fetching PR diff...") raw_diff = fetch_pr_diff(repo, pr_number, github_token) print("Filtering diff payload...") clean_diff = filter_diff(raw_diff) if not clean_diff.strip(): print("No relevant code changes found to audit.") sys.exit(0) print("Querying OpenAI Audit Engine...") audit_report = audit_code_changes(clean_diff) print("Posting report feedback...") post_github_comment(repo, pr_number, audit_report, github_token) `

Step 4: Setting Up the GitHub Action Workflow

To make this run automatically on every single Pull Request, create a directory in your project root called .github/workflows and create a file named pr_auditor.yml:

`yaml name: Automated Code Auditor

on: pull_request: types: [opened, synchronize]

jobs: audit: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v4

- name: Set up Python uses: actions/setup-python@v5 with: python-node-version: '3.11'

- name: Install dependencies run: pip install openai

- name: Run audit script env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} PR_NUMBER: ${{ github.event.pull_request.number }} GITHUB_REPOSITORY: ${{ github.repository }} run: python audit_diff.py `

Customisation and Best Practices

Before you let this script loose on your main team repositories, make sure to add your OPENAI_API_KEY to your repository's GitHub Action Secrets setting.

By running this lightweight system inside your developer loops, you’ll free up your engineering team to focus on the high-level, human aspects of code review. No more endless, formatting debates—just solid, automated architecture audits built with OpenAI.