Back to Technical
Technical
AI AgentsDependabotGitHub ActionsGitHub AppDockerAutomationSecurity

Teaching an AI Agent to review Dependabot PRs without giving it my GitHub account

•Zaneta Gebka

At the client I work with, we are considering extending how we use AI in our workflows. Since I am interested in this area and have been running a few experiments of my own, I started thinking about what could be a realistic quick win.

A few ideas came to mind, but I picked one that felt like a good starting point: Dependabot development dependency updates.

Before suggesting anything at work, I wanted to try the idea in my own project first and understand where the risks actually are. So I picked this website - the one you are currently reading.

I started with the boring part: enabling Dependabot with a small, intentionally simple configuration.

version: 2

updates:
  - package-ecosystem: npm
    directory: /your-app
    schedule:
      interval: weekly
    open-pull-requests-limit: 10

The actual directory name is different, but the idea is the same: check npm dependencies weekly and do not let an unlimited number of update PRs pile up.

For now, Dependabot was only opening pull requests. Nothing was reviewed or merged automatically yet. I wanted to see what kind of updates appeared first, which checks they triggered and whether there was a safe enough subset to automate later.

I had a few Dependabot pull requests waiting in my portfolio repository. Most of them were small dependency updates, but small does not automatically mean harmless. And I guess you know what is happening here or there right now with many of dependecies... A package update can still break a build, change a peer dependency or simply arrive at the wrong moment. Or do even worse things.

I did not want an agent that merges everything just because a checkmark is green.

I wanted GitHub to decide which updates were even worth considering, while the agent could read the diff and release notes for that small group. If something was unclear, the decision should come back to me. And because the agent runs on my laptop, I also wanted to make sure it did not quietly inherit my normal GitHub access.

That became the first version of the workflow.

Why I chose Hermes

Once I knew what I wanted to automate, I had to decide where the agent part should live.

I was not looking for an agent that could work on the whole repository, write code or make general decisions about dependencies. I wanted something much smaller: an agent I could run locally, give a narrow skill, observe first and schedule later.

Hermes felt like a good fit for that experiment. It gave me profiles, custom skills, a local chat for trying the workflow manually and a cron scheduler once I was ready to let it run on its own.

The important part is that Hermes was not supposed to replace GitHub Actions. GitHub Actions handles the deterministic part: deciding whether a Dependabot PR is even eligible for review. Hermes handles the part where some context helps: reading the diff, checking the available release notes and deciding whether there is any reason to stop.

And the final GitHub action does not come directly from the model. It goes through a small wrapper that validates the PR again before commenting or merging it.

So this is not really “an AI agent with access to GitHub.” It is a narrow workflow where each part has a different job.

The first rule: the agent does not choose its own scope

The most important decision was made before I touched Hermes, prompts or GitHub credentials.

The first boundary lives in GitHub Actions. The agent only sees a pull request after GitHub has checked that it comes from Dependabot, changes exactly one direct development dependency, is a patch or minor update, and touches only package.json and package-lock.json.

The relevant part looks like this:

if [[
  "${dependency_count}" == "1" &&
  "${DEPENDENCY_TYPE}" == "direct:development" &&
  (
    "${UPDATE_TYPE}" == "version-update:semver-patch" ||
    "${UPDATE_TYPE}" == "version-update:semver-minor"
  ) &&
  "${changed_files}" == "${expected_files}"
]]; then
  gh pr edit "${PR_NUMBER}" \
    --repo "${GITHUB_REPOSITORY}" \
    --add-label "agent-review"
else
  gh pr edit "${PR_NUMBER}" \
    --repo "${GITHUB_REPOSITORY}" \
    --remove-label "agent-review" || true
fi

The agent-review label is not approval to merge. It is only permission to start a review. That means the agent does not look at every dependency update and decide on its own that something feels safe.

Two Dependabot pull requests, where only the eligible Jest environment update has the agent-review label.
Only explicitly eligible Dependabot PRs get the agent-review label.

First I made the agent observe

Before giving the agent any write access, I used it as a reviewer.

The first version of the skill did not make any changes. It started by listing open pull requests and checking the scope again instead of trusting the label blindly:

labels = {label["name"] for label in pull_request["labels"]}

if (
    pull_request["base"]["ref"] == "main"
    and "agent-review" in labels
    and not pull_request["draft"]
    and pull_request["user"]["login"] == "dependabot[bot]"
):
    candidates.append(
        {
            "number": pull_request["number"],
            "title": pull_request["title"],
            "head_sha": pull_request["head"]["sha"],
        }
    )

This is intentionally repetitive. The GitHub Action decides which PRs may be considered, but the review script still confirms that it is looking at an open, non-draft Dependabot PR targeting main.

For each labeled PR, it gave me a compact picture of what was actually happening: who opened it, which dependency and versions were changing, which files were touched and whether the visible CI checks had finished successfully. It also called out anything that still needed a human look.

That stage was useful because it showed me where assumptions were hiding. A nice-looking Dependabot title is not enough. I needed to know whether the change was really limited to the files I expected and whether the relevant CI job had actually run.

I also started with real, boring examples instead of an artificial test PR. One update to @testing-library/react had a successful lint, test and build check. Another update to jest-environment-jsdom looked structurally similar but did not have that CI check at all.

Those two PRs later became my ACCEPT and HOLD examples.

A prompt is not a safety boundary

It would have been easy to write a prompt saying: “merge only safe PRs.” I do not think that is enough.

The agent can make a judgement call after reading release notes, but the mechanical rules should not depend on whether it remembers every line of a prompt. So I wrote a small Python wrapper around the GitHub API with three commands:

evaluate PR_NUMBER
hold PR_NUMBER
merge PR_NUMBER

evaluate checks the allowed files, the direct development dependency change, the SemVer update type, mergeability and required checks. hold posts one comment for a given head SHA and refuses to duplicate it. merge re-fetches the PR and validates it again immediately before a squash merge.

The important part was that merge did not trust an earlier answer from the agent. Before it called GitHub, it evaluated the current state of the PR again:

def squash_merge(number: str, dry_run: bool):
    token = create_installation_token()
    result = evaluate(number, token)

    if not result["automation_eligible"]:
        raise SystemExit(
            "Refusing to merge: " + " ".join(result["reasons"])
        )

    body = {
        "sha": result["head_sha"],
        "merge_method": "squash",
        "commit_title": f"chore(deps): merge Dependabot PR #{number}",
    }

    if dry_run:
        return {"action": "would_squash_merge", "request": body}

    return github_request(
        "PUT",
        f"/repos/{OWNER}/{REPOSITORY}/pulls/{number}/merge",
        token,
        body,
    )

That last part matters. A Dependabot branch can change between a review and a merge. The merge command is pinned to the reviewed commit SHA, not just the PR number.

The agent still has a job: inspect the concise diff and official release notes. But it cannot skip the deterministic gate just because it sounds confident.

HOLD should be useful, not silent

The jest-environment-jsdom PR did not have the required Lint, test and build check. The correct answer was not “probably fine.” It was HOLD.

Instead of failing quietly, the wrapper left a short comment mentioning me and naming the technical reason. It also included a hidden marker with the current SHA, so the daily job will not leave the same comment again tomorrow.

A GitHub comment explaining that automated merge was paused because the required lint, test and build check is missing.
A HOLD is a request for a human decision, not a failed attempt to automate one.

The other PR passed the deterministic checks and the release note review. The agent merged it with a squash merge. Seeing one actual ACCEPT and one actual HOLD was much more useful than only testing the happy path.

Why I used a GitHub App

Reading pull requests was no longer enough once the workflow could act on them. The agent needed a GitHub identity that could read checks, leave a HOLD comment and, in a very narrow case, merge a PR.

My first attempt used the GitHub CLI and a personal token. It was convenient because I was already authenticated on my laptop. But that was exactly the problem: the agent would be acting as me.

A personal token is tied to a person, usually lives for longer and can easily end up with broader access than one small automation actually needs. Even if I told the agent to use it only for one repository, the credential itself would still be more powerful than the workflow.

A GitHub App gave me a separate identity for this one job. I installed it only on zanetagebka/my_website and gave it only the permissions the workflow needs:

  • read pull requests, checks and commit statuses,
  • write an issue comment when a PR is on HOLD,
  • merge an accepted pull request.

The app has its own private key, stored locally and mounted into the agent container as a read-only file. When the wrapper runs, it uses that key to request a short-lived installation token. The token is limited to the repository where the app is installed.

This does not make the automation safe on its own. The wrapper and review policy still matter. But it gives the workflow a much clearer boundary: the agent does not borrow my normal GitHub account, and removing the app from the repository immediately removes its access.

The security problem I almost ignored

By then, the GitHub API side was working. An environment check revealed another issue.

Although Hermes had a separate profile, its local terminal could still see my normal HOME directory. That meant gh auth status was authenticated with my usual GitHub CLI account.

Technically, the skill told the agent not to use gh. But “please do not use this powerful credential” is not the same as making it unavailable.

So I stopped before enabling the scheduled job.

terminal:
  backend: docker
  cwd: /workspace
  docker_mount_cwd_to_workspace: false
  docker_forward_env: []
  docker_volumes:
    - /path/to/github-app.pem:/run/github-app.pem:ro

The important part was not Docker itself, but what the container did not receive. I did not mount my local repository or home directory, and I did not forward environment variables from my laptop. The only credential available inside the container was the GitHub App private key, mounted read-only.

Inside the container, gh auth status is not authenticated, while the GitHub App wrapper can still create a short-lived installation token and read the checks it needs. That was the boundary I wanted before allowing unattended merges.

A daily job, not a robot with unlimited authority

Finally, I scheduled the agent to run once a day at 10:00 in my timezone. It lists eligible PRs, evaluates them, reads the available official release notes and chooses ACCEPT or HOLD according to the same policy.

The Hermes scheduled job named dependabot-review-merge, configured to run daily at 10:00 with a prompt restricting its actions to the Dependabot review skill.
The scheduled task has a deliberately narrow prompt and runs once a day.

I kept the policy deliberately narrow. I am not asking it to handle major upgrades or production dependencies, and it ignores anything without the agent-review label. A missing CI check, a failing status or unclear release notes are enough to stop it.

It does not use auto-merge or change repository settings either. If a PR falls outside those boundaries, the job leaves it for me instead of trying to be helpful.

What I learned

The interesting part was not teaching an agent to press “merge.” It was deciding what the agent should never be allowed to decide.

The LLM is useful for reading a diff and putting release notes into context. But labels, file allowlists, CI requirements, SHA checks and repository-scoped credentials are what make the system predictable.

This is still an experiment. I want to watch it for a while before expanding the policy. Maybe I will eventually allow more packages or another kind of low-risk update. Maybe I will find that the daily review is enough and leave it small.

For now, I am happy with a system that can save me a repetitive review, knows when to stop, and does not quietly inherit my whole GitHub account in the process.