Automating Pull Request Reviews With CodiumAI PR-Agent

Motivation

If you are a developer, you know how much time you spend on pull request reviews. We all try to write the best code we can. When we are done, we ask our team members to review our code. Sometimes, making a pull request is more complex than it sounds. Over time, teams learn that pull requests need to be standardized to speed things up and improve communication. That is why we have a pull request template, for example. Sometimes, even that is not enough. Because the speed at which the code is sent out is increasing. More pull requests are created. And the number of pull requests that need to be reviewed is expanding. Ah…

Question: In the era of AI, can we automate pull request reviews? At least it can dissect a pull request and give us some insights.

Let’s do this with some small pull requests.

Looking for the right tool

I need a tool or AI assistant that can do this for me. Obviously, as I have my code on GitHub, I need a tool that can integrate with GitHub. So, the first choice was GitHub’s Copilot for pull requests. And it is not free. It is still in beta, and you need to sign into the waitlist to access it. I’m not so patient, so I searched further.

Next, I found a solution CodiumAI PR-Agent. It is free and open-source. Just from its description, it looks like something that can fit my needs. I decided to give it a try.

Installation & usage

Requirements

Since I have my remote repository on GitHub to enable CodiumAI PR-Agent to access my code and do its magic, I need to create a GitHub token. Also, CodiumAI PR-Agent uses OpenAI API, which means I must also have an OpenAI API key. So, before you start, make sure you have both of them.

How to create a GitHub token

I hope you already have a GitHub account. If not, create one. Then follow these steps:

1. Go to your GitHub account settings
2. Click on Developer settings
3. Click on Personal access tokens
4. Click on Generate new token
5. Give your token a name and select scopes
6. Click on Generate token
7. Copy your token and save it somewhere safe
8. You will need it later

Or you can follow this link for more detailed instructions.

How to create OpenAI API key

1. Go to OpenAI and create an account
2. Go to your account settings
3. Click on API keys
4. Click on Create a new API key
5. Copy your API key and save it somewhere safe
6. You will need it later

Or go to this link to add an API key.

How to use Codium PR-Agent from CLI

Setup

There are two ways to do this: Docker or a Python virtual environment. I’ll be using Docker. So, to start using Codium PR-Agent from CLI, first set the above-received API keys as environment variables. You can do this by running these commands:

export GITHUB_TOKEN=<your_github_token>
export OPENAI_API_KEY=<your_openai_api_key>

Open a pull request on your GitHub repository. Copy the URL of your pull request and export it as an environment variable:

export PR_URL=<your_pr_url>

Then you can run Codium PR-Agent from CLI by running this command:

docker run --rm -it -e OPENAI.KEY=${OPENAI_API_KEY} -e GITHUB.USER_TOKEN=${GITHUB_TOKEN} codiumai/pr-agent:latest --pr_url ${PR_URL} <command>

Even better would be making an alias for this command:

alias pragent='docker run --rm -it -e OPENAI.KEY=${OPENAI_API_KEY} -e GITHUB.USER_TOKEN=${GITHUB_TOKEN} codiumai/pr-agent:latest --pr_url ${PR_URL}'

Now you can run Codium PR-Agent from CLI by running this command:

pragent <command>

Usage

Codium PR-Agent has several commands. Let’s start with ones related to pull requests.

Generate Pull Request Description

Let’s first start with generating a pull request description. To do this, run this command:

$ pragent describe --keep_original_user_title=true --add_original_user_description=true

...

This command will generate a description for your pull request and add it to your pull request. I want to keep your original description and title, so I added --keep_original_user_title=true and --add_original_user_description=true flags. You can omit these flags if you don’t want to keep your original title and description.

And the results are surprisingly good but also a bit disappointing. Let me explain why. The description, in my case, is 100% accurate. It describes my changes in three blocks: change type, description, and the pull request walkthrough. See the image below.

It even added a label to pinpoint what this pull request is about. Which is nice. This is opening some other possibilities. I would like to process a pull request based on its labels. Next time about that.

But it did change the original title and removed the original description, even though I told it not to!? These two options should be set by default and not vice versa. And Codium PR-Agent should add its description and title to the original ones. Why? First, the person who created the pull request should have written the title and description to add some context and motivation, add a link to a ticketing system, etc. But ok, I can live with that at the moment. In the image above, I marked the yellow parts updated by Codium PR-Agent: removed and updated the title and description. And with green parts that are added as I expected: the label.

Generate Pull Request Review

Ok, let’s move on to the next command. This command will generate a pull request review. To do this, run this command:

$ pragent review

...

Using the default configuration gives me nice results. Everything that needs to be addressed in a green box is nicely marked. The yellow box is why I wanted to keep the original title and description. The CodiumAI PR-Agent should just add its description next to the original one.

Give Me Some Suggestions

Let’s see what Codium PR-Agent can suggest to me. Since changes are simple, I’m going with the default configuration. To do this, run this command:

$ pragent improve

...

I have got three suggestions. Above, you just see one. Even if the cases are straightforward, suggestions are on point as clean code suggestions. I can accept or reject them as in any other pull request.

Generate Changelog And Update Docs

These two commands are interesting to me. Let’s start with the changelog. To do this, run this command:

$ pragent update_changelog

...

Running with the default configuration gives me a nice changelog. Even if I did not have one, it created one for me. However, it did not update the pull request. The reason is apparent: I did not instruct it to do so. But CodiumAI PR-Agent is smart enough to tell me so and give me instructions on how to do it. So I do it:

$ pragent update_changelog --pr_update_changelog.push_changelog_changes=true

...

Excellent, let us try updating docs:

$ pragent add_docs

...

They are added as code improvements. I can accept them or reject them as in any other pull request. Nice.

Troubleshooting

You can find documentation on CodiumAI PR-Agent GitHub repository is quite good. You’ll find most of your answers there. I want to point to one thing related to giving the correct permissions to CodiumAI PR-Agent when creating a GitHub API token. Which permissions you need to give depends on what you want to do with it.

For any integration, you should give CodiumAI PR-Agent only the permissions it needs to do its job. If you give it less permissions than it needs, it will not work. I’m fond of the principle of least privilege. So, I gave it only the permissions it needed to do its job.

How will you now? Use the CLI and see what it says. Here is an example:

$ pragent update_changelog --pr_update_changelog.push_changelog_changes=true

2023-12-10 18:20:45.902 | INFO     | pr_agent.algo.utils:update_settings_from_args:275 - Updated setting PR_UPDATE_CHANGELOG.PUSH_CHANGELOG_CHANGES to: "True"
2023-12-10 18:20:47.768 | INFO     | pr_agent.tools.pr_update_changelog:_get_changlog_file:152 - No CHANGELOG.md file found in the repository. Creating one...
Traceback (most recent call last):
  File "/app/pr_agent/tools/pr_update_changelog.py", line 144, in _get_changlog_file
...snip...
github.GithubException.UnknownObjectException: 404 {"message": "Not Found", "documentation_url": "https://docs.github.com/rest/repos/contents#get-repository-content"}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/pr_agent/cli.py", line 68, in <module>
...snip...
github.GithubException.GithubException: 403 {"message": "Resource not accessible by personal access token", "documentation_url": "https://docs.github.com/rest/repos/contents#create-or-update-file-contents"}

I did not give enough permissions to CodiumAI PR-Agent to update the changelog. So, it failed. I gave it the proper permissions, and it worked. Simple as that. I suggest you run all your scenarios with CLI first. It will save you a lot of time.

How to use Codium PR-Agent as GitHub Action

Now that I have tried my scenarios with CLI, I want to automate them. I want to use Codium PR-Agent as GitHub Action.

Setup CodiumAI PR-Agent GitHub Action

Thanks to the nice documentation on CodiumAI PR-Agent GitHub repository, I do this with ease. I just need to add a new workflow file to my repository. I named it codium-pr-agent.yml. Here is the content of the file:

name: PRAgent
on:
  pull_request:
    branches: ["*"]
  issue_comment:
    types: [created]

jobs:
  pr_agent_job:
    runs-on: ubuntu-latest
    permissions:
      issues: write
      pull-requests: write
      contents: write
    name: Run pr agent on every pull request, respond to user comments
    steps:
      - name: PR Agent action step
        id: pragent
        uses: Codium-ai/pr-agent@main
        env:
          OPENAI_KEY: ${{ secrets.OPENAI }}
          GITHUB_TOKEN: ${{ secrets.API_TOKEN }}
          pr_description.use_description_markers: "true"
          pr_description.include_generated_by_header: "true"
          github_action.auto_review: "true"
          github_action.auto_describe: "true"
          github_action.auto_improve: "true"
          pr_update_changelog.push_changelog_changes: "true"

Also, I added a pull request template to my repository in a .github/pull_request_template.md. Here is the content of the file:

## Describe your changes

## Issue ticket number and link

## Checklist before requesting a review
- [ ] I have performed a self-review of my code
- [ ] If it is a core feature, I have added thorough tests.
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation

## PR Description

pr_agent:summary

## PR Walkthrough

pr_agent:walkthrough

And remember to add your API keys as secrets to your repository. I named them OPENAI and API_TOKEN. Add them to your repository by going to your repository’s settings and then to Secrets. Click on Add a new secret and add your API keys. Read more on GitHub secrets here.

For additional CodiumAI-PR-Agent configuration options, you can check CodiumAI PR-Agent GitHub repository.

The above configuration will run CodiumAI PR-Agent on every pull request, use the pull request template to generate a pull request description, create a pull request review, and give me some suggestions. Nice. I got all I wanted. Next time a pull request is made on my repository, CodiumAI PR-Agent will do its magic. I can quickly do future reviews.

Gotchas

Even if all sounds nice, there is one gotcha. That is OpenAI. It is not free, and you’ll need to pay for it. To reduce costs, you need to spend some time configuring CodiumAI PR-Agent, which does only what you need on a limited set of files, primarily if you use code generation.

Conclusion

CodiumAI PR-Agent is an excellent tool. It can help you with your pull requests. It can generate a pull request description, review, give you some suggestions, update changelog and docs, and chit-chat with it if you want. See this nice /ask command and its answer. Check it yourself and see what it can do by setting up scenarios. It is worth it trying it.

I hope you enjoyed this article. If you have any questions or comments, please leave them below. I’ll be happy to answer them.