Administrator setup guide

Install and lock down CodeMD GitHub PR Checks

Install CodeMD on only the repositories you choose, protect every source branch with a GitHub ruleset, and leave only the dedicated codemd-test-runs branch available for test execution.

Step 1

Before you start

  • You need permission to install GitHub Apps for the account or organization that owns the repository. Otherwise, GitHub will let you request approval from an owner.
  • GitHub Actions must be enabled if you want CodeMD Impacted Tests to run.
  • Choose whether CodeMD should access all repositories or only selected repositories. Selected repositories is the least-privilege choice.
  • Ask whether organization rulesets already protect the repository before adding another one.

Recommended: Start with one non-production repository, verify the complete flow, and then add more repositories.

Step 2

Install the CodeMD.dev GitHub PR Checks App

1

Open the installation page

Select Install CodeMD on GitHub. GitHub opens the CodeMD.dev PR Checks App installation screen.

2

Choose the owner

Select the personal account or organization that owns the repository. If installations require approval, choose Request and send it to an organization owner.

3

Choose repositories

Select Only select repositories, then choose each repository where CodeMD should run. Avoid All repositories unless that access is intentional.

4

Review and approve

Review the permissions GitHub displays, then select Install. An owner may need to approve updated permissions before test runs can start.

5

Confirm access

Open the CodeMD PR Checks Dashboard, sign in with GitHub, and confirm the selected repository is listed.

Step 3

Understand the requested permissions

GitHub App permissions cover an entire repository. GitHub does not provide branch-scoped GitHub App permissions, so CodeMD requests repository-level write permission for test execution.

PermissionWhy it is neededHow CodeMD uses it
Contents: read and writeRead the checked commit and update the test branch.Writes only to codemd-test-runs, never the default, source, feature, or pull-request branches.
Workflows: read and writePut the CodeMD test workflow on the test branch.Writes .github/workflows/codemd-tests.yml only on codemd-test-runs.
Actions: readWatch the test run and read its result.Reports pass, fail, and changed-line coverage.
Checks: read and writePost results on commits and pull requests.Creates and updates CodeMD check results.
Pull requests and metadata: readIdentify commits and changed files.Reads pull-request context; it does not edit or merge pull requests.

Defense in depth: The ruleset below is enforced by GitHub independently of CodeMD. It prevents the app from updating source branches even though GitHub grants Contents write at repository scope.

Step 4

Lock CodeMD out of source branches

This strong ruleset targets every branch except codemd-test-runs. Only the trusted people or teams on its bypass list can update protected branches. Never add CodeMD to that list.

1

Open repository rules

Open the repository and go to Settings → Rules → Rulesets → New ruleset → New branch ruleset.

2

Name and activate it

Name it Protect source branches from apps. Set enforcement to Active. If your plan offers Evaluate, you can review its effect first and activate it afterward.

3

Target all branches, with one exception

Under Target branches, include all branches. Add an exclusion for the exact branch codemd-test-runs. Without that exclusion, CodeMD cannot start tests.

4

Set the bypass list

Add only named, trusted people or teams that need to update ordinary branches. Add another trusted bot only if your workflow requires it. Do not add the CodeMD.dev PR Checks App. Avoid broad role-based bypasses when a named maintainer team will work.

5

Enable protections

  • Enable Restrict updates.
  • Enable Restrict deletions.
  • Keep force pushes blocked.
  • Optionally require pull requests, reviews, and status checks for critical branches.
6

Review, save, and test

Check the target preview and bypass list, then save. Verify that maintainers can still push feature branches and merge pull requests. Restricting all updates without the correct human/team bypass will also block legitimate work.

7

Inspect inherited rules

Review every active repository and organization ruleset. Rulesets combine, so an inherited rule that targets codemd-test-runs can still block tests.

Expected result: GitHub allows CodeMD to update codemd-test-runs, but rejects the app if it tries to update the default branch or another protected branch.

See GitHub's current documentation for creating repository rulesets and available rules and plan limits.

Step 5

Verify the complete setup

  1. Create a small branch and open a test pull request against the default branch.
  2. In the pull request's Checks area, confirm CodeMD Call-Path Security and CodeMD Impacted Tests appear.
  3. In the repository's Actions tab, confirm a CodeMD test run starts from codemd-test-runs.
  4. Under Code → Branches, confirm CodeMD updated only codemd-test-runs.
  5. Merge or close the test pull request using your normal process to confirm maintainers are not blocked.
  6. If available, check Settings → Rules → Insights for branch and actor activity.

Step 6

Troubleshooting

“Tests didn't run” or a permission error appears

An owner may not have accepted the current Contents, Workflows, and Actions permissions. Open GitHub App installations, choose CodeMD.dev PR Checks, and review pending permission requests.

No Actions run appears

Confirm Actions is enabled, codemd-test-runs is excluded from every restrictive ruleset, and organization policy permits the workflow.

The repository is missing

Open the app's Configure page and verify the repository is selected and the organization approved the installation.

Developers or merges are blocked

Restrict updates permits updates only from bypass actors. Add the appropriate trusted maintainer team or narrow protection to critical branches. Do not add CodeMD to the bypass list.

A required check never completes

Temporarily remove CodeMD from required status checks while diagnosing permissions or Actions policy. Restore it after a test pull request succeeds.

Step 7

Change or remove access

Open GitHub Settings → Applications → Installed GitHub Apps and select Configure for CodeMD.dev PR Checks.

  • Edit Repository access to add or remove repositories.
  • Select Uninstall to stop CodeMD entirely.
  • After uninstalling, CodeMD cannot read the repository or post checks. See the Privacy Policy for deletion and retention details.
  • You may delete codemd-test-runs after uninstalling if you do not need its history.