A Pitest badge without a Pitest service

In the past I wrote a complex & deeply flawed, hosted Python service to display a GitHub badge for Pitest mutation coverage. These days, there’s a far simpler method…

Line coverage tells you which code your tests run. Mutation testing tells you whether they’d notice if that code were wrong. Mutation testing makes small changes (“mutants”) to your bytecode, flipping a < to <=, deleting a method call, returning true instead of false and reruns your tests against each one. A mutant your tests catch is killed; one they don’t is survived. My Java emulator EmuRox and its faster-moving, AI-accelerated NES counterpart EmuRoxLab both lean on it heavily, as I think all Java code should. Especially AI-developed code!

Code coverage has Codecov to turn a report into a README badge. Pitest has no hosted equivalent, so I built one from three free pieces: a small Gradle task, GitHub Pages, and Shields.io. The result is a badge that also links to the full report.

1. A Gradle task that turns the report into badge JSON

Pitest writes build/reports/pitest/mutations.xml, one <mutation status="..."> per mutant. A pitestBadge task, written inline in build.gradle.kts, runs after pitest and writes a Shields endpoint JSON file:

tasks.register("pitestBadge") {
    description = "Substitute for the lack of a Pitest report service"
    dependsOn("pitest")

    doLast {
        val mutations = DocumentBuilderFactory.newInstance()
            .newDocumentBuilder()
            .parse(file("build/reports/pitest/mutations.xml"))
            .getElementsByTagName("mutation")

        val statuses = (0 until mutations.length)
            .map { mutations.item(it).attributes.getNamedItem("status").nodeValue }

        // Pitest's own "detected" statuses - the mutants a test actually caught
        val detectedStatuses = setOf("KILLED", "TIMED_OUT", "NON_VIABLE", "MEMORY_ERROR", "RUN_ERROR")
        val total = statuses.size
        val detected = statuses.count { it in detectedStatuses }
        val score = if (total == 0) 0 else ((detected.toDouble() / total) * 100).roundToInt()

        val color = when {
            score >= 80 -> "brightgreen"
            score >= 60 -> "yellow"
            else -> "red"
        }

        file("badges").mkdirs()
        file("badges/pitest.json").writeText("""
            {"schemaVersion": 1, "label": "mutation coverage", "message": "$score%", "color": "$color"}
        """.trimIndent())
    }
}

It needs no plugin or extra dependency: the JDK’s XML parser is enough.

2. A CI job that publishes it to GitHub Pages

A full mutation run takes around a quarter of an hour, so it doesn’t run on every push. The mutation job in .github/workflows/gradle.yml runs only when started by hand, or when the commit message contains +fullBuild. I put +fullBuild in PR titles, and GitHub copies the title into the merge commit, so every merge republishes the badge.

mutation:
  needs: build
  continue-on-error: true
  if: contains(github.event.head_commit.message, '+fullBuild') || github.event_name == 'workflow_dispatch'
  permissions:
    contents: read
    pages: write
    id-token: write
  environment:
    name: github-pages
  steps:
    # ...checkout, JDK, Gradle...
    - run: ./gradlew pitestBadge
    - run: |
        mkdir -p public/pitest
        cp badges/pitest.json public/pitest.json
        cp -r build/reports/pitest/* public/pitest/
    - uses: actions/upload-pages-artifact@v4
      with:
        path: public
    - uses: actions/deploy-pages@v4

One deployment publishes both the badge data (/pitest.json) and the full, browsable HTML report (/pitest/). continue-on-error keeps a timing-sensitive mutation run from turning the whole workflow red.

Two things to set up first. The repository’s Pages source must be set to GitHub Actions (Settings → Pages → Build and deployment → Source). And deploy-pages replaces the whole Pages site, so if the repository already publishes anything else to Pages, that needs copying into public/ as well.

3. A Shields endpoint badge in the README

The Shields.io endpoint badge renders any JSON in that schema, so the README points it at the Pages URL, and links the badge to the report:

Click the badge and you land on Pitest’s own report.

Day to day

Mutation testing is expensive, as it runs the test suite over and over against variations of the code. So during development, Pitest is scoped to just the package being worked on.

./gradlew pitest -PpitestScope=com.rox.apu.*

pitestScope isn’t a Pitest option, just a Gradle property that narrows targetClasses and targetTests, defaulting to the whole project. Scoping the tests as well as the classes is where most of the time saving comes from, as Pitest then only runs that package’s tests:

val pitestScope = (project.findProperty("pitestScope") as String?)?.split(",") ?: listOf("com.rox.*")

pitest {
    targetClasses.set(pitestScope)
    targetTests.set(pitestScope)
}

It takes a comma-separated list, so several packages can be checked at once, e.g. -PpitestScope=com.rox.apu.*,com.rox.ppu.*.

The badge then tracks the whole project, while each change is checked locally in a quicker development cycle.

Posts