Auto-generating a Maester config from your custom Pester tests

If you use Maester to continuously test the security configuration of a Microsoft 365 tenant, you have probably discovered that it happily runs your own Pester tests next to the built-in ones. That is where it gets powerful: you write a test once, and Maester runs it, scores it, and drops it into a nice report.

There is one small piece of friction, though. Maester lets you override the severity and the title of every test through a maester-config.json file. Maintaining that JSON by hand is tedious, error-prone, and worst of all it silently drifts out of sync with your actual tests the moment someone adds a new one.

This post shows a tiny PowerShell script that removes that friction entirely: it reads your test files, figures out which tests exist, looks up the severity for each of them, and writes a fresh maester-config.json for you. Run it as part of your build and the config can never fall behind again.

The problem in one picture

Maester's per-test overrides live in a file that looks roughly like this:

{
  "TestSettings": [
    {
      "Id": "MFA-001",
      "Severity": "Critical",
      "Title": "MFA-001 - Require MFA for privileged roles"
    },
    {
      "Id": "SPO-004",
      "Severity": "Medium",
      "Title": "SPO-004 - Restrict external sharing"
    }
  ]
}

Every object binds a test Id to a Severity (Critical, High, Medium, Low, Info) and a display Title. Maester uses those values when it renders the report and when it decides how loud a failure should be.

Now imagine a repository with dozens of custom tests spread across many files. Keeping this list correct by hand means:

Remembering to add an entry every time you write a test.
Removing entries for tests you deleted.
Keeping the title in the JSON identical to the title in the test.
Never fat-fingering a severity.

That is exactly the kind of bookkeeping a computer should do.

The idea

The trick is to treat your test files as the single source of truth and generate the config from them. That only works if your tests follow a small, predictable naming convention. A convention like this is easy to adopt and pays off immediately:

Describe 'Evaluating multi-factor authentication' {

    It 'MFA-001: Require MFA for privileged roles' -Tag "Entra ID", "Access control" {
        # ... your assertions ...
    }
}

The important part is the It line. Each test starts with a short, structured code (MFA-001), a colon, and a human-readable description. That single line gives us everything we need:

The part before the colon (MFA-001) becomes the Id.
The part after the colon becomes the description.

The script then builds the title as Id - description, so MFA-001: Require MFA for privileged roles becomes the title MFA-001 - Require MFA for privileged roles. Repeating the code in the title is deliberate: it means the Id is visible at a glance in the Maester report, not just in the raw config. If you would rather show only the description, drop the "$id - " prefix in the script — it is a one-line change.

The only thing still missing is the severity, and that lives in a separate lookup so that non-engineers can own it. More on that below.

A place to keep severities

Rather than hard-coding severities in the script, keep them in a simple mapping file. A two-column CSV is more than enough and can be edited by anyone, including people who do not touch PowerShell:

Id;Severity
MFA-001;Critical
SPO-004;Medium
EXO-002;High
LOG-001;Info

Tip: In practice this mapping file is a great place to also record why a control has a given severity, or to link it to whatever framework you report against (CIS, ISO 27001, NIS2, your own internal baseline). The script below only cares about the Id and Severity columns, so you can add as many extra columns as you like without changing a single line of code.

The script

Here is the whole thing. It is deliberately small, a little over fifty lines of PowerShell with no external modules.

# Stop on any error
$ErrorActionPreference = "Stop"

# Folder layout assumed here:
#   .\generate-maester-config.ps1   <- this script
#   .\severity-mapping.csv          <- Id;Severity lookup
#   ..\tests\Custom\*.Tests.ps1     <- your custom Pester tests

$scriptRoot = $PSScriptRoot
$repoRoot   = Split-Path -Parent $scriptRoot

# 1. Load the severity lookup into a hashtable for fast access
$csvPath = Join-Path -Path $scriptRoot -ChildPath "severity-mapping.csv"

$severityLookup = @{}
Import-Csv -Path $csvPath -Delimiter ";" | ForEach-Object {
    $severityLookup[$_.Id] = $_.Severity
}

# 2. Prepare the result set and the regex that recognises a test
$testSettings = [System.Collections.Generic.List[object]]::new()

# Matches:  It 'CODE-123: Some description' ...
$pattern = "^\s*It\s+'([A-Z]+-\d+):\s*(.*?)'"

# 3. Find every custom test file
$testsPath = Join-Path -Path $repoRoot -ChildPath "tests\Custom"
$files = Get-ChildItem -Path $testsPath -Recurse -Filter *.Tests.ps1

foreach ($file in $files) {
    foreach ($line in (Get-Content $file.FullName)) {
        if ($line -match $pattern) {
            $id    = $matches[1]
            $desc  = $matches[2]
            $title = "$id - $desc"

            # Look up the severity, fall back gracefully if it's missing
            $severity = $severityLookup[$id]
            if (-not $severity) {
                $severity = "Unknown"
                Write-Warning "No severity found for $id"
            }

            $testSettings.Add([PSCustomObject]@{
                Id       = $id
                Severity = $severity
                Title    = $title
            })

            Write-Host "  [+] $title ($severity)"
        }
    }
}

if ($testSettings.Count -eq 0) {
    Write-Warning "No tests found. Does your It-pattern look like  It 'CODE-123: Description' ?"
    return
}

# 4. Build the JSON and write it next to your tests
$json = @{ TestSettings = $testSettings } | ConvertTo-Json -Depth 3
$outputPath = Join-Path -Path $testsPath -ChildPath "maester-config.json"
$json | Out-File -FilePath $outputPath -Encoding UTF8

Write-Host "`nDone. $($testSettings.Count) test(s) written to maester-config.json"

How it works, step by step

1. Load the severities into a hashtable. Reading the CSV once into a @{} hashtable means every lookup afterwards is an instant key access instead of a repeated scan of the file. For a handful of tests it hardly matters; for hundreds it keeps things snappy.

2. Define the recognition pattern. The regex ^\s*It\s+'([A-Z]+-\d+):\s*(.*?)' is the heart of the script. It says: a line that starts with It, followed by a quoted string that begins with an uppercase code, a dash, some digits, a colon, and then the description. The two capture groups hand us the Id and the description directly. If your codes use a different shape (say numbers only, or a longer prefix), this one line is all you need to adjust.

3. Walk every test file. Get-ChildItem -Recurse -Filter *.Tests.ps1 collects your Pester test files (and only those, thanks to the .Tests.ps1 filter), and the script reads them line by line. Every line that matches the pattern becomes one entry in the result set. Because it works purely on text, it does not need to run your tests, so generating the config is fast and has no side effects on your tenant.

4. Fail loudly, then continue. If a test has no matching severity in the CSV, the script does not crash. It assigns Unknown and emits a Write-Warning. That way a forgotten mapping shows up as a visible nudge in your build log rather than a broken pipeline. You get the config and the reminder to fix the mapping.

5. Emit the JSON. Finally, ConvertTo-Json turns the collected objects into exactly the structure Maester expects, and it lands right next to your tests as maester-config.json.

A note on the collection. The results go into a System.Collections.Generic.List[object] rather than a plain $array += .... It is a small thing, but += rebuilds the entire array on every iteration, which is exactly the kind of quiet inefficiency you do not want in a script whose whole point is letting the machine do the tedious work. The list just grows.

Wiring it into your workflow

Because the script is idempotent and side-effect free, it fits anywhere:

Locally, run it after adding a test so your config is always current before you commit.
In CI, run it as a build step and either commit the result or fail the build if the generated file differs from the checked-in one. That turns "the config drifted" into a red pipeline instead of a silent bug.
As a pre-commit hook, so nobody can forget.

A useful CI check is to regenerate and compare:

.\generate-maester-config.ps1
git diff --exit-code tests\Custom\maester-config.json

If that git diff returns a non-zero exit code, someone added or changed a test without regenerating the config, and CI will tell them.

Why bother?

A generated config buys you three things that a hand-written one never will:

It cannot drift. The config is derived from the tests, so it is correct by construction.
Severities become a shared, reviewable artefact. A CSV that non-engineers can edit means your security or compliance people can tune severities without opening a .ps1 file, and every change is visible in a pull request.
Onboarding is trivial. New tests just need to follow the naming convention. The plumbing takes care of itself.

The whole thing is a little over fifty lines of vanilla PowerShell, has no dependencies beyond what ships with the box, and works with any Maester setup that runs custom Pester tests. If you have been maintaining maester-config.json by hand, this is a small afternoon's work that you will be glad you did.

Have a different naming convention or a richer severity model? The regex and the lookup are the only two things you need to touch. Everything else stays the same.

← Alle posts Contact