How we test
What is this about?
We check Illumentis on several levels: with automated tests, with tests that search the whole code for patterns, with targeted attack attempts and with scans. A change has to pass these checks before it is released. Figures and results are in the monthly report.
Protection goal
A safeguard should not just work once, but keep working with every change to the code. New errors should show up before users run into them.
How it works
The checks consist of layers that complement each other:
- Functional tests: unit and functional tests in the backend (PHPUnit), tests of the interface (Vitest) and end-to-end tests in the browser (Playwright).
- Security tests: every security-relevant safeguard has its own test class.
- Tests across the whole code: they look for patterns instead of known single cases, for example whether every route has an access check. More in a separate article.
- Static analysis: PHPStan finds type and logic errors, Deptrac checks the architecture rules.
- Mutation testing: Infection deliberately builds errors into permission checks and checks whether the tests notice them.
- Dependencies and secrets: checking the libraries for known vulnerabilities and licenses, and searching for credentials that were committed by accident.
- Attack attempts and scans: targeted attempts to reach other people’s data as a signed-in user, and a scan of the beta instance from outside.
Measures
We use these checks in a way that does not depend on good moods:
- The check pipeline runs on every change. A change is only released when the mandatory checks pass.
- Security-critical code has a higher coverage threshold than the rest, and it is anchored as a check in the pipeline.
- The dependency check runs on every change; a release also produces a bill of materials of the libraries used.
- Mutation testing runs monthly as a report. It does not block a release, but it should show where tests need to be added.
- The test lists themselves are checked: there are tests that make sure named tests exist and are not skipped silently.
What we do not promise
- Tests do not find every error. They check what we thought of beforehand.
- A high mutation testing value shows that tests would notice changes, but not that they check the right thing from a functional point of view.
- Mutation testing is a report and not a block in the pipeline.
- The checks are our own. An independent external penetration test has not taken place so far.
How we check this
Results and figures are in the monthly report; on this page we deliberately name no numbers that would go out of date. The table shows tests that safeguard the checks themselves.
Evidence tests
| Test | What the test proves |
|---|---|
FunctionalCoverageManifestTest |
Every requirement in the product specification has at least one test that explicitly covers it. |
SecurityTestManifestCoverageTest |
Every security test class named as mandatory really exists as a file. |
NoSkippedFunctionalTestsTest |
Functional tests are not skipped silently. |
RedTeamValidationCatalogCoverageTest |
The report names only test classes that exist. |
Change history
- : First version.