Tests that check the whole code
What is this about?
An ordinary test checks a case we already know. A meta test searches the whole code for a pattern and fails as soon as something new violates it. This way a security rule is not lost just because we build something new.
Protection goal
Security rules should also apply to code that does not exist yet. Especially when building quickly, nobody should be able to introduce something by accident that bypasses a rule.
How it works
Instead of checking that one particular route is protected, a meta test checks every route. If a new one lacks protection, the test fails and the change is not released. Examples of such patterns:
- Every route has an explicit access decision. If one is missing, the request is also rejected at runtime.
- Every text field that holds content is mapped as encrypted.
- Every table that belongs to a project is covered by the project filter, so that no query reaches across project boundaries.
- New content types are moved to the trash first when deleted.
- Responses contain only fields that are explicitly released.
- Every permission check takes the account status into account, and every rate limit is set up so that it works.
- Hierarchies that refer to themselves reject cycles.
- Writing requests are protected against double processing in case a request is repeated after a dropped connection.
- Texts for users go through translation keys.
Measures
- We prefer meta tests over single-case tests when both are possible, because a pattern prevents a whole class of errors while a single case prevents only the one known error.
- The meta tests run on every change in the check pipeline. A change that violates a pattern is not released.
- For access checks there is also a counterpart at runtime that rejects a route without a decision, even if the check pipeline had not run.
- Every new security-relevant safeguard gets a test before it counts as done.
What we do not promise
- Meta tests find only patterns we know. New kinds of errors need new patterns.
- They show that a check exists, not that it checks the right thing from a functional point of view. That is why mutation testing additionally looks at the permission checks.
- They replace neither people reading the code nor an independent penetration test. There has not been one so far.
How we check this
The table below names the meta tests. Whether our tests would notice errors in the permission checks is checked monthly with mutation testing; the result is in the monthly report.
Evidence tests
| Test | What the test proves |
|---|---|
AuthorizationCoverageTest |
Every route of the application has an explicit access decision. |
EncryptionCoverageTest |
Text fields that hold content are mapped as encrypted. |
TenantFilterCoverageTest |
Every table that belongs to a project is covered by the project filter. |
SoftDeleteCoverageTest |
New content types are moved to the trash first when deleted. |
SerializerGroupLeakageTest |
Responses contain only explicitly released fields. |
VoterUserStatusGateCoverageTest |
Every permission check takes the current account status into account. |
RateLimiterLockCoverageTest |
Every rate limit is set up so that it works. |
SelfReferencingHierarchyCycleGuardTest |
Hierarchies that refer to themselves reject cycles. |
GlobalIdempotencyCoverageTest |
Writing requests are protected against double processing when repeated. |
NoHardcodedUserFacingStringsTest |
Texts for users go through translation keys. |
Change history
- : First version.