What okfreq coverage really means¶
okfreq is a demonstrator to understand how to connect software requirements
to source files and test files on top of OKF-Schema
That is useful traceability, and can work pretty well for most opensource project but it is not the same thing as proving that the software requirement was successfully tested.
This distinction matters especially for safety-related work.
okfreqt is not a complete ISO 26262 verification or compliance tool,
and it does not claim conformance to any other engineering standard.
Its StRS/SwRS vocabulary is informed by
ISO/IEC/IEEE 29148, but using that
vocabulary does not provide the evidence required by a safety process.
But it CAN be extended and customized to reach a stronger claim about executed test coverage. But this will require to adapt your test runner and reports to integrate some tracer mechanism that can link test execution results back to the requirement IDs.
Some test report formats, especially the opensource ones, are very flexible (Junit,…) while other may require to develop custom external adapters.
Coverage model¶
okfreq keeps three coverage questions separate:
SwRS source coverage¶
Source coverage applies only to leaf SwRS requirements. A SwRS is source
covered when at least one configured @implements_req SwRS-ID marker is found.
StRS is a stakeholder-level requirement, so source coverage is not applicable
to it and is not included in the StRS schema or stakeholder coverage table.
SwRS test-link coverage¶
SwRS test-link coverage applies only to leaf SwRS requirements. A SwRS is
test-link covered when at least one configured @tests_req SwRS-ID marker is
found. This is a file-level relationship, not a test execution result.
StRS test coverage¶
StRS test coverage is computed from its directly derived SwRS requirements. The
rule is selected by strs_test_coverage_mode in config.yml:
Mode |
StRS is covered when |
|---|---|
|
It links to at least one SwRS and every linked SwRS has a test marker. |
|
The |
A validation test is defined by placing the normal test marker on the StRS ID in a configured test file:
# @tests_req StRS-CORE-001
This direct StRS marker is collected as validation-test evidence. An
@implements_req marker on an StRS remains non-leaf evidence and never creates
source coverage. The generated Markdown report therefore renders StRS and
SwRS in separate tables, and places the active StRS rule below the StRS table.
The two StRS modes deliberately answer different questions. The first asks whether the stakeholder outcome is decomposed into test-linked software behavior. The second also asks whether the stakeholder outcome has an explicit validation test definition of its own.
The simple distinction¶
Imagine a test file with this marker:
# @tests_req SwRS-CORE-001
def test_export_writes_csv():
...
The marker says: “This file contains a test related to SwRS-CORE-001.”
It does not say:
the test was executed;
the test passed;
the test was not skipped;
the test exercised the relevant behavior; or
the test result came from the required software and configuration.
okfreq currently scans the test source and records the file-level link. It
does not read the result produced by the test runner. A test can therefore be
linked to a requirement even when it fails, is skipped, or has never run.
flowchart TD
A[Requirement] --> B[Test source marker]
B --> C[okfreq can confirm a file link]
C -. does not confirm .-> D[Test was executed]
D -. does not confirm .-> E[Test passed]
The coverage fields generated by okfreq should therefore be read as
traceability or linkage coverage. SwRS fields answer “where is this
requirement mentioned in implementation and test code?” StRS test coverage
answers “does its configured chain of linked SwRS, and optionally its direct
validation-test definition, exist?” Neither answer means “was this requirement
verified successfully?”
What full SwRS test coverage would need¶
For a stronger claim about software requirement coverage, a tool would need to combine two kinds of evidence:
the requirement-to-test link, such as
SwRS-CORE-001attached to a test;the result of the relevant test execution, including whether that test ran and passed.
It would also normally need enough execution context to make the result trustworthy: the test-run configuration, software revision, target or environment, and the relevant test run or report. The exact evidence depends on the project and its safety process.
That is different from scanning a source file. Source scanning can discover a declared relationship. Test-result ingestion can establish what happened when the test was run. Both are needed for a meaningful executed-test coverage claim.
Why test-result ingestion is difficult¶
Test reports are not one common format. JUnit XML and GoogleTest JSON can carry additional metadata, or can sometimes be extended to do so. Other runners and report formats make arbitrary requirement metadata harder to preserve. A reliable solution would need adapters for the formats used by each project and clear rules for how a test emits its requirement IDs.
Even with a format that supports metadata, the mapping is not always simple. Parameterized tests are a good example:
one test definition can produce many reported test cases;
each generated case can have a different input and result; and
a marker on the source file or test definition may not identify which generated case it covers.
The tool would need to define whether the requirement belongs to the test definition, every generated case, or only selected cases. It would also need to handle retries, skipped cases, renamed tests, and partially available reports. Those are integration and evidence-management problems, not merely another field in a requirement document.
See the frontmatter reference for the generated coverage fields and the requirements traceability tutorial for the marker workflow.