What okfreq coverage really means

okfreq is a demonstrator to understand how to connect software requirements to source files and test files on top of OKF-Schema

That is useful traceability, and can work pretty well for most opensource project but it is not the same thing as proving that the software requirement was successfully tested.

This distinction matters especially for safety-related work. okfreqt is not a complete ISO 26262 verification or compliance tool, and it does not claim conformance to any other engineering standard. Its StRS/SwRS vocabulary is informed by ISO/IEC/IEEE 29148, but using that vocabulary does not provide the evidence required by a safety process.

But it CAN be extended and customized to reach a stronger claim about executed test coverage. But this will require to adapt your test runner and reports to integrate some tracer mechanism that can link test execution results back to the requirement IDs.

Some test report formats, especially the opensource ones, are very flexible (Junit,…) while other may require to develop custom external adapters.

Coverage model

okfreq keeps three coverage questions separate:

SwRS source coverage

Source coverage applies only to leaf SwRS requirements. A SwRS is source covered when at least one configured @implements_req SwRS-ID marker is found. StRS is a stakeholder-level requirement, so source coverage is not applicable to it and is not included in the StRS schema or stakeholder coverage table.

StRS test coverage

StRS test coverage is computed from its directly derived SwRS requirements. The rule is selected by strs_test_coverage_mode in config.yml:

Mode

StRS is covered when

linked-swrs

It links to at least one SwRS and every linked SwRS has a test marker.

linked-swrs-and-validation-test

The linked-swrs rule is satisfied and at least one validation test is defined directly for the StRS.

A validation test is defined by placing the normal test marker on the StRS ID in a configured test file:

# @tests_req StRS-CORE-001

This direct StRS marker is collected as validation-test evidence. An @implements_req marker on an StRS remains non-leaf evidence and never creates source coverage. The generated Markdown report therefore renders StRS and SwRS in separate tables, and places the active StRS rule below the StRS table.

The two StRS modes deliberately answer different questions. The first asks whether the stakeholder outcome is decomposed into test-linked software behavior. The second also asks whether the stakeholder outcome has an explicit validation test definition of its own.

The simple distinction

Imagine a test file with this marker:

# @tests_req SwRS-CORE-001
def test_export_writes_csv():
    ...

The marker says: “This file contains a test related to SwRS-CORE-001.”

It does not say:

  • the test was executed;

  • the test passed;

  • the test was not skipped;

  • the test exercised the relevant behavior; or

  • the test result came from the required software and configuration.

okfreq currently scans the test source and records the file-level link. It does not read the result produced by the test runner. A test can therefore be linked to a requirement even when it fails, is skipped, or has never run.

        flowchart TD
    A[Requirement] --> B[Test source marker]
    B --> C[okfreq can confirm a file link]
    C -. does not confirm .-> D[Test was executed]
    D -. does not confirm .-> E[Test passed]
    

The coverage fields generated by okfreq should therefore be read as traceability or linkage coverage. SwRS fields answer “where is this requirement mentioned in implementation and test code?” StRS test coverage answers “does its configured chain of linked SwRS, and optionally its direct validation-test definition, exist?” Neither answer means “was this requirement verified successfully?”

What full SwRS test coverage would need

For a stronger claim about software requirement coverage, a tool would need to combine two kinds of evidence:

  1. the requirement-to-test link, such as SwRS-CORE-001 attached to a test;

  2. the result of the relevant test execution, including whether that test ran and passed.

It would also normally need enough execution context to make the result trustworthy: the test-run configuration, software revision, target or environment, and the relevant test run or report. The exact evidence depends on the project and its safety process.

That is different from scanning a source file. Source scanning can discover a declared relationship. Test-result ingestion can establish what happened when the test was run. Both are needed for a meaningful executed-test coverage claim.

Why test-result ingestion is difficult

Test reports are not one common format. JUnit XML and GoogleTest JSON can carry additional metadata, or can sometimes be extended to do so. Other runners and report formats make arbitrary requirement metadata harder to preserve. A reliable solution would need adapters for the formats used by each project and clear rules for how a test emits its requirement IDs.

Even with a format that supports metadata, the mapping is not always simple. Parameterized tests are a good example:

  • one test definition can produce many reported test cases;

  • each generated case can have a different input and result; and

  • a marker on the source file or test definition may not identify which generated case it covers.

The tool would need to define whether the requirement belongs to the test definition, every generated case, or only selected cases. It would also need to handle retries, skipped cases, renamed tests, and partially available reports. Those are integration and evidence-management problems, not merely another field in a requirement document.

See the frontmatter reference for the generated coverage fields and the requirements traceability tutorial for the marker workflow.