Reference/Output schema

Output schema

Edit on GitHub ↗

File locations, the results file structure, result values, and SARIF.

File locations

For each target, Core writes two files:

<write-directory>/
  <target>/
    <target>.log      # execution log
    <target>.yaml     # results; .json or .sarif when output is json or sarif

<write-directory> defaults to evaluation_results. <target> is the target’s name under targets: in the config. The gemara output format also writes <target>.yaml. The output formats are listed in the CLI reference.

Results file structure

With output: yaml (the default) or output: json, the results file has one entry per evaluation suite. Each suite carries a Gemara Evaluation Log under control-evaluations. Abridged JSON from a run of the GitHub repo scanner:

{
  "service-name": "my-github-repo",
  "plugin-name": "github-repo",
  "plugin-uri": "https://github.com/ossf/pvtr-github-repo-scanner",
  "plugin-version": "0.33.2",
  "evaluation-suites": [
    {
      "name": "my-github-repo_osps-baseline",
      "result": "Failed",
      "catalog-id": "osps-baseline",
      "control-evaluations": {
        "metadata": { "type": "EvaluationLog", "gemara-version": "v1.5.0" },
        "result": "Failed",
        "evaluations": [
          {
            "name": "Secure Secrets and Credentials",
            "result": "Failed",
            "message": "GitHub reports secret scanning and push protection are both disabled",
            "control": { "reference-id": "osps-baseline", "entry-id": "OSPS-BR-07" },
            "assessment-logs": [
              {
                "requirement": { "reference-id": "osps-baseline", "entry-id": "OSPS-BR-07.01" },
                "result": "Failed",
                "message": "GitHub reports secret scanning and push protection are both disabled",
                "applicability": ["maturity-1", "maturity-2", "maturity-3"],
                "steps-executed": 1
              }
            ]
          }
        ]
      }
    }
  ]
}

With output: gemara, the file is a YAML list of the Gemara Evaluation Logs alone, without the Privateer wrapper.

Result values

Suites, controls and requirements each carry one of the six Gemara results. The file spells them as shown here:

ValueGo nameMeaningHalts a step chain
Passedgemara.PassedThe requirement was checked and metno
Failedgemara.FailedThe requirement was checked and not metyes
Needs Reviewgemara.NeedsReviewThe check could not decide: the data was absent or ambiguous, or only a heuristic could judge it; a person has to lookno
Not Applicablegemara.NotApplicableThe requirement does not apply under the selected applicability, or to this targetyes
Not Rungemara.NotRunNo step produced a result; plugin authors return it only from placeholder stepsno
Unknowngemara.UnknownThe check could not run, or no step was defined for the requirement; from a step, almost always a loader bugno

The Go name is what a plugin’s assessment step returns. A requirement’s steps run in order and stop at the first Failed or Not Applicable; every other result is folded into the requirement’s result and the next step runs.

A suite with any Failed, Needs Review or Unknown result makes pvtr run exit 1; see exit codes.

SARIF output

With output: sarif, Core writes a SARIF 2.1.0 file. Each assessed requirement appears as a result whose ruleId is the requirement ID (for example OSPS-BR-07.01); failed requirements have level: error. The file works with the GitHub Security tab and any SARIF 2.1.0 consumer.