ISO Schematron Conformance Tests

Schematron Users Meetup @ MarkupUK · London, June 7th 2025

Have a peek

Repository
https://github.com/schematron/schematron-conformance
Current version (branch master)
v2.0 (November 2024)
Next version (branch next)
v3.0

Conformance test

In software development, a test suite, less commonly known as a validation suite, is a collection of test cases that are intended to be used to test a software program to show that it has some specified set of behaviors.

Why conformance tests?

Conformance tests help to automatically check if or to what degree the program implements the ISO Schematron standard.

They make the standard explicit regarding the expected behavior given a predefined input.

In this, they can help standard document authors clarify important aspects of a standard.

Input & behavior

The input of a schematron operation are one or more schemas and one or more instance documents.

The behavior can only be observed via the output of a Schematron validation, the validation report.

Challenges

  • A test case may require multiple schemas, one for each possible query language.
    ☞ In practice tests are written for query languages using XPath 3.

  • Certain test cases are only valid for a specific query language binding (todo).
    ☞ Group test cases into query language specific test suites.

  • ISO Schematron comes in editions with differing feature sets (started).
    ☞ Group test cases into edition specific test suites.

Structure of a test case document

testcase expect="valid|invalid|error" phase="…"
  label
  schemas
    sch:schema
    …
  documents
    document filename="…"
    …
  assertions
    assertion test="…"

Structure of a testsuite document

testsuite
  label
  include href="…"
  …
  testcase href="…"
  …

Directory structure

The test case and test suite files are grouped into subdirectories for ease of maintenance.

  • rule-based validation
  • schema composition
  • reporting
  • pattern templating
  • instance document selection

Running the test suite

The SchXslt2 Testsuite Harness is a Java application that runs conformance tests for the SchXslt2 ISO Schematron to XSLT 3.0 transpiler.

The SchXslt2 Testsuite Harness CLI is an experimental command line application that runs conformance tests for pyschematron.

You find both programs at SchXslt's new home codeberg.org/schxslt.

Work in progress

Add more test cases for all Schematron editions (e.g. scope of variables).

Add test cases for the 2025 edition.