Some tooling for generating regression shaping tests

[madig]

We'd like to use the regression shaping tests in fontbakery, but found that writing and updating the JSON files by hand is tedious. Are there any plans on adding tooling for this?

If not, I had the idea of porting some existing tooling to output fontbakery-compatible JSON files instead. It consists of

• A CLI tool that takes a bunch of TOML input files and transforms them into regression test expectations files,
• and the TOML files themselves, which you edit by hand.

The split of input and output files makes it easier to edit what you'd like to see, because tests bigger than a couple of characters lead to big and unwieldy output, especially if you have more than a couple of fonts to check, or even fvar entries.

You would have a directory of input TOML files that look like this:

[input.features]
ccmp = true
c2sc = true
smcp = true

[input]
script = "latn"
direction = "ltr"
comparison_mode = "full"
text = [
    "i̊i̮̊j̮̊", # idotless and jdotless
    "į į̀ į́ į̂ į̃ į̄ į̌ і́ ј́ fį fị", # Soft-dotted glyphs
]

[input.features]
frac = true

[input]
comparison_mode = "glyphstream"
text = [ 
    "1/2 1/3 2/3 1/4 3/4 1/8 3/8 5/8 7/8",
    "1234567890/1234567890123",
    "11111111111/2222222222222",
    "0/0/0 00/000/00000 0//0",
    "2 1/2 222222222222 1/2"
]

Note how features are global to all test strings. The comparison_mode keyword decides whether you are interested in the shaping output with (full) or without (glyphstream) positioning information. Then, for each string in the text list, you'd generate a fontbakery-side regression test for each font under test (using the only key). For variable fonts, you generate a test for each fvar-entry (using the only and variations keys) for each string. The TOML structure could also be improved a bit by e.g. allowing specific features for specific test strings.

The generated JSON file would then be rather big even on a medium-sized project.

If you find this agreeable, where should the transformation tool live? As a submodule somewhere in fontbakery? Alternatively, I can make this an external tool and workflow unrelated to fontbakery. Then it would be nice though to discuss what tooling you'd be willing to onboard, to make the regression testing workflow less tedious.