[otfautohint] points “optimized away” in flex-like scenario (print only)

[frankrolf]

This is a curious bug, and I can’t say if otfautohint (and precursors) is at fault, or if the problem is within PDF.
It is specifically related to print output, and not reproducible on-screen. The attached project is an example for what I’m seeing.

These glyphs

reproduce in printed output like so:

The first glyph is a real-world example, other glyphs are reproductions of the same scenario.

What needs to happen to reproduce this:

• two chained curve segments where the final on-curve point is at the same y-coordinate as the first on-curve point, like here:

• UFO processed by otfautohint or psautohint (does not make a difference)

---

As we can see, the final on-curve point seems to have disappeared in the printed rasterization. This seems to be related to the amount of “flex”, and it has some visible consequences in my sample glyphs, but more drastic problems in โ. The software printing out the PDF is irrelevant:

However, if the glyphs are unhinted, this is the result:

Also, if the end of the 2nd segment is just moved 1 unit out of alignment, there will be no problem.
Adding the --no-flex option during hinting does not make a difference.

Test project attached, steps to reproduce:

checkoutlinesufo -e FlexTest-Thin.ufo
otfautohint FlexTest-Thin.ufo
makeotf -r -f FlexTest-Thin.ufo
waterfallplot -wfr 90,80,70,60,50 FlexTest-Thin.ot

FlexTest-Thin.zip

[skef]

Can you tx dump the hinted glyphs and post the output here? Or possibly do that for just the most affected bar? (Maybe also including the output for the unhinted glyphs in case a point is disappearing from the UFO.)

[frankrolf]

I can :-)
Here are a few different test scenarios:

The unhinted glyph is the same for all tests, it contains no overlaps:

unhinted

### CharStrings (flattened)
--- glyph[tag]={name,path}
[5]={flex.02,
  50 370 rmoveto
  84 83 -20 84 83 83 20 83 hhcurveto
  30 vlineto
  -83 -83 -20 -83 -84 -83 20 -84 hhcurveto
  endchar
}

test1 ❌

otfautohint, makeotf (no subroutinization, no checkoutlinesufo)

### CharStrings (flattened)
--- glyph[tag]={name,path}
[5]={flex.02,
  350 30 -10 30 hstemhm
  50 500 hintmask[60]
  50 370 rmoveto
  84 83 -20 84 hhcurveto
  hintmask[A0]
  83 83 20 83 hhcurveto
  hintmask[60]
  30 vlineto
  -83 -83 -20 -83 -84 -83 -84 hflex
  endchar
}

→ printout of waterfallplot PDF contains outline error.

test2 ❌

checkoutlinesufo, otfautohint, makeotf (no subroutinization)

### CharStrings (flattened)
--- glyph[tag]={name,path}
[5]={flex.02,
  350 30 -10 30 hstemhm
  50 500 hintmask[60]
  50 370 rmoveto
  84 83 -20 84 hhcurveto
  hintmask[A0]
  83 83 20 83 hhcurveto
  hintmask[60]
  30 vlineto
  -83 -83 -20 -83 -84 -83 -84 hflex
  endchar
}

exactly the same as test1
→ printout of waterfallplot PDF contains outline error.

test3 ✅

otfautohint --no-flex, makeotf (no subroutinization, no checkoutlinesufo)

### CharStrings (flattened)
--- glyph[tag]={name,path}
[5]={flex.02,
  350 30 -10 30 hstemhm
  50 500 hintmask[60]
  50 370 rmoveto
  84 83 -20 84 hhcurveto
  hintmask[A0]
  83 83 20 83 hhcurveto
  hintmask[60]
  30 vlineto
  -83 -83 -20 -83 -84 -83 20 -84 hhcurveto
  endchar
}

→ printed output is OK

test4 ✅

checkoutlinesufo, otfautohint --no-flex, makeotf (no subroutinization)

### CharStrings (flattened)
--- glyph[tag]={name,path}
[5]={flex.02,
  350 30 -10 30 hstemhm
  50 500 hintmask[60]
  50 370 rmoveto
  84 83 -20 84 hhcurveto
  hintmask[A0]
  83 83 20 83 hhcurveto
  hintmask[60]
  30 vlineto
  -83 -83 -20 -83 -84 -83 20 -84 hhcurveto
  endchar
}

exactly the same as test3
→ printed output is OK

test5 ❌

checkoutlinesufo, otfautohint, makeotf -r

### CharStrings (flattened)
--- glyph[tag]={name,path}
[5]={flex.02,
  350 30 -10 30 hstemhm
  50 500 hintmask[60]
  50 370 rmoveto
  84 83 -20 84 hhcurveto
  hintmask[A0]
  83 83 20 83 hhcurveto
  hintmask[60]
  30 vlineto
  -83 -83 -20 -83 -84 -83 -84 hflex
  endchar
}

same as test1, test2
→ printout of waterfallplot PDF contains outline error.

test6 ✅

checkoutlinesufo, otfautohint --no-flex, makeotf -r

### CharStrings (flattened)
--- glyph[tag]={name,path}
[5]={flex.02,
  350 30 -10 30 hstemhm
  50 500 hintmask[60]
  50 370 rmoveto
  84 83 -20 84 hhcurveto
  hintmask[A0]
  83 83 20 83 hhcurveto
  hintmask[60]
  30 vlineto
  -83 -83 -20 -83 -84 -83 20 -84 hhcurveto
  endchar
}

exactly the same as test3
→ printed output is OK

---

All test scenarios including output PDFs attached:
all_test_scenarios.zip

[frankrolf]

I guess the obvious solution for the glyphs in question would be to disable flex hinting, but still – shouldn’t flex only be active for small sizes anyway? And why does the problem only show in printed output (including obvious outline corruption beyond the area of the applied flex)?

[frankrolf]

Responding to a special request from @skef “what happens when all the start points are moved”?

test1 (start point moved to next on-curve point)

otfautohint, makeotf (no subroutinization, no checkoutlinesufo)

unhinted

### CharStrings (flattened)
--- glyph[tag]={name,path}
[5]={flex.02,
  301 350 rmoveto
  83 83 20 83 hhcurveto
  30 vlineto
  -83 -83 -20 -83 -84 -83 20 -84 hhcurveto
  -30 vlineto
  84 83 -20 84 hhcurveto
  endchar
}

hinted

### CharStrings (flattened)
--- glyph[tag]={name,path}
[5]={flex.02,
  350 30 -10 30 hstemhm
  50 500 hintmask[A0]
  301 350 rmoveto
  83 83 20 83 hhcurveto
  hintmask[60]
  30 vlineto
  -83 -83 -20 -83 -84 -83 -84 hflex
  -30 vlineto
  84 83 -20 84 hhcurveto
  endchar
}

original:

moved:

[frankrolf]

Same tests as above, this time focusing on โ
The unhinted glyph for all tests:

unhinted

### CharStrings (flattened)
--- glyph[tag]={name,path}
[2]={uni0E42,
  -20 251 -8 rmoveto
  48 33 34 48 47 -31 33 -45 -21 -20 -9 -12 -10 hvcurveto
  1 24 rlineto
  374 vlineto
  95 -40 37 -121 33 vhcurveto
  52 22 26 18 38 42 40 -19 45 hhcurveto
  29 29 3 15 35 hvcurveto
  32 vlineto
  -15 -34 -35 -4 -28 -46 -31 19 -48 hhcurveto
  -65 -43 -54 -50 -13 hvcurveto
  -17 vlineto
  110 -23 53 -34 -86 vvcurveto
  -450 vlineto
  -53 31 -38 49 vhcurveto
  2 25 rmoveto
  -29 -21 24 33 33 21 23 29 29 21 -23 -33 -33 -21 -24 -29 hvcurveto
  endchar
}

test1

makeotf only (no subroutinization, no checkoutlinesufo)

hinted

### CharStrings (flattened)
--- glyph[tag]={name,path}
[2]={uni0E42,
  -20 -8 25 113 24 593 31 -12 31 hstemhm
  171 32 -32 35 97 29 hintmask[D6]
  251 -8 rmoveto
  48 33 34 48 47 -31 33 -45 -21 -20 -9 -12 -10 hvcurveto
  1 24 rlineto
  374 vlineto
  95 -40 37 -121 33 vhcurveto
  52 22 26 18 38 42 40 -19 45 hhcurveto
  hintmask[E6]
  29 29 3 15 35 hvcurveto
  hintmask[D6]
  32 vlineto
  hintmask[EA]
  -34 -15 -35 -4 -28 -46 -31 19 -48 hflex1
  hintmask[D6]
  -65 -43 -54 -50 -13 hvcurveto
  -17 vlineto
  110 -23 53 -34 -86 vvcurveto
  -450 vlineto
  -53 31 -38 49 vhcurveto
  2 25 rmoveto
  hintmask[DA]
  -29 -21 24 33 33 21 23 29 29 21 -23 -33 -33 -21 -24 -29 hvcurveto
  endchar
}

test2

makeotf (no subroutinization), with checkoutlinesufo

hinted

### CharStrings (flattened)
--- glyph[tag]={name,path}
[2]={uni0E42,
  -20 -8 25 113 24 593 31 -12 31 hstemhm
  171 32 -32 35 97 29 hintmask[D6]
  251 -8 rmoveto
  48 33 34 48 47 -31 33 -45 -21 -20 -9 -12 -10 hvcurveto
  1 24 rlineto
  374 vlineto
  95 -40 37 -121 33 vhcurveto
  52 22 26 18 38 42 40 -19 45 hhcurveto
  hintmask[E6]
  29 29 3 15 35 hvcurveto
  hintmask[D6]
  32 vlineto
  hintmask[EA]
  -34 -15 -35 -4 -28 -46 -31 19 -48 hflex1
  hintmask[D6]
  -65 -43 -54 -50 -13 hvcurveto
  -17 vlineto
  110 -23 53 -34 -86 vvcurveto
  -450 vlineto
  -53 31 -38 49 vhcurveto
  2 25 rmoveto
  hintmask[DA]
  -29 -21 24 33 33 21 23 29 29 21 -23 -33 -33 -21 -24 -29 hvcurveto
  endchar
}

test3

makeotf only (no subroutinization, no checkoutlines), hinting with --no-flex

hinted

### CharStrings (flattened)
--- glyph[tag]={name,path}
[2]={uni0E42,
  -20 -8 25 113 24 593 31 -12 31 hstemhm
  171 32 -32 35 97 29 hintmask[D6]
  251 -8 rmoveto
  48 33 34 48 47 -31 33 -45 -21 -20 -9 -12 -10 hvcurveto
  1 24 rlineto
  374 vlineto
  95 -40 37 -121 33 vhcurveto
  52 22 26 18 38 42 40 -19 45 hhcurveto
  hintmask[E6]
  29 29 3 15 35 hvcurveto
  hintmask[D6]
  32 vlineto
  hintmask[EA]
  -15 -34 -35 -4 -28 -46 -31 19 -48 hhcurveto
  hintmask[D6]
  -65 -43 -54 -50 -13 hvcurveto
  -17 vlineto
  110 -23 53 -34 -86 vvcurveto
  -450 vlineto
  -53 31 -38 49 vhcurveto
  2 25 rmoveto
  hintmask[EA]
  -29 -21 24 33 33 21 23 29 29 21 -23 -33 -33 -21 -24 -29 hvcurveto
  endchar
}

test4

makeotf (no subroutinization), checkoutlines, hinting with --no-flex

hinted

### CharStrings (flattened)
--- glyph[tag]={name,path}
[2]={uni0E42,
  -20 -8 25 113 24 593 31 -12 31 hstemhm
  171 32 -32 35 97 29 hintmask[D6]
  251 -8 rmoveto
  48 33 34 48 47 -31 33 -45 -21 -20 -9 -12 -10 hvcurveto
  1 24 rlineto
  374 vlineto
  95 -40 37 -121 33 vhcurveto
  52 22 26 18 38 42 40 -19 45 hhcurveto
  hintmask[E6]
  29 29 3 15 35 hvcurveto
  hintmask[D6]
  32 vlineto
  hintmask[EA]
  -15 -34 -35 -4 -28 -46 -31 19 -48 hhcurveto
  hintmask[D6]
  -65 -43 -54 -50 -13 hvcurveto
  -17 vlineto
  110 -23 53 -34 -86 vvcurveto
  -450 vlineto
  -53 31 -38 49 vhcurveto
  2 25 rmoveto
  hintmask[EA]
  -29 -21 24 33 33 21 23 29 29 21 -23 -33 -33 -21 -24 -29 hvcurveto
  endchar
}

test5

makeotf release mode, checkoutlines

hinted

### CharStrings (flattened)
--- glyph[tag]={name,path}
[2]={uni0E42,
  -20 -8 25 113 24 593 31 -12 31 hstemhm
  171 32 -32 35 97 29 hintmask[D6]
  251 -8 rmoveto
  48 33 34 48 47 -31 33 -45 -21 -20 -9 -12 -10 hvcurveto
  1 24 rlineto
  374 vlineto
  95 -40 37 -121 33 vhcurveto
  52 22 26 18 38 42 40 -19 45 hhcurveto
  hintmask[E6]
  29 29 3 15 35 hvcurveto
  hintmask[D6]
  32 vlineto
  hintmask[EA]
  -34 -15 -35 -4 -28 -46 -31 19 -48 hflex1
  hintmask[D6]
  -65 -43 -54 -50 -13 hvcurveto
  -17 vlineto
  110 -23 53 -34 -86 vvcurveto
  -450 vlineto
  -53 31 -38 49 vhcurveto
  2 25 rmoveto
  hintmask[DA]
  -29 -21 24 33 33 21 23 29 29 21 -23 -33 -33 -21 -24 -29 hvcurveto
  endchar
}

[skef]

The problem with the bars suggests one possible explanation (improper handling of an implicit subpath-closing line), but the problem with the full glyph can't be explained that way.

If indeed the most recent test 4 lacks the problem with the lower oval, and the most recent test 5 has it, the only difference between those two are:

1d0
< ### CharStrings (flattened)
17c16
<  -15 -34 -35 -4 -28 -46 -31 19 -48 hhcurveto
---
>  -34 -15 -35 -4 -28 -46 -31 19 -48 hflex1
25c24
<  hintmask[EA]
---
>  hintmask[DA]

The slightly different hintmask for the oval is probably incidental. In any case it doesn't seem to affect the rendering of the inside of the oval, which is what it precedes. So the only difference is the presence of the flex hint in an entirely different part of the glyph, away from the mangled oval. I don't see how that can be a hinting bug, all signs would point to either a bug in the rasterizer itself or, possibly more likely, some conversion tool (e.g. a whatever-to-PDF converter).

[skef]

That makes the question of your tool-chain particularly important. We should consider the steps in creating the PDF and possibly look at alternative tools for each step to see if the problem goes away.

[frankrolf]

Regarding the tool chain: this issue was first discovered when proofing a font (including the โ) via InDesign’s “Export PDF” feature. The PDF was printed via Apple Preview.
I opted to use waterfallplot, because this tool embeds the actual font in the PDF as well.

[skef]

I guess one thing to try would be extracting the font from the PDF (I believe there are tools that can do that) and seeing if it looks different.

[frankrolf]

Extracted from test1 PDF, opened the resulting data blob in FontForge (the only application that’ll do so these days):

[skef]

FontForge for the (rare) win.

OK, so I guess the first question is whether the flex hint itself is wrong and screwing things up, or if it's the tool's interpretation of the flex hints that are causing the problem. I think the former is mostly ruled out by the bar cases -- the hints themselves look about right. (And in a UFO a flex hint is just a notation on a point.)

So something can't properly interpret flex hints, or perhaps these optimized flex hints like hflex.

[frankrolf]

Starting at the supposed flex-point (which is the one on top of the green rectangle), all on-curves have been moved upward by 38 units. 19 is the amount of flex, so it seems some tool is over-zealous here by applying it twice.

default.mov

[skef]

We're pretty sure this isn't "our" bug. Seems to be something in Apple's MacOS print stack.

[skef]

@frankrolf should we convert this to a card about the MacOS problem (or just close it)?

[frankrolf]

Probably warrants a "3rd Party Problem" label. I'll file a macOS bug and will close this after.