"OSError: cannot open resource" when trying to load more than exactly 509 ImageFonts

[Luux]

What did you do?

For a data generator, I need to use lots of fonts in different sizes. Randomly one of that is used to generate a sample. Since I didn't want to load a font every time we generate a sample, I created a nested dictionary that dynamicalle loads fonts of a given size when it wasn't loaded previously. It worked fine at the beginning. Then I tried to generate some more samples (>1k) and got "OSError: cannot open resource" every time. I double checked the paths and tried to load the font with the same path and that worked just fine. It definitely isn't related to the path encoding bugs.

To reproduce that, you can use the code below.

What did you expect to happen?

It should be able to load fonts until the RAM is full ;)

What actually happened?

OSError after loading exactly 509 fonts.
After clearing my dictionary as soon as the error occurs (temporary workaround), it works fine again.

What are your OS, Python and Pillow versions?

• OS: Win10
• Python: 3.6
• Pillow: 5.4.1

Please include code that reproduces the issue and whenever possible, an image that demonstrates the issue. Please upload images to GitHub, not to third-party file hosting sites. If necessary, add the image to a zip or tar archive.

The best reproductions are self-contained scripts with minimal dependencies. If you are using a framework such as plone, Django, or buildout, try to replicate the issue just using Pillow.

from PIL import ImageFont

test = []

for i in range(1000):
        test.append(ImageFont.truetype("fonts/AbyssinicaSIL-R.ttf", 15))

[radarhere]

When I run your script with a different font on my macOS machine, I have no problems. However, when I run the following code -

from PIL import ImageFont

test = []

for i in range(1000):
	print(i)
	test.append(open("fonts/AbyssinicaSIL-R.ttf", "r"))

It stops at 253 with OSError: [Errno 24] Too many open files. Could you run this code? I suspect that it will stop at 509, revealing that you are experiencing a problem with too many open files.

[Luux]

Nope, no problem there. The "too many open files" error occurs only if I try to get beyond 8187 - that's far from the 509 here. :/

[Luux]

After a bit of experimenting, the code below seems to work fine as workaround:

from PIL import ImageFont
from io import BytesIO

test = []

for i in range(1000):
    file = open("fonts/AbyssinicaSIL-R.ttf", "rb")
    bytes_font = BytesIO(file.read())
    test.append(ImageFont.truetype(bytes_font, 15))

[radarhere]

Does this happen for just this font, or for multiple different fonts, including standard ones like Helvetica?

[radarhere]

https://docs.microsoft.com/en-us/cpp/c-runtime-library/file-handling?view=vs-2017

The C run-time libraries have a 512 limit for the number of files that can be open at any one time.

I presume that is the limit that you are hitting. So I wouldn't think of this as a bug in Pillow. Any thoughts?

[Luux]

I randomly choose from a range of different fonts in a directory in my original code, so this happens for other fonts, too.

Since the limit is at 512 and this error occurs after exactly 509, I'm not sure if this is connected to each other. Even if it was, a more meaningful exception should be raised in that case.

Is there actually any need for the file to be kept open in the current implementation of Pillow when calling ImageFont.truetype?

[radarhere]

I don't know what a more meaningful exception could be. If you have ideas, feel free to put them forward.

Pillow is not handling the fonts here by itself, it is making use of FreeType. As an aside to anyone else reading this, now that Pillow has FreeType support in AppVeyor, I was able to replicate the error there.

From my understanding, it is FreeType that is actually keeping the file pointer open, and the 'Cannot open resource' message originates from FreeType. We call FT_Done_Face when we are done with the font, but that's not applicable here, since the fonts in your array might still be used.

So I don't see any reasonable action to take to improve this. We could stop using FreeType, but that's extreme. We could load the load into memory, like you're doing with BytesIO, but there's no advantage over that being run externally by you. Feel free to add any more information or disagree with my thoughts.

[Luux]

If you can track down the "real" error before it's thrown maybe just a reference to that issue could be enough.
Or we just hope that if somebody has the same issue he will stumble upon this himself, but i'd still prefer an more meaningful error message if it's not too complicated.

To stop using FreeType because of this would indeed be decent overkill ;)

[delzac]

Hi, i experienced the same exact issue where opening exactly 509 fonts will raise an OSError.

I'm using Win 10, Python 3.6, Pillow 5.4.1

I'm also working on a project that uses up to 60,000 different fonts.

@Luux did you manage to get it resolved?

[radarhere]

@delzac the workaround is to read the font into memory first

[ismael-elatifi]

I have the same issue with Windows 10, Python 3.6.8, Pillow 5.4.1.
To add some information about the issue, the maximum number of font objects that can be kept in memory at the same time is not deterministic. For example, one time I run my program and can load only 372 font objects, and if rerun it with no modification I can now load only 386.
But fortunately the workaround of @Luux using BytesIO is working fine. Thanks to him for that !
Here is below a very simple code to reproduce the issue by creating many font objects with the same font path :
Code failing with "OSError: cannot open resource" :

    font_path = './some_path/some_font.ttf'
    assert os.path.isfile(font_path)
    [ImageFont.truetype(font=font_path) for _ in range(1000)]

With the workaround of @Luux it is working fine :

    [ImageFont.truetype(font=BytesIO(open(font_path, "rb").read())) for _ in range(1000)]

[radarhere]

Okay, I'm going to suggest resolving this by noting the situation in the docs - to that end, I've created PR #4020. Let me know if anyone has any thoughts.

[delzac]

I think documenting it is a great idea!

[ismael-elatifi]

Good to document this issue. You could also add the BytesIO workaround in the doc.

[radarhere]

PR #6485 adds a note that copying the files into memory is a workaround.