ENH: enable support for writing to memory

[brendan-ward]

Resolves #249

This enables basic writing to memory using an io.BytesIO object. This does not currently support append / adding layers; I tried working that out unsuccessfully in earlier commits, but we could return to this in a later PR. It seems like it should be possible in theory.

This specifically blacklists ESRI Shapefile and OpenFileGDB because they write multiple files, which can't be returned successfully as a single memory file.

[jorisvandenbossche]

Some things we still need to add / figure out:

• append / add layers (still trying to figure out how to make this work)

FWIW, I would personally first focus on getting the plain writing to memory work without append (just raise for now if the mode is append)
(fiona also raises an error if mode="a" for writing to file-like objects)

[brendan-ward]

Fiona has specific tests for appending so I assumed it was possible and expected behavior, but indeed in the interest of getting this out sooner we can more strictly limit this to write-only mode.

[jorisvandenbossche]

Fiona has specific tests for appending so I assumed it was possible and expected behavior

I assume this only works when directly using the MemoryFile interface. Because at that point you have the same object, where you first write to, and then append to.
But when creating a new memfile, you can't append to it (test). And that is essentially what happens in geopandas when people pass a BytesIO object to to_file, we each time create a new MemoryFile object through fiona.open(), and thus only pure writing is supported (fiona explicitly raise in case of append mode in open() for a writeable object).

So longer term that is also something we could do, but it might requiring creating some similar custom object like fiona's MemoryFile?

[jorisvandenbossche]

Should this still call vsi_path(..)? (what it did before in the deleted code below)

[brendan-ward]

I took it out because I wasn't convinced that it would work properly and my instead emit confusing errors (though stringifying something that would have gotten a VSI path would be confusing too). We don't have any tests that prove writing to /vsi*/ paths would be successful. I also didn't see support for writing to a /vsi*/ path in Fiona; they are largely used for reading.

For instance, it looks like it is not possible to write via the /vsizip/ handler:

write_dataframe(df, "/vsizip//tmp/test.shp.zip", driver="ESRI Shapefile")

Traceback (most recent call last):
...
pyogrio.errors.DataSourceError: Driver ESRI Shapefile does not support write functionality

Are you aware of any things that we can write to a /vsi*/ path that should succeed? I can also restore the original behavior, in case our current unknown / untested functionality is working for someone.

[jorisvandenbossche]

In theory /vsizip should have write capabilities (https://gdal.org/user/virtual_file_systems.html#write-capabilities), but indeed I also don't get that working (maybe the way we handle that path internally isn't fit for working with such a vsi path)

But something like writing to S3 with vsis3 should in theory also be supported? (although that is indeed something we don't test)

[jorisvandenbossche]

I quickly tested this with a public test bucket, and I can write to an "s3://" path (which will get converted to /vsis3 by our preprocessing)

[brendan-ward]

Thanks for testing! That confirms we should indeed leave it in place for now.

[jorisvandenbossche]

What was the reason for this change? (we should at least also ensure writing to s3:// works as well, so maybe parametrize the test for both options?)

[brendan-ward]

These were producing failures when I had disabled handling the vsi paths; since that has been reverted I'm also reverting these.

[jorisvandenbossche]

Ideally for later we should find a new file format to test with that doesn't support fast total bounds?

[brendan-ward]

It looks like GeoJSONSeq still doesn't support fast total bounds, so I reverted back to using that directly.

[jorisvandenbossche]

I quickly tested this with a public test bucket, and I can write to an "s3://" path (which will get converted to /vsis3 by our preprocessing)

[jorisvandenbossche]

Looking good!