Support for wasm32-emscripten wheels

[hoodmane]

@henryiii has discussed adding a wasm32-emscripten target to cibuildwheel in the past. I would like to help with this. Currently it is not super invasive to build such wheels.

Here's numpy's CI support for Emscripten using pyodide build:
https://github.com/numpy/numpy/blob/main/.github/workflows/emscripten.yml

Rust + Python projects can directly use maturin:
https://github.com/pydantic/pydantic-core/blob/main/Makefile#L46

There's currently not all that many examples of projects building against Emscripten in their CI, but I would like to change that!. I think a lot of maintainers are enthusiastic about the concept.

[joerick]

I'm keen for this too. WASM is the next universal VM architecture! Looks like pyodide have done a lot of work to make this easier.

Why do we target wasm32 - a 32bit architecture? Is that the most common?

We don't want to depend on the setup-emsdk action, so we'd have to roll that functionality in. Luckily, that's pretty simple.

We'd also have to decide if it should be enabled by default, as I don't think PyPI is accepting these wheels yet.

[henryiii]

I think it should be a platform, so it would never being enabled by default, you’d have to ask for that platform. Currently, wheels have to target a specific version, there’s no backward compatibility guarantee, so they can’t be uploaded to PyPI; they aren’t really redistributable. But I think that’s fine for an opt-in platform, we could just have a warning for now.

I was originally thinking this should run on docker, but looks like maybe that’s not needed?

[hoodmane]

Why do we target wasm32 - a 32bit architecture? Is that the most common?

Well:

1. wasm64 doesn't really exist yet (it's specced but not implemented)
2. 32 bit pointers make code size much smaller
3. wasm is a harvard architecture -- function pointers and data pointers live in different address spaces. This reduces the amount of stuff in the address space by a fair amount.

So far very few people are running into the 4 gb limit.

[hoodmane]

I notice that build-frontend defaults to pip. We can't use pip as the build frontend for Pyodide at this point. Can I just print a warning and ignore this setting?

[joerick]

I suppose technically the build-frontend is pyodide? yes feel free to ignore this for now.

I was originally thinking this should run on docker, but looks like maybe that’s not needed?

Let's see how it goes - I think I'd slightly rather it was not on docker, but if that turns out to be a really easy way to get a reproducible environment I wouldn't be against it either.

[hoodmane]

I don't think we'll need it to be on docker.

[hoodmane]

Successfully built and tested multidict with

cibuildwheel --platform pyodide

[hoodmane]

I'm trying to build numpy now but it pins wheel==0.38.1 in it's build requires whereas cibuildwheel/resources/constraints-python310.txt has wheel==0.40.0. Am I using the cibuildwheel constraints file wrong?

[hoodmane]

It looks like numpy pins cibuildwheel==2.12.0 which may have something to do with the wheel version...

[henryiii]

Yes, environments are pinned by cibuildwheel version (unless you request otherwise, via https://cibuildwheel.readthedocs.io/en/stable/options/#dependency-versions )

How is numpy pinning cibuildwheel if you are building it?

[henryiii]

As for the build frontend, I'd say ideally make it a warning or even an error if a user specifically sets it to pip, but no warning/error if it's unset, if possible. Not sure when we think we might want to change the default.

[hoodmane]

Numpy pins the version here: https://github.com/numpy/numpy/blob/main/tools/ci/cirrus_general.yml#L3
I get a crash because of the wheel version conflict. I removed one of the wheel versions and now I successfully build the wheel. Working on getting the numpy tests to run...