arm64

[philippemiron]

Hi everyone,

I got a new computer, and since it comes with Apple Silicon (M1 arm processor), instead of an anaconda/miniconda install, I went with miniforge, which supports various CPU architectures. It looks like packages have to be added to the https://conda.anaconda.org/conda-forge/osx-arm64 to be supported.

I don't know how much work is involved in porting the current code, but I wondered if this is a plan for future updates? I could help with testing/porting if necessary.

Cheers,

[erikvansebille]

Thanks @philippemiron. Not sure though how to proceed with this. Perhaps @CKehl or @willirath can chip in?

[willirath]

I think ARM support would be great!

To me, it looks like there's two separate parts to the problem: Scipy mode and JIT mode.

Here's how I'd proceed for Scipy mode:

Build an env that includes all dependencies except for parcels and the compiler packages (see this env file for a starting point) and then manually install Parcels as described in the installation docs for Parcels. As I believe all of the dependencies needed in Scipy mode are available on for ARM, this should allow for running Parcels in Scipy mode (ie without just-in-time compilation).

And for JIT mode:

When running experiments, Parcels creates C-code and writes it to one or more temporary files. Then, it calls the C compiler and creates shared object files which are then loaded (not sure about the details here). Currently all is pretty hard-wired for GCC on X86 systems. For getting a feeling for how difficult it will be to also support ARM, I'd grab these C-code files and try to compile them on the M1. If this is working, the adaptions for finding and configuring the compiler and for compiling on ARM at runtime will be relatively simple.

Pitfalls:

As Parcels was so far almost exclusively developed and used in the GCC / X86 / Unix world, I'm sure there's a few obstacles to overcome to even import it in the ARM env:

• If I remember correctly, Parcels will compile a random number generator even if used in Scipy mode?
• I'm not sure how well netCDF is supported on ARM and if Parcels would be usable without the netCDF module being available for import.
• ...

[willirath]

If supporting Scipy mode is successful, this would allow for adding a conda-package that at least supports scipy mode on ARM. Then, people could use their ARM devices for at least learning about Parcels, but still would need to switch to an X86 machine for larger production runs.

[CKehl]

Hi,

Thanks for your feedback.
First of all, I agree and join with every single one of @willirath 's statements.

I agree to try getting it run on SciPy mode first. If you get that to run easily, then that's fine - if not, it's not worth thinking about the fast JIT-mode. Thing is that our dependency packages (which themselves are build on C/C++ libraries - good to remember that they are always the base) depend on the conda-forge provided version. That has to to with format compatibility (e.g. HDF5, NetCDF) and also compiler compatibility (e.g. the libc-ng and the gcc_linux-64 packages instead of libc and gcc that are distributed by other channels such as anaconda itself). Try it out and we'll see if that works - would be amazing. Trickiest part is probably gonna be some of the more complex dependencies, such as dask, scikit-learn and the file interfaces netcdf4 and hdf5.

For JIT-mode, procedure would be like that: if SciPy mode works (pre-requisite) then it depends a bit on what compiler-package you get. To my knowledge, gcc is also the standard compiler for ARM architectures such as M1, v7, v7a and neon. If the calling convention and parameters are nowadays the same as for x86_64, then there's a good chance we can get it to work. Then we just need to include it in some OS-bridges in the code. If not, then you're out of luck of the soon future. We are very prototypically developing Numba-support these days, but it will be a considerable time before that Numba-version can do all the things our current JIT can do.

Hope that helps as feedback.

Christian

[philippemiron]

Hi everyone,

Reporting what I have done so far:

For the installation, the only missing package in the osx-arm64 channel is:

• pykdtree

You all know better than me where this is used in Parcels, but for now, I commented it from the env file to test further.

Modifications to the environment file:
- remove defaults channel as the only supported channel on miniforge is conda-forge
- change clang_osx-64 to clang_osx-arm64
- commented pykdtree
with those changes, the parcels installation completes without error.

I then went to try running a few tutorials using ScipyParticles:
- Agulhas particles
- Analytical advection
- Delayed start
- Interpolation
- Nemo Curvilinear
- NestedFields
which all completed fine.

Although, when running examples/tutorials using JITParticle, I get the following error:

sh: None: command not found
Info: Compiled ArrayJITParticleAdvectionRK4 ==> /var/_/_.so

and a kernel crash.

I saw that @erikvansebille was running into a similar error #1054 but in his case, it was non-fatal and would disappear after installing mpi4py (installed here). Any suggestions on where to go from here?

[philippemiron]

Hi everyone, reporting again. I have good and bad news.

Starting with the good news: the JITParticle actually seems to be working! I was working with one of my scripts today, and because it was pretty slow with the ScipyParticle, I decided to try to understand why the kernel was crashing. Once I switched to the JITParticle, the script ran right away without issue. I still get the message: sh: None: command not found but no issues otherwise.

This leads me to the bad news, I tried to understand why I was getting an error yesterday, and I can't figure it out. So I uninstalled the environment and reinstalled parcels, but it is still working.

Here are the full steps to install the environment file (below) and the kernel to use in jupyter-lab.

conda env create -f environment_py3_osx_arm.yml
conda activate parcels
cd parcels/
python setup.py install
ipython kernel install --user --name=parcels
conda install cartopy
conda deactivate

The only thing to figure out would be how to install pykdtree, the only missing dependency.

File: environment_py3_osx_arm.yml

name: parcels
channels:
  - conda-forge

dependencies:
  - python>=3.4
  - cachetools>=1.0.0
  - cgen
  - clang_osx-arm64
  - coverage
  - ffmpeg>=3.2.3
  - flake8>=2.1.0
  - git
  - mpi4py>=3.0.1
  - mpich>=3.2.1
  - jupyter
  - matplotlib>=2.0.2
  - netcdf4>=1.1.9
  - numpy>=1.9.1
  - progressbar2
  - py>=1.4.27
  - pymbolic
  - python-dateutil
  - scipy>=0.16.0
  - six>=1.10.0
  - xarray>=0.10.8
  - dask>=2.0
  - cftime>=1.3.1
  - pytest
  - nbval
  - scikit-learn
# - pykdtree # not available

[erikvansebille]

Great to hear, @philippemiron! The pykdtree package is used in Parcels in the new Particle-Particle interaction functionality and in ParticleSet.populate_indices() function to speed up initialisation of Particles on Curvilinear grids, see below

parcels/parcels/particleset/particlesetsoa.py

Lines 292 to 320 in 86b5e79

	def populate_indices(self):
	"""Pre-populate guesses of particle xi/yi indices using a kdtree.
	This is only intended for curvilinear grids, where the initial index search
	may be quite expensive.
	"""
	if self.fieldset is None:
	# we need to be attached to a fieldset to have a valid
	# gridset to search for indices
	return
	if KDTree is None:
	return
	else:
	for i, grid in enumerate(self.fieldset.gridset.grids):
	if not isinstance(grid, CurvilinearGrid):
	continue
	tree_data = np.stack((grid.lon.flat, grid.lat.flat), axis=-1)
	tree = KDTree(tree_data)
	# stack all the particle positions for a single query
	pts = np.stack((self._collection.data['lon'], self._collection.data['lat']), axis=-1)
	# query datatype needs to match tree datatype
	_, idx = tree.query(pts.astype(tree_data.dtype))
	yi, xi = np.unravel_index(idx, grid.lon.shape)
	self._collection.data['xi'][:, i] = xi
	self._collection.data['yi'][:, i] = yi

.

But neither of these are necessary for 'normal' Parcels use, so you could run codes well without this package.

I do see, by the way, that Ubuntu has an arm64 version of pykdtree? SO may be that it is soon ported for conda too?

[philippemiron]

Latest update on this discussion (storpipfugl/pykdtree#61). pykdtree is now available for arm64 (since Dec 2021), so there is no more need to modify the default parcels environment file to install on M1/M2 or any other ARM machines.

[erikvansebille]

Hi @philippemiron. Could you have a look at #1220? @graceebc9 seems too have an issue with the linker for osx-arm. You seem to have solved this when you installed Parcels on your ARM64-mac? Could you give some pointers how you did this?

[philippemiron]

I don't remember having linker/compiler problems... I pasted my notes on the issue.