Add support for `virtual_specs` checks before installation #809

jaimergp · 2024-06-21T09:07:28Z

Description

Adds a new key virtual_specs.

This key allows installers to run some solver checks before the installation proceeds. Meant to be used with minimum required versions for virtual packages like __osx or __glibc. We need it because the solver only runs at installer build time, not on the target machine.

We perform an offline dry-run just to check that the solver passes there. On PKG installers, we fast track the __osx checks so the native checks run as soon as possible.

Checklist - did you ...

Add a file to the news directory (using the template) for the next release's release notes?
Add / update necessary tests?
Add / update outdated documentation?

constructor/construct.py

constructor/header.sh

constructor/osxpkg.py

constructor/construct.py

examples/virtual_specs/construct.yaml

Co-authored-by: Marco Esters <mesters@anaconda.com>

jaimergp · 2024-07-27T16:34:52Z

I tried the glibc detection one-liner based on ldd in several common distros and it looks like it works?

$ docker run -it centos:7 bash
Unable to find image 'centos:7' locally
7: Pulling from library/centos
6717b8ec66cd: Pull complete 
Digest: sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4
Status: Downloaded newer image for centos:7
[root@0df9cc6512de /]# ldd --version | awk 'NR==1{print $NF}'
2.17
$ docker run -it debian:stable bash
Unable to find image 'debian:stable' locally
stable: Pulling from library/debian
9210b4f6492f: Pull complete 
Digest: sha256:6c07c2a542b2f1f477bb9b2f218e3b1f46266ec5db68801ad3129a36bff89004
Status: Downloaded newer image for debian:stable
root@54aac5e36c73:/# ldd --version | awk 'NR==1{print $NF}'
2.36
$ docker run -it ubuntu:22.04 bash 
Unable to find image 'ubuntu:22.04' locally
22.04: Pulling from library/ubuntu
4ce000a43472: Pull complete 
Digest: sha256:340d9b015b194dc6e2a13938944e0d016e57b9679963fdeb9ce021daac430221
Status: Downloaded newer image for ubuntu:22.04
root@dca70274216a:/# ldd --version | awk 'NR==1{print $NF}'
2.35
$ docker run -it opensuse/leap bash      
Unable to find image 'opensuse/leap:latest' locally
latest: Pulling from opensuse/leap
bf8f05b8d435: Pull complete 
Digest: sha256:1cf79e78bb69f39fb2f78a7c2c7ebc4b64cf8d82eb1df76cd36767a595ada7a8
Status: Downloaded newer image for opensuse/leap:latest
8c074df96d68:/# ldd --version | awk 'NR==1{print $NF}'
2.38
$ docker run -it --platform linux/amd64 archlinux:latest bash
Unable to find image 'archlinux:latest' locally
latest: Pulling from library/archlinux
d60d5665bb19: Pull complete 
ad89e6a4db72: Pull complete 
Digest: sha256:a58fb33d8c7206869f7b718844ac8c849fa6a1ed811964153d33d41d096f600f
Status: Downloaded newer image for archlinux:latest
[root@3a2206ee84a1 /]# ldd --version | awk 'NR==1{print $NF}'
2.39

marcoesters · 2024-07-30T18:50:04Z

constructor/utils.py

+            if operator == "ge":
+                specs[spec.name]["min"] = str(version.matcher_vo)
+            elif operator == "lt" and spec.name == "__osx":
+                specs[spec.name]["before"] = str(version.matcher_vo)


Maybe I'm still jet lagged, but is this used anywhere?

Yes, at https://github.com/conda/constructor/pull/809/files#diff-620cc606622328c2ee309e832c215dbf23da87df82a5b10bd09ced66c2f9b503R200 ('before' it's part of the XML attributes that key accepts)

isuruf · 2024-07-31T15:29:38Z

constructor/header.sh

+#if linux and min_glibc_version
+min_glibc_version="__MIN_GLIBC_VERSION__"
+# ldd reports glibc in the last field of the first line
+system_glibc_version=$(ldd --version | awk 'NR==1{print $NF}')


ldd on an alpine docker image with glibc installed will have a different output. ldd here would give musl version and not glibc. /lib64/ld-linux-x86-64.so.2 --version might be better.

You can use https://github.com/Docker-Hub-frolvlad/docker-alpine-glibc if you want to test.

Thanks Isuru! Added a workaround for musl systems. Invoking ld.so directly in CentOS does not work so I'll leave it with those two branches for now.

hmaarrfk · 2024-08-07T10:52:32Z

CONSTRUCT.md

+
+A list of virtual packages that must be satisfied at install time. Virtual
+packages must start with `__`. For example, `__osx>=11` or `__glibc>=2.24`.
+These specs are dry-run solved offline by the bundled `--conda-exe` binary.


Is the condo exe compatible with a much lower bound than one would otherwise expect. Say 2.12 on Linux.

Is seems that on Linux you use ldd but that on osx you use a dry run.

any way to also use shell commands on osx? I feel like that would be more robust to increases in minimum requirements in osx

It's there:

https://github.com/conda/constructor/pull/809/files#diff-d5126ee7f99e6583ba9574579ff73994eaf0eebcfdc792a6deb00a0dc2158f6eR24-R35

And also for PKG:

https://github.com/conda/constructor/pull/809/files#diff-620cc606622328c2ee309e832c215dbf23da87df82a5b10bd09ced66c2f9b503R193-R203

amazing, thanks!

Add support for virtual_specs checks before installation

46c8f19

conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Jun 21, 2024

jaimergp mentioned this pull request Jun 21, 2024

Kernel is dead message when starting Spyder spyder-ide/spyder#22179

Closed

jaimergp added 4 commits June 21, 2024 11:13

ignore shellcheck here

65ddaba

run inside PKG too (for cases other than __osx)

39b61cd

pre-commit

75dfbfa

add news

b6c4233

jaimergp force-pushed the check-os-versions branch from 75874ec to b6c4233 Compare June 21, 2024 09:32

jaimergp added 8 commits June 21, 2024 11:34

update docs

d9f7133

reveal exceptions just to make sure

a017d7d

Make sure we fail for the right reason

220f762

return the process

2960219

ignore errors, process tuple

b36495f

fix test on windows

26ccbe4

Pass as unescaped variable

39f689d

adjust error message for .pkg

3154b02

jaimergp marked this pull request as ready for review June 22, 2024 10:49

jaimergp requested a review from a team as a code owner June 22, 2024 10:49

marcoesters reviewed Jun 24, 2024

View reviewed changes

jaimergp and others added 8 commits June 25, 2024 15:10

Adjust docs

247bd45

validate virtual_specs

fd55c51

use <= for osx

5399871

add comment about solver=classic

ec575b6

typo

df02fbd

Parse , (AND) in __osx

8600666

Co-authored-by: Marco Esters <mesters@anaconda.com>

Fix PKG os-version control

adcbab9

pre-commit

263bd0e

jaimergp mentioned this pull request Jul 24, 2024

Enforcing system requirements - Miniforge3 Blocker conda-forge/miniforge#614

Closed

jaimergp added 2 commits July 25, 2024 12:01

Add Bash-only checks for SH installers

b17085e

fix tests

ab60642

jaimergp added 6 commits July 25, 2024 12:34

make it posix

8b9146c

pre-commit

553b41a

adjust expected output in test

86b958d

pre-commit

6576d53

fix dict access?

28f74d5

always provide a value even if falsey

a59d2c5

jaimergp added 2 commits July 28, 2024 13:10

Fix TypeError

858586d

better test?

4f46d6c

marcoesters reviewed Jul 30, 2024

View reviewed changes

marcoesters previously approved these changes Jul 31, 2024

View reviewed changes

isuruf reviewed Jul 31, 2024

View reviewed changes

add workaround for musl

3930766

jaimergp dismissed marcoesters’s stale review via 3930766 August 1, 2024 09:04

jaimergp added 2 commits August 1, 2024 11:10

star this too

4c6664f

use find

471bad4

marcoesters approved these changes Aug 5, 2024

View reviewed changes

jaimergp merged commit 0d275c1 into conda:main Aug 7, 2024
20 checks passed

hmaarrfk reviewed Aug 7, 2024

View reviewed changes

drewgilliam mentioned this pull request Sep 4, 2024

virtual_specs glibc #850

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for `virtual_specs` checks before installation #809

Add support for `virtual_specs` checks before installation #809

jaimergp commented Jun 21, 2024 •

edited

Loading

jaimergp commented Jul 27, 2024

marcoesters Jul 30, 2024

jaimergp Jul 31, 2024

isuruf Jul 31, 2024

isuruf Jul 31, 2024

jaimergp Aug 1, 2024

hmaarrfk Aug 7, 2024

hmaarrfk Aug 7, 2024

jaimergp Aug 7, 2024

hmaarrfk Aug 7, 2024

Add support for virtual_specs checks before installation #809

Add support for virtual_specs checks before installation #809

Conversation

jaimergp commented Jun 21, 2024 • edited Loading

Description

Checklist - did you ...

jaimergp commented Jul 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add support for `virtual_specs` checks before installation #809

Add support for `virtual_specs` checks before installation #809

jaimergp commented Jun 21, 2024 •

edited

Loading