Add constructor manager api #46

goanpeca · 2022-11-29T04:39:49Z

Split from #34

Overview

Constructor based installers

These installers will contain 3 main environments when created:

A base environment which will contain:
- conda (package and environment manager)
- mamba (conda "replacement" for performing faster solves, transactions and downloads)
- conda-lock (create locked and reproducible environments)
A constructor-manager environment which will contain:
- constructor-manager (Add constructor manager cli #34)
- constructor-manager-api (this PR)
- constructor-manager-ui (Update UI and connect to constructor manager API #63)
An application environment, using the conventions <package-name>-<version>, for the example image above, napari-0.4.15

More information on packaging on https://napari.org/dev/developers/packaging.html#constructor-based-installers

constructor-manager-api

This pull request is in charge of creating the constructor-manager-api package which in its present form is a qtpy (worker and signal) based library providing an API to call the constructor-manager-backend. It provides a python library to run workers and query information on the different processes managed by the constructor manager

The library is imported with from constructor_manager_api import check_updates, update, revert, restore, open_application

check_updates: Query for new updates available for the managed application by providing one or more anaconda.org channels. By default we query the conda-forge channel located at https://anaconda.org/conda-forge/
update: update a current application to a new version of it. This process will create a new conda environment following the convention <package-name>-<new-version>, create new menu shortcuts (using the menuinst branch https://github.com/conda/menuinst/tree/cep-devel/menuinst), create a new restore point (using conda-lock), remove the old environment and the corresponding shortcuts for the old versions of the managed application.
restore: similar to revert but restore to a previously found state of the current version. This command could become a single one by providing the specific restore file (which is created by conda lock)
revert: this will revert the current application to the previously installed version on the computer if a restore point is found. This will follow a similar process of creating a new conda environment with the convention <package-name>-<old-version>.
lock-environment: create a lock file of the current state of the application environment. This can be called by the applications (using constructor-manager-api) to check for changes in the environment. If no changes are detected, no lock is created.
open_manager: open the constructor manager ui for a given package.
open_application: open a give application created with conda and menuinst un a cross platform way

Some of these commands can be run in parallel others create a lock to prevent multiple instances of an update/restore/revert process.

This library is meant to be used in the application environment by the applications themselves so they can query the information to know about new updates and either trigger them directly in a detached process (to be implemented in a following version) or to open the constructor-manager-ui to handle those actions directly.

More information on the README of the package

Work in progress

Tests
Release package to pypi
Release package to conda-forge

Future work

provide non-qt based worker workflows (perhaps use async or psygnal?). Currently tied to qt because napari uses that, but should not rely on it necessarily
provide uninstall method
provide get status method of a current command execution (update, revert, reset, restore, lock)

constructor-manager/src/constructor_manager/api.py

… manager ui

goanpeca · 2023-03-13T12:59:34Z

Fixing the tests, but this one is ready for review @aganders3, @dalthviz, @potating-potato

Thanks!

dalthviz

Hi @goanpeca ! Left an initial review with a couple of suggestions to update the README description, some comments regarding docstrings and maybe a missing dependency (pyyaml) and package metadata (autor-email).

I will try later to setup the constructor-manager-cli from #34 to see if I can run the functions (like check_updates for example). But besides the comments left this LGTM 👍

dalthviz · 2023-03-13T17:00:59Z

constructor-manager-api/setup.cfg

-author_email = napari-steering-council@googlegroups.com
+url = https://github.com/napari/packaging/constructor-manager-api
+author = napari
+author_email = TODO


I guess there is not a email to use here, right?

Not sure what email we should use 🤔

@kephale do you have a suggestion ?

napari-steering-council@googlegroups.com is still the official napari SC email address, this can be reverted.

dalthviz · 2023-03-13T17:04:52Z

constructor-manager-api/setup.cfg

@@ -23,7 +23,7 @@ classifiers =
    Topic :: Scientific/Engineering :: Image Processing
 project_urls =
    Bug Tracker = https://github.com/napari/packaging/issues
-    Source Code = https://github.com/napari/packaging/constructor-manager
+    Source Code = https://github.com/napari/packaging/constructor-manager-api


Should pyyaml be added in the install_requires section of this package? I tried doing:

from constructor_manager_api.api import check_updates

and I got an error related to the import yaml done for the settings module:

>>> from constructor_manager_api.api import check_updates Traceback (most recent call last): File "<stdin>", line 1, in <module> File "E:\Acer\Documentos\Quansight\Napari\packaging-otros\gonzalo\packaging\constructor-manager-api\src\constructor_manager_api\api.py", line 10, in <module> from constructor_manager_api.utils.settings import save_settings File "E:\Acer\Documentos\Quansight\Napari\packaging-otros\gonzalo\packaging\constructor-manager-api\src\constructor_manager_api\utils\settings.py", line 5, in <module> import yaml ModuleNotFoundError: No module named 'yaml'

Yes you are right!

dalthviz · 2023-03-13T17:08:35Z

constructor-manager-api/README.md

+    print(result)
+
+
+worker = check_updates(package_name="napari", current_version="0.4.10", channel="conda-forge")


Checking the check_updates function signature I think this should be something like:

Suggested change

worker = check_updates(package_name="napari", current_version="0.4.10", channel="conda-forge")

worker = check_updates(package_name="napari", current_version="0.4.10", channels=["conda-forge"])

dalthviz · 2023-03-13T17:09:18Z

constructor-manager-api/README.md

+## Usage
+
+```python
+from constructor_manager.api import check_updates


With the directory structure change I think this should be something like:

Suggested change

from constructor_manager.api import check_updates

from constructor_manager_api.api import check_updates

Yes, but I think I prefer adding those imports imports on the init so there is no need for the second api submodule (which looks weird :)

dalthviz · 2023-03-13T17:16:26Z

constructor-manager-api/src/constructor_manager_api/api.py

+    """Check for updates.
+
+    Parameters
+    ----------
+    package_name : str
+        Name of the package to check for updates.
+    version : str
+        Version of package to execute action on, by default ``None``.
+
+    Returns
+    -------
+    ConstructorManagerWorker
+        Worker to check for updates. Includes a finished signal that returns
+        a ``dict`` with the result.
+    """


Seems like this docstring needs some changes (missing plugins_url kwarg docstring, main docstring refers to check for updates instead of package)

dalthviz · 2023-03-13T17:17:03Z

constructor-manager-api/src/constructor_manager_api/api.py

+    """Check for updates.
+
+    Parameters
+    ----------
+    package_name : str
+        Name of the package to check for updates.
+
+    Returns
+    -------
+    ConstructorManagerWorker
+        Worker to check for updates. Includes a finished signal that returns
+        a ``dict`` with the result.
+    """


Seems like this docstring needs some changes (description refers to check for updates instead of version)

dalthviz · 2023-03-13T17:19:30Z

constructor-manager-api/src/constructor_manager_api/run.py

+    # worker = check_updates(
+    #     "napari",
+    #     current_version="0.4.15",
+    #     channel="napari",
+    #     dev=True,
+    # )
+    # worker = check_updates(
+    #     "napari",
+    #     build_string="pyside",
+    #     plugins_url="https://api.napari-hub.org/plugins",
+    # )
+    # worker = check_version("napari")
+    # worker.finished.connect(_finished)
+    # worker.start()


Maybe this code could be added as part of the README as example usage of the package?

Yes that was the plan, will move it there!

aganders3

Sorry I haven't had a chance to actually run this yet. It looks good so far, just a few nitpicks.

aganders3 · 2023-03-13T16:26:30Z

constructor-manager-api/src/constructor_manager_api/api.py

+    detached = cmd != "status"
+    detached = False


ActionsEnum does not have a "status" member, but I would also compare directly with the enum instead of a raw string here. Though this is hard-coded on the next line anyway so maybe this is just old debugging code left in?

Whoops., you are right, fixing!

aganders3 · 2023-03-13T18:03:24Z

constructor-manager-api/src/constructor_manager_api/utils/conda.py

+    if (current / "envs").exists() and (current / "envs").is_dir():
+        return current
+
+    if current.parent.name == "envs" and current.parent.is_dir():


Should this also check if the path exists (as above)? It might be enough for each to just check is_dir() as I don't think it just returns False if the path doesn't exist.

Good point!

aganders3 · 2023-03-13T18:22:22Z

constructor-manager-api/src/constructor_manager_api/utils/settings.py

+    loaded_settings = _default_settings.copy()
+    if path.exists():
+        with open(path) as f:
+            data = yaml.load(f, Loader=yaml.FullLoader)


Maybe not a huge deal, but could this instead use SafeLoader? (or I think just safe_load will do the same thing)

yes, it is better. Will update!

aganders3 · 2023-03-13T18:26:38Z

constructor-manager-api/src/constructor_manager_api/utils/worker.py

+        if not self._program.is_file():
+            raise FileNotFoundError(f"Could not find {self._program}")


I don't think this is necessary as _executable() looks like it will already raise such an exception.

True, code refactor that left in there 🤦🏼

Thanks!

dalthviz · 2023-03-14T15:52:58Z

constructor-manager-api/src/constructor_manager_api/utils/worker.py

+        bin = "constructor-manager-cli"
+        envs = ["_constructor-manager", "constructor-manager", "base"]
+        for env in envs:
+            program = get_prefix_by_name(env) / "bin" / bin


Was checking on Windows and I think there instead of bin the directory where executables are stored is Scripts and the executable has the .exe extension so for Windows probably this should be something like:

bin = "constructor-manager-cli.exe" program = get_prefix_by_name(env) / "Scripts" / bin

HI @dalthviz, indeed, I ran into the same issue. Thanks for checking!, although it should be .bat the extension, no?

Double checking it appears in my env as an .exe:

I think the .bat executables exist but for conda inside the condabin directory:

On Windows entry points use a C launcher that mimics what a shebang would do.

constructor-manager-cli.exe will find:

constructor-manager-cli-script.py` next to itself

a python.exe interpreter nearby

After that it will run python.exe constructor-manager-cli-script.py.

So yes, it's an .exe in Scripts. I think a subprocess without the .exe extension still finds the required executable, but I am not sure. Better be explicit.

goanpeca self-assigned this Nov 29, 2022

goanpeca changed the title ~~Add constructor manager CLI~~ Add constructor manager api Nov 29, 2022

goanpeca mentioned this pull request Nov 29, 2022

Add package stubs and structure #48

Merged

liu-ziyang reviewed Nov 30, 2022

View reviewed changes

constructor-manager/src/constructor_manager/api.py Outdated Show resolved Hide resolved

goanpeca added 2 commits January 25, 2023 18:08

Add constructor manager CLI

f41c905

Update API to match backend and add helper method to open constructor…

2f76bcd

… manager ui

goanpeca force-pushed the constructor-cli branch from 8eab40e to 2f76bcd Compare January 26, 2023 21:06

Update API handling and open commands, add logging

0c20d70

goanpeca changed the title ~~Add constructor manager api~~ Add constructor manager client Feb 15, 2023

This was referenced Feb 15, 2023

Add constructor manager cli #34

Open

Update UI and connect to constructor manager API #63

Open

goanpeca changed the title ~~Add constructor manager client~~ Add constructor manager api Mar 8, 2023

goanpeca added 2 commits March 7, 2023 19:16

Rename package

016165b

Add settings

aca9451

goanpeca requested review from dalthviz and liu-ziyang March 13, 2023 12:25

goanpeca marked this pull request as ready for review March 13, 2023 12:29

goanpeca added 3 commits March 13, 2023 07:45

Update docstrings and types

d8b7935

Fix code style

76c66a7

Use enum for actions

3641dd2

goanpeca added 8 commits March 13, 2023 08:16

Update workflows

073cc39

Test

c0ccc01

Test

903d242

Test

9fa4de2

test

4fa4c4b

test

2a4cbc6

test

5193ed6

test

b7e970e

dalthviz reviewed Mar 13, 2023

View reviewed changes

aganders3 suggested changes Mar 13, 2023

View reviewed changes

dalthviz reviewed Mar 14, 2023

View reviewed changes

Add code review comments

dffe0cb

dalthviz mentioned this pull request Jun 26, 2024

Application manager: Decouple from napari #154

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add constructor manager api #46

Add constructor manager api #46

goanpeca commented Nov 29, 2022 •

edited

Loading

goanpeca commented Mar 13, 2023

dalthviz left a comment

dalthviz Mar 13, 2023

goanpeca May 31, 2023

kephale Jun 1, 2023

dalthviz Mar 13, 2023

goanpeca Mar 13, 2023

goanpeca May 31, 2023

dalthviz Mar 13, 2023 •

edited

Loading

goanpeca May 31, 2023

dalthviz Mar 13, 2023

goanpeca May 31, 2023

dalthviz Mar 13, 2023

goanpeca May 31, 2023

dalthviz Mar 13, 2023

dalthviz Mar 13, 2023

goanpeca Mar 13, 2023

aganders3 left a comment

aganders3 Mar 13, 2023

goanpeca Mar 13, 2023

aganders3 Mar 13, 2023

goanpeca Mar 13, 2023

aganders3 Mar 13, 2023

goanpeca Mar 13, 2023

aganders3 Mar 13, 2023

goanpeca Mar 13, 2023

dalthviz Mar 14, 2023 •

edited

Loading

goanpeca Mar 15, 2023 •

edited

Loading

dalthviz Mar 15, 2023

jaimergp Jun 2, 2023

		print(result)


		worker = check_updates(package_name="napari", current_version="0.4.10", channel="conda-forge")

	from constructor_manager.api import check_updates
	from constructor_manager_api.api import check_updates

		if not self._program.is_file():
		raise FileNotFoundError(f"Could not find {self._program}")

Add constructor manager api #46

Are you sure you want to change the base?

Add constructor manager api #46

Conversation

goanpeca commented Nov 29, 2022 • edited Loading

Overview

Constructor based installers

constructor-manager-api

Work in progress

Future work

goanpeca commented Mar 13, 2023

dalthviz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dalthviz Mar 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aganders3 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dalthviz Mar 14, 2023 • edited Loading

Choose a reason for hiding this comment

goanpeca Mar 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goanpeca commented Nov 29, 2022 •

edited

Loading

dalthviz Mar 13, 2023 •

edited

Loading

dalthviz Mar 14, 2023 •

edited

Loading

goanpeca Mar 15, 2023 •

edited

Loading