Migrate to CKAN 2.9.4 & python 3.7

[mdutoo]

In this issue is aggregated all best practices and errors & patches encountered in migrating all CKAN extensions, since ckanext-ozwillo-theme is the most specific Ozwillo CKAN extension.

Migration overview

• python 2 is not supported anymore since 20210101 (patches, but also CA certs problems...) https://www.infoworld.com/article/3365221/python-2-end-of-life-how-to-survive-the-end-of-python-2.html
• latest CKAN (2.9.4) has been ported to python 3, mostly / seemingly : Implement changes to allow running CKAN in both Python 2 and 3 ckan/ckan#4801 . But extensions outside most common ones have mostly not.
• choosing python 3.7 provided by Debian 10 Buster (CKAN Dockerfile is based on Debian), because Debian 11 (Bullseye) comes with python 3.9 which is incompatible (buster provides python 3.7), raises error https://stackoverflow.com/questions/64264563/attributeerror-elementtree-object-has-no-attribute-getiterator-when-trying
• previous CKAN version deployed on Ozwillo : 2.8.2 (deployed source commit hash is the same as https://github.com/ozwillo/ckan/tree/ckan-2.8.2 )
• diffs : (1000+ files...) ozwillo/ckan@ckan-2.8.2...ckan:ckan-2.9.4 and 186 changes, 7000 lines in git diff ckan-2.8.2 ckan-2.9.4 -- ckan/templates/

Process

• deploy and install latest CKAN (2.9.4) on local dev machine, from source, using Docker Compose setup and official guide
• as well as main bundled extensions
• for other extensions (Ozwillo, community), try to find Python 3.7 compatible versions if any (ckanext-showcase...)
• otherwise for each extension, one by one :

# clone extension source git repo :
git clone <extension.git>
# deploy and install according to its doc (mostly README), **from source**, from within the CKAN Docker container, mostly :
docker exec -it ckan /bin/bash
. $CKAN_HOME/venv/bin/activate
cd <extension>
python setup.py develop
# AND add their plugins in production.ini's ckan.plugins
# then restart CKAN :
docker-compose restart ckan
#look at errors that appear in CKAN logs :
docker-compose logs -f ckan
# patch them (see below)
# reinstall the extension, then redeploy, until there are no more errors in the CKAN logs.

see also :
https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions
Python 3 support ckan/ckan#4681
Flask and Python 3 support on selected extension ckan/ckan#4793
https://docs.ckan.org/en/2.9/maintaining/upgrading/upgrade-to-python3.html
(old) Flask and Python 3 support on selected extensions ckan/ckan#4793

Below are common migration errors and patches.

CKAN static resources are handled by webassets rather than fanstatic

• fanstatic dir renamed to assets
• create webassets.yml descriptor (according to fanstatic's resource.config, to be deleted)
• in html "resource" becomes "asset" and moves to header.html
• css goes in a separate entry from js, ex. ozwillo-css IN ADDITION to ozwillo, else css and js get mixed ! so have to be put in separate places

NB. migrating to latest CKAN HTML (and especially Bootstrap 3) still remains to be done : #2

see https://docs.ckan.org/en/2.9/theming/webassets.html
add webassets.yml config https://docs.ckan.org/en/2.9/contributing/frontend/assets.html
reimports : see ckan/ckan/public/base/javascript/webassets.yml ckan/ckan/public/base/javascript/webassets.yml

vi ckanext-resource-location/ckanext/location/templates/package/snippets/resource_form.html
resource becomes asset
mv ckanext-resource-location/ckanext/location/fanstatic ckanext-resource-location/ckanext/location/assets
vi ckanext-resource-location/ckanext/location/fanassets/webassets.yml

Example:
webassets https://github.com/ozwillo/ckanext-ozwillo-theme/blob/master/ckanext/ozwillo_theme/assets/webassets.yml
as required by 3b51e2a

see also :
ckan/ckan@4a0ff5c
obsolete https://docs.ckan.org/en/2.8/theming/fanstatic.html
obsolete : https://docs.ckan.org/en/2.8/contributing/frontend/resources.html

CKAN uses Flask web framework instead of Pylons

Pylons config is moved to the compatibility toolkit, as is ckan.logic :

vi ckanext-mapviews/ckanext/mapviews/plugin.py
import pylons.config as config => config = p.toolkit.config # IConfigurable : The extension point works as always, but extensions should always use toolkit.config & ex. see ckan/ckan/cli/views.py https://github.com/ckan/ckan/wiki/Migration-from-Pylons-to-Flask

vi ckanext-datarequests/ckanext/datarequests/[plugin/actions].py
from ckan.plugins.toolkit import config

Pylons Controllers (& their routes) become Flask blueprint.py :

vi ckanext-ozwillo-organization-api/ckanext/ozwillo_organization_api/plugin.py
# error AttributeError: module 'ckan.lib.base' has no attribute 'BaseController': indeed, only if python2 :
# https://github.com/ckan/ckan/blob/ckan-2.9.4/ckan/plugins/toolkit.py#L323
# ckan          | ['CkanVersionException', 'DefaultDatasetForm', 'DefaultGroupForm', 'DefaultOrganizationForm', 'HelperError', 'Invalid', 'NotAuthorized', 'ObjectNotFound', 'StopOnError', 'UnknownValidator', 'ValidationError', '_', 'abort', 'add_ckan_admin_tab', 'add_public_directory', 'add_resource', 'add_template_directory', 'asbool', 'asint', 'aslist', 'auth_allow_anonymous_access', 'auth_disallow_anonymous_access', 'auth_sysadmins_check', 'c', 'chained_action', 'chained_auth_function', 'check_access', 'check_ckan_version', 'config', 'enqueue_job', 'error_shout', 'g', 'get_action', 'get_converter', 'get_endpoint', 'get_or_bust', 'get_validator', 'h', 'literal', 'mail_recipient', 'mail_user', 'missing', 'navl_validate', 'redirect_to', 'render', 'render_snippet', 'request', 'requires_ckan_version', 'side_effect_free', 'ungettext', 'url_for']
# => has to migrate controllers to Flask, as said at https://github.com/ckan/ckan/issues/4791
# and as in ckan/ckanext/datapusher/plugin.py ckan/ckanext/datapusher/blueprint.py

see also :
ckan/ckan#4791
https://docs.ckan.org/en/2.9/extensions/flask-migration.html
https://github.com/ckan/ckan/wiki/Migration-from-Pylons-to-Flask
Flask routes : https://flask.palletsprojects.com/en/2.0.x/api/
https://docs.ckan.org/en/2.9/extensions/plugins-toolkit.html

CKAN nore mode model.new_revision()

vi ckanext-ozwillo-organization-api/ckanext/ozwillo_organization_api/plugin.py
# no model.repo.new_revision() in 2.9 like in 2.8.2, see ckan/action/create.py diff & https://pythonrepo.com/repo/ckan-ckan-python-science

CKAN dataset API change

werkzeug.routing.BuildError: Could not build url for endpoint 'package.read' with values ['id']. Did you mean 'dataset.read' instead?
=>
package becomes dataset :
{{ h.link_to(h.truncate(title, truncate_title), h.url_for(controller='package', action='read', id=package.name)) }}
becomes
{{ h.link_to(h.truncate(title, truncate_title), h.url_for('%s.read' % package.type, id=package.name)) }}
in
ckanext/ozwillo_theme/templates/base.html
ckanext/ozwillo_theme/templates/home/snippets/dataset_bloc.html
ckanext/ozwillo_theme/templates/package/snippets/package_basic_fields.html
ckanext/ozwillo_theme/templates/package/snippets/package_form.html
ckanext/ozwillo_theme/templates/page.html
ckanext/ozwillo_theme/templates/snippets/package_item.html

Example :
find . -name "." -exec grep "dataset_read" {} ; -print
dataset_url = url_for('dataset_read',
./ckanext-dcat/ckanext/dcat/profiles.py
_map.connect('dataset_read', '/dataset/{_id}',
./ckanext-dcat/ckanext/dcat/plugins/pylons_plugin.py

Python2/3 Import syntax

vi ckanext-mapviews/ckanext/mapviews/plugin.py
from urllib.parse import urlparse # else ckan ModuleNotFoundError: No module named 'urlparse' https://github.com/opendatatrentino/ckan-api-client/issues/30

vi ckanext-ozwillo-pyoidc/ckanext/ozwillo_pyoidc/plugin.py
from . import conf
from .oidc..., from .blueprints...

vi ckanext-ozwillo-pyoidc/ckanext/ozwillo_pyoidc/oidc.py
from . import conf
# else  ModuleNotFoundError: No module named 'conf' => Relative imports are gone in python3 https://stackoverflow.com/a/34462008/2862821 https://wiki.python.org/moin/PortingPythonToPy3k#relative-imports

vi ckanext-ozwillo-pyoidc/ckanext/ozwillo_pyoidc/blueprints.py
from . import conf
from .oidc...

vi ckanext-datarequests/ckanext/datarequests/[actions/auth/db/helpers/plugin/validator].py
from . import...

Python2/3 Exceptions syntax

vi ckanext-ozwillo-organization-api/ckanext/ozwillo_organization_api/plugin.py
except (logic.ValidationError, e):
except (ValueError, requests.ConnectionError, e):
# else SyntaxError: invalid syntax https://stackoverflow.com/questions/20057719/how-to-fix-invalid-syntax-error-at-except-valueerror

vi ckanext-ozwillo-pyoidc/ckanext/ozwillo_pyoidc/oidc.py
except OIDCError, e: => except OIDCError as e:
and all other "except" the same

Python2/3 unicode type becomes str

vi ckanext-dashboard/ckanext/dashboard/plugin.py
unicode => str # else 'schema': {'json': [ignore_empty, unicode],  ckan          | NameError: name 'unicode' is not defined, see https://stackoverflow.com/questions/19877306/nameerror-global-name-unicode-is-not-defined-in-python-3

vi ckanext-ozwillo-organization-api/ckanext/ozwillo_organization_api/plugin.py
# since python 3, bytes are not directly str so must be encoded : https://stackoverflow.com/a/43882903/2862821
# else ERROR [ckan.views.api] key: expected bytes or bytearray, but got 'str'
computed_hmac = hmac.new(bytes(api_secret, 'utf-8'), request.get_data(), sha1).hexdigest()