Dependency conflict with Sceptre v2.7.1 and Jinja2 v2.11.3

[jlehtiniemi]

Subject of the issue

Jinja2 v2.11.3 causes dependency issue due to deprecation of soft_unicode function in latest release of MarkupSafe v2.1.0 (dependency of Jinja2).

  File "/.../venv/lib/python3.9/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/.../venv/lib/python3.9/site-packages/markupsafe/__init__.py)

Your environment

• version of sceptre: 2.7.1
• version of python: 3.9.10
• which OS/distro: MacOS Monterey 12.2.1

❯ pip freeze
attrs==21.4.0
aws-sam-translator==1.42.0
boto3==1.21.2
botocore==1.24.2
certifi==2021.10.8
cfgv==3.3.1
cfn-flip==1.3.0
cfn-lint==0.58.0
charset-normalizer==2.0.12
click==8.0.3
colorama==0.4.4
CustomResolvers==0.3.3
decorator==4.4.2
deepdiff==5.7.0
distlib==0.3.4
filelock==3.6.0
identify==2.4.10
idna==3.3
Jinja2==2.11.3
jmespath==0.10.0
jschema-to-python==1.2.3
jsonpatch==1.32
jsonpickle==2.1.0
jsonpointer==2.2
jsonschema==3.2.0
junit-xml==1.9
MarkupSafe==2.1.0
networkx==2.5.1
nodeenv==1.6.0
ordered-set==4.0.2
packaging==16.8
pbr==5.8.1
pcli-account-resolver==0.1.0
platformdirs==2.5.0
pre-commit==2.17.0
pyparsing==3.0.7
pyrsistent==0.18.1
python-dateutil==2.8.2
PyYAML==5.4.1
requests==2.27.1
ruamel.yaml==0.15.100
s3transfer==0.5.1
sarif-om==1.0.4
sceptre==2.7.1
sceptre-cmd-resolver==1.2.0
sceptre-file-resolver==1.0.4
sceptre-ssm-resolver==1.2.1
sceptre-stack-termination-protection-hook==1.1.0
six==1.16.0
toml==0.10.2
urllib3==1.26.8
virtualenv==20.13.1

Steps to reproduce

Expected behaviour

Actual behaviour

Sceptre crashes with stacktrace:

  File "/.../venv/lib/python3.9/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/.../venv/lib/python3.9/site-packages/markupsafe/__init__.py)

Proposed solution

Upgrade Sceptre's Jinja2 dependency to v3.x.x

[heikkima]

This is related the jinja2 issue pallets/jinja#1587

[davidism]

You are using an unsupported version of Jinja, please update to the latest version. Additionally, please read https://hynek.me/articles/semver-will-not-save-you/, then use a tool like pip-tools to pin your dependencies and control when you get updates. Be sure to run your tests with deprecation warnings treated as errors so that you get notified of these types of changes early.

[jlehtiniemi]

A working work-around is to fix the MarkupSafe version to MarkupSafe==2.0.1 in your local dependencies. This can be used to avoid the issue that causes Sceptre to crash until the Sceptre's dependencies are updated.

[jfalkenstein]

@jlehtiniemi you have suggested that this be fixed within Sceptre. What are you suggesting we would do? Upgrade Jinja? Or require MarkupSafe<2.1?

[jlehtiniemi]

@jfalkenstein Upgrading Sceptre's Jinja2 dependency to a supported version as @davidism suggested would be the obvious solution. Updated my post above, as it may have been a bit unclear regarding this.

[zaro0508]

the markupsafe change breaks all sorts of tools that depend on jinja2 ver 2.x, pallets/markupsafe#284

I first noticed it in my pre-commit plugin that also depends on jinja2 ver 2.x. I tried the workaround of setting markupsafe==2.0.1 however that didn't fix the issue for me. I think the only viable option is to update sceptre's Jinja2 version to 3.x

[jfalkenstein]

For the record, this markupsafe thing also caused me some problems with SAM CLI. Seems like it's hitting a bunch of places.

aws/aws-sam-cli#3661