THUMBNAIL_CACHE_CONFIG = init_thumbnail_cache fails superset init

[jeffreyorourke]

Hello!

I am hoping you can provide an example of a functioning S3Cache configuration--the code suggested in the documentation doesn't seem to work...

THUMBNAIL_CACHE_CONFIG = init_thumbnail_cache produces an error when running superset init...

Environment

Python 3.7
Superset 1.0.1

Expected results

For superset init to complete successfully.

Actual results

  File "/venv/bin/superset", line 8, in <module>
    sys.exit(superset())
  File "/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/venv/lib/python3.7/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/venv/lib/python3.7/site-packages/click/core.py", line 1256, in invoke
    Command.invoke(self, ctx)
  File "/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/venv/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/venv/lib/python3.7/site-packages/flask/cli.py", line 425, in decorator
    with __ctx.ensure_object(ScriptInfo).load_app().app_context():
  File "/venv/lib/python3.7/site-packages/flask/cli.py", line 381, in load_app
    app = call_factory(self, self.create_app)
  File "/venv/lib/python3.7/site-packages/flask/cli.py", line 119, in call_factory
    return app_factory()
  File "/venv/lib/python3.7/site-packages/superset/app.py", line 69, in create_app
    raise ex
  File "/venv/lib/python3.7/site-packages/superset/app.py", line 62, in create_app
    app_initializer.init_app()
  File "/venv/lib/python3.7/site-packages/superset/app.py", line 548, in init_app
    self.configure_cache()
  File "/venv/lib/python3.7/site-packages/superset/app.py", line 570, in configure_cache
    cache_manager.init_app(self.flask_app)
  File "/venv/lib/python3.7/site-packages/superset/utils/cache_manager.py", line 48, in init_app
    **app.config["THUMBNAIL_CACHE_CONFIG"],
TypeError: 'function' object is not a mapping

Relevant lines from my superset_config.py

from flask import Flask
from flask_caching import Cache
from s3cache.s3cache import S3Cache
from superset.config import *

FEATURE_FLAGS: Dict[str, bool] = {
    "THUMBNAILS": True,
    "THUMBNAILS_SQLA_LISTENERS": True,
    "DASHBOARD_CACHE": True,
}

CACHE_CONFIG = {
    'CACHE_TYPE': 'redis',
    'CACHE_DEFAULT_TIMEOUT': 60 * 60 * 24,
    'CACHE_KEY_PREFIX': 'superset_',
    'CACHE_REDIS_HOST': 'localhost',
    'CACHE_REDIS_PORT': 6379,
    'CACHE_REDIS_DB': 1,
    'CACHE_REDIS_URL': 'redis://localhost:6379/0'
}

DATA_CACHE_CONFIG = {
    'CACHE_TYPE': 'redis',
    'CACHE_DEFAULT_TIMEOUT': 60 * 60 * 24, # 1 day default (in secs)
    'CACHE_KEY_PREFIX': 'superset_results',
    'CACHE_REDIS_URL': 'redis://localhost:6379/1',
}

# Async selenium thumbnail task will use the following user
THUMBNAIL_SELENIUM_USER = "Admin"
THUMBNAIL_CACHE_CONFIG: CacheConfig = {
    'CACHE_TYPE': 'redis',
    'CACHE_DEFAULT_TIMEOUT': 24*60*60*7,
    'CACHE_KEY_PREFIX': 'thumbnail_',
    'CACHE_NO_NULL_WARNING': True,
    'CACHE_REDIS_URL': 'redis://localhost:6379/2'
}

class CeleryConfig(object):
    BROKER_URL = "redis://localhost:6379/0"
    CELERY_IMPORTS = ("superset.sql_lab", "superset.tasks", "superset.tasks.thumbnails",)
    CELERY_RESULT_BACKEND = "redis://localhost:6379/0"
    CELERYD_LOG_LEVEL = 'DEBUG'
    CELERYD_PREFETCH_MULTIPLIER = 1
    CELERY_ACKS_LATE = True
	
CELERY_CONFIG = CeleryConfig

def init_thumbnail_cache(app: Flask) -> S3Cache:
    return S3Cache('mybucket-superset', 'thumbs_cache/')

THUMBNAIL_CACHE_CONFIG = init_thumbnail_cache

# On S3
from s3cache.s3cache import S3Cache
S3_CACHE_BUCKET = 'mybucket-superset'
S3_CACHE_KEY_PREFIX = 'sql_lab_result'
RESULTS_BACKEND = S3Cache(S3_CACHE_BUCKET, S3_CACHE_KEY_PREFIX)

How to reproduce the bug

Run superset init after having followed the instructions in the documentation at https://github.com/apache/superset/blob/4cfcaebb612bd599f5cf29bf43fc75f387f17e76/docs/src/pages/docs/installation/caching.mdx

Additional context

I am guessing that I have missed something obvious... but I have been going around the block on this for a few days and can't seem to figure it out :(

It would be great if you could advise how to fix this here, and possible also include a working example of how superset_config.py needs to be configured to enable S3Cache...

[junlincc]

#12867 (comment)

please check out similar issue and related solution ^ and let us know if it works
@jeffcarey

[jeffcarey]

I think you meant @jeffreyorourke

cc @junlincc

[zach-data]

RESULTS_BACKEND is the only cache/backend that is configured to take a function like that. Every other cache type (THUMBNAIL_CACHE_CONFIG, DATA_CACHE_CONFIG, CACHE_CONFIG) needs to be represented as a mapping since it is merged to another dict. They share an overloaded type hint that is a bit misleading when paired with the incorrect documentation.

[didva]

@jeffreyorourke
Here is an example that may be helpful for anyone facing the same issue.

def init_thumbnail_cache(app: Flask, config, cache_args, cache_options) -> S3Cache:
    return S3Cache(get_env_variable('SUPERSET_CACHE_THUMBNAIL_S3_BUCKET'), get_env_variable('SUPERSET_CACHE_THUMBNAIL_S3_KEY_PREFIX'))

THUMBNAIL_CACHE_CONFIG = {'CACHE_TYPE': 'superset_config.init_thumbnail_cache'}`