sync/async worker support

[atarkowska]

This PR doesn't resolve download issue, only makes deployment parameters clear and easy configurable

This PR deprecates existing commend line args and allows fully customize web deployment using parameters, see documentation https://www.openmicroscopy.org/site/support/omero5.2-staging/sysadmins/unix/install-web/install-nginx.html

• Test Default:
  • pip install -r share/web/requirements-py27-nginx.txt
  • pip install futures
  • bin/omero web start
  • test additional params:
    • omero.web.wsgi_workers (2 x NUM_CORES) + 1
    • omero.web.wsgi_threads (2-4 x NUM_CORES)
    • omero.web.application_server.max_requests 500
    • omero.web.wsgi_timeout 30
    • omero.web.wsgi_args additional arguments
• Test Async:
  • pip install -r share/web/requirements-py27-nginx.txt
  • pip install gevent
  • bin/omero web start
  • test additional params:
    • omero.web.wsgi_workers (2 x NUM_CORES) + 1
    • omero.web.wsgi_worker_connections 1000
    • omero.web.wsgi_worker_class gevent
    • omero.web.application_server.max_requests 0
    • omero.web.wsgi_timeout 30
    • omero.web.wsgi_args Additional arguments

always check ps aux | grep /home/omero/OMERO.server/var/django.pid if parameters were correctly applied

gunicorn -D -p /home/omero/OMERO.server/var/django.pid --bind 127.0.0.1:4080 --workers 5 --threads 5 --timeout 60 --max-requests 500 omeroweb.wsgi:application

or

gunicorn -D -p /home/omero/OMERO.server/var/django.pid --bind 127.0.0.1:4080 --workers 5 --worker-connections 1000 --worker-class gevent --timeout 60 --max-requests 0 omeroweb.wsgi:application

[chris-allan]

All Gunicorn WSGI command line features seem to be working nicely.

Your expectation is that omero.web.wsgi_worker_class gevent is used to test the asynchronous workers, correct? This is missing from the current testing outline in the description.

Given that we are unable to guarantee OMERO.web thread safety, it was never designed or tested in such an environment, I would strongly suggest we label omero.web.wsgi_threads as experimental and for developer use only. Even if we stated this as a goal for the next release, the testing burden would be enormous especially in light of the potential Ice thread interplay. I would apply the same criteria to the use of gevent or eventlet asynchronous configurations.

/cc @joshmoore, @jburel

[atarkowska]

@chris-allan thank you for your help with testing. I moved recommended settings to development section. Documentation was updated accordingly. We performed brief testing in https://trello.com/c/TLsw5ffl/73-testing-async-workers. Definitely more test needed.

I am sorry worker_class was missing from the description. I am glade was easy to find that in settings directly.

[chris-allan]

Definitely wasn't a problem to find in settings directly after I realized it shouldn't be as difficult as I was making it. Since I have significant prior experience with Gunicorn I was adding omero.web.wsgi_args '-k gevent'. No harm in testing both ways of doing things. :)

With respect to testing threaded or eventlet worker modes, I'm less worried about downloads specifically and more worried about the application as a whole. Especially custom template tags. Any shared state in package variables, classes or similar exposes potential race conditions, crashes, and other safety issues when run in a threaded environment.

Quite simply, it is incredibly dangerous to take the amount of code we have in OMERO.web and just assume it is completely thread safe. Especially when that was never a stated design or development consideration to begin with.

Adding separate comments to ome/omero-documentation#1396 regarding the location of experimental feature documentation on the page.

/cc @will-moore

[jburel]

@hflynn: the change gunicorn>=19.3 will have to reflected in the scripts in omero-install (then the doc) I will take care of it. since we are not yet using the requirements file (due to structure currently in place)

[jburel]

Merging thanks all

[jburel]

PRs now opened:

• Gunicorn version omero-install#92