AttributeError: 'TmpStore' object has no attribute 'sync'

[hdm-benni]

QUESTION

Using Zope 5.1 with ZEO and multiple Zope instances we are getting the following exception when serving objects from ZODB:
AttributeError: 'TmpStore' object has no attribute 'sync'

What I did:

Unfortunately we cannot trace back the exception to one transaction or object access, it happens randomly while serving different objects of different types (mainly on images).
When the same object is served multiple times, sometimes the exception isn't raised on the second or third try.

What I expect to happen:

Object gets served without exception.

What actually happened:

Exemplary error message:

2021-02-25 15:42:17 ERROR [waitress:358][waitress-2] Exception while serving /VirtualHostBase/https/www.hdm-stuttgart.de:443/VirtualHostRoot/icons/ico_linkedin_white.png
Traceback (most recent call last):
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/ZPublisher/WSGIPublisher.py", line 164, in transaction_pubevents
tm.begin()
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/transaction/_manager.py", line 248, in begin
return self.manager.begin()
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/transaction/_manager.py", line 80, in begin
_new_transaction(txn, self._synchs)
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/transaction/_manager.py", line 52, in _new_transaction
synchs.map(lambda s: s.newTransaction(txn))
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/transaction/weakset.py", line 62, in map
f(elt)
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/transaction/_manager.py", line 52, in <lambda>
synchs.map(lambda s: s.newTransaction(txn))
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/ZODB/Connection.py", line 736, in newTransaction
self._storage.sync(sync)
AttributeError: 'TmpStore' object has no attribute 'sync'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/waitress/channel.py", line 350, in service
task.service()
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/waitress/task.py", line 171, in service
self.execute()
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/waitress/task.py", line 441, in execute
app_iter = self.channel.server.application(environ, start_response)
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/ZPublisher/httpexceptions.py", line 30, in __call__
return self.application(environ, start_response)
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/paste/translogger.py", line 69, in __call__
return self.application(environ, replacement_start_response)
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/ZPublisher/WSGIPublisher.py", line 375, in publish_module
with transaction_pubevents(request, response):
File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/ZPublisher/WSGIPublisher.py", line 220, in transaction_pubevents
tm.abort()
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/transaction/_manager.py", line 260, in abort
return self.manager.abort()
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/transaction/_manager.py", line 139, in abort
return self.get().abort()
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/transaction/_transaction.py", line 565, in abort
self._synchronizers.map(lambda s: s.afterCompletion(self))
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/transaction/weakset.py", line 62, in map
f(elt)
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/transaction/_transaction.py", line 565, in <lambda>
self._synchronizers.map(lambda s: s.afterCompletion(self))
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/ZODB/Connection.py", line 757, in afterCompletion
self.newTransaction(transaction, False)
File "/opt/zope5/zope5.1/lib/python3.7/site-packages/ZODB/Connection.py", line 736, in newTransaction
self._storage.sync(sync)
AttributeError: 'TmpStore' object has no attribute 'sync'

What version of Python and Zope/Addons I am using:

• Debian 10
• Python 3.7
• Zope 5.1
• WSGI server: waitress

We are using ZEO in combination with 4 Zope instances (created using mkwsgiinstance).
We could also reproduce the error when only using one instance.

We upgraded our Zope from Zope 2.13.29 to Zope 5.1 (which is admittedly a huge jump) recently, since then the issue occurs.
We upgraded using the recommended upgrade path from Zope 2 to Zope 4 and 5.

I'm very new to the Zope ecosystem, so I may have posted this under the wrong project and without necessary information.
If so, I'm very sorry and hope I didn't cause any trouble.

Thank you for any help, it is greatly appreciated :)

[jamadden]

TmpStore is an object that the Connection uses when it has been asked to make a savepoint(). If you then attempt to begin() a new (non-explicit) transaction without aborting or committing the current transaction, you get this error.

>>> from ZODB.DB import DB
>>> from ZODB.DemoStorage import DemoStorage
>>> from transaction import manager
>>> db = DB(DemoStorage())
>>> conn = db.open()
>>> conn.savepoint()
<ZODB.Connection.Savepoint object at 0x10189d960>
>>> manager.begin()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "//lib/python3.9/site-packages/transaction/_manager.py", line 248, in begin
    return self.manager.begin()
  File "//lib/python3.9/site-packages/transaction/_manager.py", line 80, in begin
    _new_transaction(txn, self._synchs)
  File "//lib/python3.9/site-packages/transaction/_manager.py", line 52, in _new_transaction
    synchs.map(lambda s: s.newTransaction(txn))
  File "//lib/python3.9/site-packages/transaction/weakset.py", line 62, in map
    f(elt)
  File "//lib/python3.9/site-packages/transaction/_manager.py", line 52, in <lambda>
    synchs.map(lambda s: s.newTransaction(txn))
  File "//lib/python3.9/site-packages/ZODB/Connection.py", line 736, in newTransaction
    self._storage.sync(sync)
AttributeError: 'TmpStore' object has no attribute 'sync'

Aborting the transaction also produces this error. Unfortunately, the Connection object is now in a bad state, such that closing it to put it back in the pool and then opening it again will continue to produce this error.

>>> conn.close()
None
>>> conn = db.open()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "//lib/python3.9/site-packages/ZODB/DB.py", line 777, in open
    result.open(transaction_manager)
  File "//lib/python3.9/site-packages/ZODB/Connection.py", line 921, in open
    self.newTransaction(None, False)
  File "//lib/python3.9/site-packages/ZODB/Connection.py", line 736, in newTransaction
    self._storage.sync(sync)
AttributeError: 'TmpStore' object has no attribute 'sync'

Fixing it requires a few iterations. The ZODB connection could probably handle this better...

Unfortunately, I don't know anything about ZPublisher or core Zope transaction management, so I can't suggest where to look other than look closely at any place using a savepoint().