[pylons-discuss] Relationship between requests and Zope transactions

[j***@gmail.com]

Hi,

According to the documentation here
<https://docs.pylonsproject.org/projects/pyramid_tm/en/latest/>, “Pyramid
<https://docs.pylonsproject.org/projects/pyramid_tm/en/latest/glossary.html#term-pyramid>
requests [to] join the active transaction
<https://docs.pylonsproject.org/projects/pyramid_tm/en/latest/glossary.html#term-transaction>
as provided by the Python transaction
<https://pypi.org/project/transaction/> package”. Looking at the
transactions code, the default transaction manager used is a
ThreadTransactionManager
<https://github.com/zopefoundation/transaction/blob/501a2934a1031f4def92701a44ebca8a317dd8a6/transaction/_manager.py#L212>,
i.e. one transaction per Python execution thread.

Now I did follow the Pyramid-SQLAlchemy-Cookiecutter
<https://github.com/Pylons/pyramid-cookiecutter-alchemy/> recipe where DB
sessions join the request’s transaction. Considering that a single incoming
request is handled in Pyramid by a single thread (correct?) is it safe to
say that the following is true?

import transaction

@view_config(
)
def some_view(request):
# Request's transaction manager is the thread's transaction manager.
request.tm == transaction.manager
# Request's and thread's and tm's transaction is the same object.
request.tm.get() == transaction.get() # == transaction.manager.get()

Thanks!
Jens

[Michael Merickel]

Jens, by default your example is true, but it is not true in the
cookiecutter configuration. The value of request.tm is defined by the
tm.manager_hook setting and by default it is the threadlocal
transaction.manager. The cookiecutter overrides the hook (and I suggest you
do as well) to define a non-threadlocal manager configured in explicit=True
mode which will help weed out bugs accessing transactions after they have
been committed. I strongly suggest any code you write that needs the tm
should use request.tm, not transaction.manager - as the former is
configurable.

- Michael

Post by j***@gmail.com
Hi,
According to the documentation here
<https://docs.pylonsproject.org/projects/pyramid_tm/en/latest/>, “Pyramid
<https://docs.pylonsproject.org/projects/pyramid_tm/en/latest/glossary.html#term-pyramid>
requests [to] join the active transaction
<https://docs.pylonsproject.org/projects/pyramid_tm/en/latest/glossary.html#term-transaction>
as provided by the Python transaction
<https://pypi.org/project/transaction/> package”. Looking at the
transactions code, the default transaction manager used is a
ThreadTransactionManager
<https://github.com/zopefoundation/transaction/blob/501a2934a1031f4def92701a44ebca8a317dd8a6/transaction/_manager.py#L212>,
i.e. one transaction per Python execution thread.
Now I did follow the Pyramid-SQLAlchemy-Cookiecutter
<https://github.com/Pylons/pyramid-cookiecutter-alchemy/> recipe where DB
sessions join the request’s transaction. Considering that a single incoming
request is handled in Pyramid by a single thread (correct?) is it safe to
say that the following is true?
import transaction
@view_config(
)
# Request's transaction manager is the thread's transaction manager.
request.tm == transaction.manager
# Request's and thread's and tm's transaction is the same object.
request.tm.get() == transaction.get() # == transaction.manager.get()
Thanks!
Jens
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/79da9d5a-91e9-483e-b96f-b5e54cc3f1a5%40googlegroups.com
<https://groups.google.com/d/msgid/pylons-discuss/79da9d5a-91e9-483e-b96f-b5e54cc3f1a5%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[j***@gmail.com]

Thanks Michael!

You are talking about these two lines of code
<https://github.com/Pylons/pyramid-cookiecutter-alchemy/blob/c84ab5bff19d6fbef9e36295cf3f74199b7a9a50/%7B%7Bcookiecutter.repo_name%7D%7D/%7B%7Bcookiecutter.repo_name%7D%7D/models/__init__.py#L59-L60> in
the cookiecutter’s model/__init__.py, correct?

settings = config.get_settings()
settings['tm.manager_hook'] = 'pyramid_tm.explicit_manager'

That manager hook allocates its own explicit transaction manager (see code
<https://github.com/Pylons/pyramid_tm/blob/master/pyramid_tm/__init__.py#L179-L188>)
instead of using the transaction.manager default, as you’ve mentioned.

Using that manager hook I can then derive from an explicit manager and
extend with custom functionality—and that would be the recommended way of
doing so? That may help me solve a question I just asked at the transaction
Github repo (see issue #62
<https://github.com/zopefoundation/transaction/issues/62> there).

Post by Michael Merickel
Jens, by default your example is true, but it is not true in the
cookiecutter configuration. The value of request.tm is defined by the
tm.manager_hook setting and by default it is the threadlocal
transaction.manager. The cookiecutter overrides the hook (and I suggest you
do as well) to define a non-threadlocal manager configured in explicit=True
mode which will help weed out bugs accessing transactions after they have
been committed. I strongly suggest any code you write that needs the tm
should use request.tm, not transaction.manager - as the former is
configurable.
- Michael

[j***@gmail.com]

Maybe some background
 the reason why I asked the initial question is
because a few (and a growing number of) request handlers are complex; that
is, they call multiple function levels deep.

Now request.dbsession, for example, is SQLA’s current DB session tied to
the request’s transaction. To get access to it, functions keep handing that
`dbsession` reference around. As we now begin to tie more data managers to
the request’s transaction, handing their context around convolutes the
parameters handed to functions.

So I thought it would be useful if much/all of a request’s execution
context would be available through the current thread local
<https://docs.python.org/3/library/threading.html#thread-local-data>
memory. If requests during their life cycle
<https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/router.html>
would derive from threading.local and initialize such a storage area upon
construction (e.g. after tween ingress) and tear it down (e.g. after tween
egress) then deeply nested function could still access a request’s context
*without* the need to pass that context around in parameters.

Would it make sense to use a custom request factory to implement such a
thread local approach? Are there other recommendations, considering the
warnings in pyramid.threadlocal
<https://docs.pylonsproject.org/projects/pyramid/en/latest/api/threadlocal.html>
?

[Mike Orr]

If requests during their life cycle would derive from threading.local and initialize such a storage area upon construction (e.g. after tween ingress) and tear it down (e.g. after tween egress) then deeply nested function could still access a request’s context without the need to pass that context around in parameters.

This is thinking about it the wrong way around. The best solution is
to add a 'request' argument to those functions, or the specific
request subobject they need. Sometimes you can't because you have to
pass through a function that wasn't written for Pyramid, but that's
what the threadlocal functions are for.

Pylons had a magic request object and a magic "template variables"
object that the subfunctions could import; they were threadlocal proxy
stacks behind the scenes. That was to replicate the popular behavior
in Myghty (definitely) and Rails (I assume). But they caused a lot of
problems and were more trouble than they were worth. So it was
considered an improvement when we migrated to Pyramid which passes an
explicit request object instead.

Why would you want to go back to threadlocals? Even if it sometimes
uses a threadlocal function to find the Zope registry; I consider that
a flaw rather than a feature.Semantically the subobjects are
REQUEST-local, not THREAD-local, so why not treat them that way?

[Jonathan Vanasco]

+1 on being explicit by passing around the request (or something similar).
there are way too many edge cases when you try to compute the current
request/transaction, and it's incredibly hard (a nightmare) to write tests
that work correctly when you rely on the derived/computed methods.

[j***@gmail.com]

Thank you Mike for the information, that all makes sense. I ended up with
the same approach that SQLA takes: a “Session” object owned by a
DataManager, joined to the Request’s TransactionManager. And then just pass
a `dbsession` and `fnsession` and `jobsession` around to functions as
needed.

Added yet another one or two parameters (sigh) to functions, but it keeps
things functional and without outside (global, thread-local) state.

[Mike Orr]

functools.partal might help in some cases.

Thank you Mike for the information, that all makes sense. I ended up with the same approach that SQLA takes: a “Session” object owned by a DataManager, joined to the Request’s TransactionManager. And then just pass a `dbsession` and `fnsession` and `jobsession` around to functions as needed.
Added yet another one or two parameters (sigh) to functions, but it keeps things functional and without outside (global, thread-local) state.
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/148c7bec-2d75-4525-9aa4-3902a4a2f2cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Merickel]

Post by j***@gmail.com
Added yet another one or two parameters (sigh) to functions, but it keeps
things functional and without outside (global, thread-local) state.

This is where something like pyramid_services [1] really excels. It'll
drastically cut down on concerns like that when you adopt some form of
inversion-of-control pattern in your codebase.

[1] https://github.com/mmerickel/pyramid_services

Post by j***@gmail.com
Thank you Mike for the information, that all makes sense. I ended up with
the same approach that SQLA takes: a “Session” object owned by a
DataManager, joined to the Request’s TransactionManager. And then just pass
a `dbsession` and `fnsession` and `jobsession` around to functions as
needed.
Added yet another one or two parameters (sigh) to functions, but it keeps
things functional and without outside (global, thread-local) state.
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/148c7bec-2d75-4525-9aa4-3902a4a2f2cb%40googlegroups.com
<https://groups.google.com/d/msgid/pylons-discuss/148c7bec-2d75-4525-9aa4-3902a4a2f2cb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[j***@gmail.com]

Maybe some background
 the reason why I asked the initial question is
because a few (and a growing number of) request handlers are complex; that
is, they call multiple function levels deep.

Now request.dbsession, for example, is SQLA’s current DB session tied to
the request’s transaction. To get access to it, functions keep handing that
`dbsession` reference around. As we now begin to tie more data managers to
the request’s transaction, handing their context around convolutes the
parameters handed to functions.

So I thought it would be useful if much/all of a request’s execution
context would be available through the current thread local
<https://docs.python.org/3/library/threading.html#thread-local-data> memory.
However, reading Why You Shouldn’t Abuse Thread Local
<https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/threadlocals.html#why-you-shouldn-t-abuse-thread-locals> (see
the pyramid.threadlocal
<https://docs.pylonsproject.org/projects/pyramid/en/latest/api/threadlocal.html>
module) recommends against such an approach. Are there more recent and
different recommendations?

[Michael Merickel]

Post by j***@gmail.com
You are talking about these two lines of code
<https://github.com/Pylons/pyramid-cookiecutter-alchemy/blob/c84ab5bff19d6fbef9e36295cf3f74199b7a9a50/%7B%7Bcookiecutter.repo_name%7D%7D/%7B%7Bcookiecutter.repo_name%7D%7D/models/__init__.py#L59-L60> in
the cookiecutter’s model/__init__.py, correct?

Right.

Using that manager hook I can then derive from an explicit manager and

Post by j***@gmail.com
extend with custom functionality—and that would be the recommended way of
doing so? That may help me solve a question I just asked at the transaction
Github repo (see issue #62
<https://github.com/zopefoundation/transaction/issues/62> there).

Yes, if you don't have a consistent location in your code where you can
push/pop threadlocals then you're going to have a bad time. For something
like the `tm` you'd still be wise to use your own threadlocal manager that
is set to explicit mode versus using the transaction.manager shipped with
the transaction project. You'd probably make a manager and register an
ISynchronizer that pushed/popped the threadlocal so that it's only
available when a transaction is open.

Would it make sense to use a custom request factory to implement such a

Post by j***@gmail.com
thread local approach? Are there other recommendations, considering the
warnings in pyramid.threadlocal
<https://docs.pylonsproject.org/projects/pyramid/en/latest/api/threadlocal.html>
?

So I thought it would be useful if much/all of a request’s execution

Post by j***@gmail.com
context would be available through the current thread local
<https://docs.python.org/3/library/threading.html#thread-local-data> memory.
However, reading Why You Shouldn’t Abuse Thread Local
<https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/threadlocals.html#why-you-shouldn-t-abuse-thread-locals> (see
the pyramid.threadlocal
<https://docs.pylonsproject.org/projects/pyramid/en/latest/api/threadlocal.html> module)
recommends against such an approach. Are there more recent and different
recommendations?

Sure you could do this with a custom request factory. The recommendations
are not "out of date", they are about general programming paradigms that
you should avoid if you want to write testable/reusable code. Pyramid tries
to avoid *forcing* you to use threadlocals anywhere - but it's not going to
try to stop you if you want to do it.

Post by j***@gmail.com
Maybe some background
 the reason why I asked the initial question is
because a few (and a growing number of) request handlers are complex; that
is, they call multiple function levels deep.
Now request.dbsession, for example, is SQLA’s current DB session tied to
the request’s transaction. To get access to it, functions keep handing that
`dbsession` reference around. As we now begin to tie more data managers to
the request’s transaction, handing their context around convolutes the
parameters handed to functions.
So I thought it would be useful if much/all of a request’s execution
context would be available through the current thread local
<https://docs.python.org/3/library/threading.html#thread-local-data> memory.
However, reading Why You Shouldn’t Abuse Thread Local
<https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/threadlocals.html#why-you-shouldn-t-abuse-thread-locals> (see
the pyramid.threadlocal
<https://docs.pylonsproject.org/projects/pyramid/en/latest/api/threadlocal.html>
module) recommends against such an approach. Are there more recent and
different recommendations?
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/5b86bb49-8fb5-4ebe-ad2b-c3a4da52bad3%40googlegroups.com
<https://groups.google.com/d/msgid/pylons-discuss/5b86bb49-8fb5-4ebe-ad2b-c3a4da52bad3%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.