[pylons-discuss] How to send task-message after request.tm has committed?

[j***@gmail.com]

Hi,

Within a view function it becomes necessary to run an async task. Celery
<http://www.celeryproject.org/>, Dramatiq <https://dramatiq.io/>, etc are
frameworks to handle that. However, my async task needs to read db objects
which have been created—and must be committed and persisted—by the view
function that spawns the task. Thus, I need to delay spawning the task
(i.e. sending the task’s message to the workers) until after the request’s
db transaction has committed.

The db transaction is a standard SQLAlchemy transaction. Using the commit
veto hook
<https://docs.pylonsproject.org/projects/pyramid_tm/en/latest/#adding-a-commit-veto-hook>
is too early. This SO answer
<https://stackoverflow.com/questions/17499171/pyramid-zopetransactionextension-with-celery-how-can-i-commit-transaction-immed#17507093>
suggests using the repoze.tm.after_end()
<http://repozetm2.readthedocs.io/en/latest/#cleanup> callback, but the
answer is about five years old now.

How do I handle this scenario correctly, i.e. how do I schedule an async
job after a request was successful? I guess I can pack up the task object,
or a signature, and its parameters and attach them to the request from
within the view function. But where is the right place to actually send the
task message?

Thanks!
Jens

[j***@gmail.com]

Poking around the documentation some more, it seems that using an
after-commit-hook
<http://zodb.readthedocs.io/en/latest/transactions.html#after-commit-hooks>
may work: i.e. when a task needs to be scheduled by the view function, then
find the request’s transaction (from `request.tm`?) and install the hook
function and and its arguments (i.e. which task needs to be scheduled and
the task’s args).

But still, what is the official and recommended way of solving this? 🀔

Cheers!

[Thierry Florac]

Hi,
Maybe you should have a look at the "DataManager" component interface
provided by the "transaction" package.
This kind of component allows you to include your hook in a "two-phase
commit" transaction safely.
Several implementations are available, for example in the "repoze.sendmail"
package...

Best regards,
Thierry

Post by j***@gmail.com
Poking around the documentation some more, it seems that using an
after-commit-hook
<http://zodb.readthedocs.io/en/latest/transactions.html#after-commit-hooks>
may work: i.e. when a task needs to be scheduled by the view function, then
find the request’s transaction (from `request.tm`?) and install the hook
function and and its arguments (i.e. which task needs to be scheduled and
the task’s args).
But still, what is the official and recommended way of solving this? 🀔
Cheers!
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/25d30da3-5079-46f7-b15a-2ffe3d44c06e%40googlegroups.com
<https://groups.google.com/d/msgid/pylons-discuss/25d30da3-5079-46f7-b15a-2ffe3d44c06e%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[j***@gmail.com]

Hello Thierry,

You are talking about these data
managers: http://zodb.readthedocs.io/en/latest/transactions.html#data-managers
Correct? Other than the Mailer <https://github.com/Pylons/pyramid_mailer>
it seems a little difficult to find more verbose documentation?

I have played a little more with the after-commit hook and this short piece
of code seems to work:

@view_config(


)
def some_view(request):

# Do some view stuff, modify db objects, etc.

# Local helper function that issues the message to the broker. This
# function has access to the closure but beware that db objects are
# committed and invalid!
def _send_message(success, *args, **kwargs):
if success:
# Send message to the broker, thus spawning a worker task.
dramatiq_actors.some_task.send(*args) # Or Celery tasks,
whatever.

# Get the transaction for this request.
t = request.tm.get()

# Arguments for the task.
args = (
foo,
3,
)
kwargs = {}

# Add the after-commit hook, i.e. call _send_message() when the
# request's transaction has committed.
t.addAfterCommitHook(_send_message, args=args, kws=kwargs)

# And return from the view function.
return {
"some": "results",
}

It looks to me that this is an acceptable solution if the called hook
function can not fail (i.e. return success/failure) and therewith can not
contribute to the transaction to fail/succeed. However, a data manager
solution makes sense if data must be managed and processing/persisting that
data may fail which impacts the overall transaction.

Any recommendations here?

Cheers,
Jens

Post by Thierry Florac
Hi,
Maybe you should have a look at the "DataManager" component interface
provided by the "transaction" package.
This kind of component allows you to include your hook in a "two-phase
commit" transaction safely.
Several implementations are available, for example in the
"repoze.sendmail" package...
Best regards,
Thierry

[Mikko Ohtamaa]

Hi Jens,

You can also look here for inspiration:

https://websauna.org/docs/narrative/misc/task.html#delayed-tasks

Post by j***@gmail.com
Hi,
Within a view function it becomes necessary to run an async task. Celery
<http://www.celeryproject.org/>, Dramatiq <https://dramatiq.io/>, etc are
frameworks to handle that. However, my async task needs to read db objects
which have been created—and must be committed and persisted—by the view
function that spawns the task. Thus, I need to delay spawning the task
(i.e. sending the task’s message to the workers) until after the request’s
db transaction has committed.
The db transaction is a standard SQLAlchemy transaction. Using the commit
veto hook
<https://docs.pylonsproject.org/projects/pyramid_tm/en/latest/#adding-a-commit-veto-hook>
is too early. This SO answer
<https://stackoverflow.com/questions/17499171/pyramid-zopetransactionextension-with-celery-how-can-i-commit-transaction-immed#17507093>
suggests using the repoze.tm.after_end()
<http://repozetm2.readthedocs.io/en/latest/#cleanup> callback, but the
answer is about five years old now.
How do I handle this scenario correctly, i.e. how do I schedule an async
job after a request was successful? I guess I can pack up the task object,
or a signature, and its parameters and attach them to the request from
within the view function. But where is the right place to actually send the
task message?
Thanks!
Jens
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit https://groups.google.com/d/
msgid/pylons-discuss/ef6bf67d-acec-4dae-b318-7749743add31%
40googlegroups.com
<https://groups.google.com/d/msgid/pylons-discuss/ef6bf67d-acec-4dae-b318-7749743add31%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.