[pylons-discuss] processing after responding to client

[Zsolt Ero]

On a fairly classic Pyramid app (sync, gunicorn, SQLAlchemy / Postgresql,
like the starter template), I am storing analytics-like events in a table
for some of my views.

Like this skeleton:

@view_config(route_name='test', request_method='POST', renderer='json')
def test(request):
# body of the view
map_id = request.json_body.get('map_id')
data = {...}

event_data = {'action': 'viewed'}
event = MapEvent(user=request.user, action=event_data, map_id=map_id)
request.dbsession.add(event)

return data



My problem is that while 99% of the views make read-only requests to the
database, and thus are very fast, the analytics event is a writing and can
be slow occasionally.

Would it be possible to somehow send the request to the client and still
keep processing the view? Like a send() + end() method or something
similar, without returning?

// Adding to a tasks queue is not an option as it'd be an overkill and an
overcomplicated solution, which wouldn't work for hundreds of thousands of
events per day.

[Tim Hoffman]

Why not publish to a redis qiueue, and then use a separate workers to do
the writes.

redis publish is light weight and fast

T

Post by Zsolt Ero
On a fairly classic Pyramid app (sync, gunicorn, SQLAlchemy / Postgresql,
like the starter template), I am storing analytics-like events in a table
for some of my views.
@view_config(route_name='test', request_method='POST', renderer='json')
# body of the view
map_id = request.json_body.get('map_id')
data = {...}
event_data = {'action': 'viewed'}
event = MapEvent(user=request.user, action=event_data, map_id=map_id)
request.dbsession.add(event)
return data
My problem is that while 99% of the views make read-only requests to the
database, and thus are very fast, the analytics event is a writing and can
be slow occasionally.
Would it be possible to somehow send the request to the client and still
keep processing the view? Like a send() + end() method or something
similar, without returning?
// Adding to a tasks queue is not an option as it'd be an overkill and an
overcomplicated solution, which wouldn't work for hundreds of thousands of
events per day.

[Mike Orr]

I use Redis to queue mail for Sendmail and to record usage stats. I
made a 'request.mailer' object with a 'send' method that takes an
email message (bytes with headers) and uses 'pyramid.tm''s two-phase
transaction to append it to a local list, and then in commit to append
it to a Redis list. I had to do that because you can't roll back in
Redis. Then a half-hourly cron job sends the messages.

It was all straightforward except for setting up the two-phased
commit. That required reading the source of 'pyramid_transaction' and
its dependencies to figure out how to attach the mailer to the
transaction. That required adding several methods, several of which
are empty because they're irrelevant.

Why not publish to a redis qiueue, and then use a separate workers to do the writes.
redis publish is light weight and fast
T

Post by 'andi' via pylons-discuss
On a fairly classic Pyramid app (sync, gunicorn, SQLAlchemy / Postgresql, like the starter template), I am storing analytics-like events in a table for some of my views.
@view_config(route_name='test', request_method='POST', renderer='json')
# body of the view
map_id = request.json_body.get('map_id')
data = {...}
event_data = {'action': 'viewed'}
event = MapEvent(user=request.user, action=event_data, map_id=map_id)
request.dbsession.add(event)
return data
My problem is that while 99% of the views make read-only requests to the database, and thus are very fast, the analytics event is a writing and can be slow occasionally.
Would it be possible to somehow send the request to the client and still keep processing the view? Like a send() + end() method or something similar, without returning?
// Adding to a tasks queue is not an option as it'd be an overkill and an overcomplicated solution, which wouldn't work for hundreds of thousands of events per day.

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/bd6f6169-92dc-47c4-933f-60eb801a1c3a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ian Wilson]

Are these calls from your own template/client? If the records are not
critical, ie. for billing, throttling, have you considered just firing a
second js request to the background *in the client* to record the event
after the first ajax request is processed. Or if there are a lot of
requests per session you could batch them together occasionally like 10 or
20 events to 1 analytic request. Without batching that would double the
amount of requests you would have to process though, but maybe would be
simpler than other solutions if the event accuracy is not critical. Just
throwing that out there.

Post by Zsolt Ero
On a fairly classic Pyramid app (sync, gunicorn, SQLAlchemy / Postgresql,
like the starter template), I am storing analytics-like events in a table
for some of my views.
@view_config(route_name='test', request_method='POST', renderer='json')
# body of the view
map_id = request.json_body.get('map_id')
data = {...}
event_data = {'action': 'viewed'}
event = MapEvent(user=request.user, action=event_data, map_id=map_id)
request.dbsession.add(event)
return data
My problem is that while 99% of the views make read-only requests to the
database, and thus are very fast, the analytics event is a writing and can
be slow occasionally.
Would it be possible to somehow send the request to the client and still
keep processing the view? Like a send() + end() method or something
similar, without returning?
// Adding to a tasks queue is not an option as it'd be an overkill and an
overcomplicated solution, which wouldn't work for hundreds of thousands of
events per day.

['andi' via pylons-discuss]

I don’t know if there is such a solution inside of the Pyramid scope. Either way, I wouldn’t do it, because it would


Mix business relevant logic with non- business relevant logic in one responsibility level. The latter must not have the ability to influence the first, which is not possible to establish if they are operating on the same level.
It would be difficult to manage DB transactions independently if you are operating in one request scope.


As you say, you’re not willing to put additional infrastructure complexity on that (which sounds fair to me for the given problem), I would look for a solution outside of your service.



Personally I would emit HTTP response headers, which contain your events information. Those can be processed in Nginx, e.g. simply logged in json format, to a separate logfile (which is your events log then).  Once you are there, you are free to decide in which cycle, with which tools to continue.



Andi





From: <pylons-***@googlegroups.com> on behalf of Zsolt Ero <***@gmail.com>
Reply-To: <pylons-***@googlegroups.com>
Date: Saturday, 10. November 2018 at 23:25
To: pylons-discuss <pylons-***@googlegroups.com>
Subject: [pylons-discuss] processing after responding to client



On a fairly classic Pyramid app (sync, gunicorn, SQLAlchemy / Postgresql, like the starter template), I am storing analytics-like events in a table for some of my views.



Like this skeleton:



@view_config(route_name='test', request_method='POST', renderer='json')
def test(request):
# body of the view
map_id = request.json_body.get('map_id')
data = {...}

event_data = {'action': 'viewed'}
event = MapEvent(user=request.user, action=event_data, map_id=map_id)
request.dbsession.add(event)

return data



My problem is that while 99% of the views make read-only requests to the database, and thus are very fast, the analytics event is a writing and can be slow occasionally.



Would it be possible to somehow send the request to the client and still keep processing the view? Like a send() + end() method or something similar, without returning?



// Adding to a tasks queue is not an option as it'd be an overkill and an overcomplicated solution, which wouldn't work for hundreds of thousands of events per day.
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/c44e1af6-8c30-478e-9baf-d7fd8c93e0b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thierry Florac]

I have several use cases where a request generates long operations (for
example, for doing conversion of an uploaded video file) but I want that
the end user receives the response as soon as possible (here, before
conversion ends).
In these cases, if the long operation is not impacted by the status of my
current transaction, and as I don't want to be dependent of a client-side
call, my choice was to:
- start a 0MQ listening process on application startup
- when a given request is received, just start a new thread to send the
required arguments to the listening 0MQ process, and send response
immediately! If this step is just dependent of current transaction status,
just be use to use an after-commit hook or a transaction data-manager...

Thierry

Post by Zsolt Ero
On a fairly classic Pyramid app (sync, gunicorn, SQLAlchemy / Postgresql,
like the starter template), I am storing analytics-like events in a table
for some of my views.
@view_config(route_name='test', request_method='POST', renderer='json')
# body of the view
map_id = request.json_body.get('map_id')
data = {...}
event_data = {'action': 'viewed'}
event = MapEvent(user=request.user, action=event_data, map_id=map_id)
request.dbsession.add(event)
return data
My problem is that while 99% of the views make read-only requests to the
database, and thus are very fast, the analytics event is a writing and can
be slow occasionally.
Would it be possible to somehow send the request to the client and still
keep processing the view? Like a send() + end() method or something
similar, without returning?
// Adding to a tasks queue is not an option as it'd be an overkill and an
overcomplicated solution, which wouldn't work for hundreds of thousands of
events per day.
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/c44e1af6-8c30-478e-9baf-d7fd8c93e0b5%40googlegroups.com
<https://groups.google.com/d/msgid/pylons-discuss/c44e1af6-8c30-478e-9baf-d7fd8c93e0b5%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Mikko Ohtamaa]

Post by Zsolt Ero
My problem is that while 99% of the views make read-only requests to the
database, and thus are very fast, the analytics event is a writing and can
be slow occasionally.

You can do post commit actions with SQLAlchemy and zope.sqlaclhemy.

1. Response gets served
2. Database transaction get committed for this request
3. Post transaction actions

Example:

https://stackoverflow.com/questions/15593516/how-to-run-a-script-after-pyramids-transaction-manager-has-returned

Please not that thread local `transaction.manager` might be discouraged
nowadays, but not sure what is the best practice to get a transaction
manager for your request.

Br,
Mikko

Post by Zsolt Ero
Would it be possible to somehow send the request to the client and still
keep processing the view? Like a send() + end() method or something
similar, without returning?
// Adding to a tasks queue is not an option as it'd be an overkill and an
overcomplicated solution, which wouldn't work for hundreds of thousands of
events per day.
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/c44e1af6-8c30-478e-9baf-d7fd8c93e0b5%40googlegroups.com
<https://groups.google.com/d/msgid/pylons-discuss/c44e1af6-8c30-478e-9baf-d7fd8c93e0b5%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Michael Merickel]

Post by Mikko Ohtamaa
https://stackoverflow.com/questions/15593516/how-to-run-a-script-after-pyramids-transaction-manager-has-returned
Please not that thread local `transaction.manager` might be discouraged
nowadays, but not sure what is the best practice to get a transaction
manager for your request.

request.tm and request.tm.get().addAfterCommitHook(...)