Discussion:
[pylons-discuss] Store class at process start, and use it to initiate inside function?
Lukasz Szybalski
2018-10-06 22:00:18 UTC
Permalink
Hello,
I have an enterprise system that is creating a class but it takes a long
time to initiate. About 2 sec, 90K _setitem from pickle. Nothing to
profile, since OS cashes the file its as fast as it gets.

I'm trying to find a way in pyramid where I can:

#store below at start, of the process, let it initiate,
#then somehow make it read only,
#so that a process can use it later and modify its own copy

from enterprise import Contract
my= Contract()


#rest of the program
my2 = copy my (or copy on write, similar how qcow format works)
my2.find_contract('abc')
my2.add_name('Lucas')
return (my2.stuff)

I don't seem to be finding the right terminology, or technique to do this,
and where to place it in pyramid?

Thanks
Lucas
--
http://lucasmanual.com/ <http://lucasmanual.com/blog/>
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Michael Merickel
2018-10-07 05:59:18 UTC
Permalink
This sounds like an application-global object. These are typically stored
on the registry at config-time. For example, in your main you could set
config.registry.foo = contract. The registry is available as
request.registry and subsequently anything you add to it. You can see lots
of examples of this in pyramid addons and things like the dbsession_factory
in the alchemy cookiecutter.
Post by Lukasz Szybalski
Hello,
I have an enterprise system that is creating a class but it takes a long
time to initiate. About 2 sec, 90K _setitem from pickle. Nothing to
profile, since OS cashes the file its as fast as it gets.
#store below at start, of the process, let it initiate,
#then somehow make it read only,
#so that a process can use it later and modify its own copy
from enterprise import Contract
my= Contract()
#rest of the program
my2 = copy my (or copy on write, similar how qcow format works)
my2.find_contract('abc')
my2.add_name('Lucas')
return (my2.stuff)
I don't seem to be finding the right terminology, or technique to do this,
and where to place it in pyramid?
Thanks
Lucas
--
http://lucasmanual.com/ <http://lucasmanual.com/blog/>
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com
<https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwEc87HDyELbK45Q5HVmQjDMT7M6U8M89e_OC7c0Mkz_mQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Lukasz Szybalski
2018-10-08 16:20:13 UTC
Permalink
Post by Michael Merickel
This sounds like an application-global object. These are typically stored
on the registry at config-time. For example, in your main you could set
config.registry.foo = contract. The registry is available as
request.registry and subsequently anything you add to it. You can see lots
of examples of this in pyramid addons and things like the dbsession_factory
in the alchemy cookiecutter.
Thank you.

I decided to add it at 1st run of the function:
try:
if not request.registry.mycontract:
request.registry.mycontract = Contract()
mycontract=copy.copy(request.registry.mycontract)
...
#rest of the code:
mycontract.add_user()
mycontract.update_terms()

*Sidenote:*
Is there a way to force this object to be "read only" or now allow
modification, to prevent somebody else in some other sections of the code
accidently modifies request.registry.mycontract?

**update run at 0.42755 sec now.

Thanks
Lucas
Post by Michael Merickel
Post by Lukasz Szybalski
Hello,
I have an enterprise system that is creating a class but it takes a long
time to initiate. About 2 sec, 90K _setitem from pickle. Nothing to
profile, since OS cashes the file its as fast as it gets.
#store below at start, of the process, let it initiate,
#then somehow make it read only,
#so that a process can use it later and modify its own copy
from enterprise import Contract
my= Contract()
#rest of the program
my2 = copy my (or copy on write, similar how qcow format works)
my2.find_contract('abc')
my2.add_name('Lucas')
return (my2.stuff)
I don't seem to be finding the right terminology, or technique to do
this, and where to place it in pyramid?
Thanks
Lucas
--
http://lucasmanual.com/ <http://lucasmanual.com/blog/>
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com
<https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/6590645a-7af6-480a-a866-ab1af78e0d42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Michael Merickel
2018-10-08 17:09:40 UTC
Permalink
If you are doing loading of data at "first run of the function" then you
have introduced a race condition in your app where unless you do
appropriate locking, two threads (most wsgi servers serve a request per
thread) may both consider themselves the first run and load the data. The
only way to do this without locks is to do things at config-time like I
suggested before. There are hacks that you can do because it's Python with
a GIL in which you can do locking in more lightweight ways in the "first
run of the function" case but I do not recommend relying on that behavior.

As far as defining an object as read-only, there is nothing specific to
Pyramid here and you'll have to find a satisfactory solution in the rest of
Python world.
Post by Lukasz Szybalski
Post by Michael Merickel
This sounds like an application-global object. These are typically stored
on the registry at config-time. For example, in your main you could set
config.registry.foo = contract. The registry is available as
request.registry and subsequently anything you add to it. You can see lots
of examples of this in pyramid addons and things like the dbsession_factory
in the alchemy cookiecutter.
Thank you.
request.registry.mycontract = Contract()
mycontract=copy.copy(request.registry.mycontract)
...
mycontract.add_user()
mycontract.update_terms()
*Sidenote:*
Is there a way to force this object to be "read only" or now allow
modification, to prevent somebody else in some other sections of the code
accidently modifies request.registry.mycontract?
**update run at 0.42755 sec now.
Thanks
Lucas
Post by Michael Merickel
Post by Lukasz Szybalski
Hello,
I have an enterprise system that is creating a class but it takes a long
time to initiate. About 2 sec, 90K _setitem from pickle. Nothing to
profile, since OS cashes the file its as fast as it gets.
#store below at start, of the process, let it initiate,
#then somehow make it read only,
#so that a process can use it later and modify its own copy
from enterprise import Contract
my= Contract()
#rest of the program
my2 = copy my (or copy on write, similar how qcow format works)
my2.find_contract('abc')
my2.add_name('Lucas')
return (my2.stuff)
I don't seem to be finding the right terminology, or technique to do
this, and where to place it in pyramid?
Thanks
Lucas
--
http://lucasmanual.com/ <http://lucasmanual.com/blog/>
--
You received this message because you are subscribed to the Google
Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com
<https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/6590645a-7af6-480a-a866-ab1af78e0d42%40googlegroups.com
<https://groups.google.com/d/msgid/pylons-discuss/6590645a-7af6-480a-a866-ab1af78e0d42%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwEJbH1qNNZ2-O8D2ie243nAAXj0knXhf8kXskLYfWYqqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Thierry Florac
2018-10-09 06:00:45 UTC
Permalink
And how do you handle such use case when working in a
multi-process/multi-hosts cluster configuration?
Post by Michael Merickel
If you are doing loading of data at "first run of the function" then you
have introduced a race condition in your app where unless you do
appropriate locking, two threads (most wsgi servers serve a request per
thread) may both consider themselves the first run and load the data. The
only way to do this without locks is to do things at config-time like I
suggested before. There are hacks that you can do because it's Python with
a GIL in which you can do locking in more lightweight ways in the "first
run of the function" case but I do not recommend relying on that behavior.
As far as defining an object as read-only, there is nothing specific to
Pyramid here and you'll have to find a satisfactory solution in the rest of
Python world.
Post by Lukasz Szybalski
Post by Michael Merickel
This sounds like an application-global object. These are typically
stored on the registry at config-time. For example, in your main you could
set config.registry.foo = contract. The registry is available as
request.registry and subsequently anything you add to it. You can see lots
of examples of this in pyramid addons and things like the dbsession_factory
in the alchemy cookiecutter.
Thank you.
request.registry.mycontract = Contract()
mycontract=copy.copy(request.registry.mycontract)
...
mycontract.add_user()
mycontract.update_terms()
*Sidenote:*
Is there a way to force this object to be "read only" or now allow
modification, to prevent somebody else in some other sections of the code
accidently modifies request.registry.mycontract?
**update run at 0.42755 sec now.
Thanks
Lucas
Post by Michael Merickel
Post by Lukasz Szybalski
Hello,
I have an enterprise system that is creating a class but it takes a
long time to initiate. About 2 sec, 90K _setitem from pickle. Nothing to
profile, since OS cashes the file its as fast as it gets.
#store below at start, of the process, let it initiate,
#then somehow make it read only,
#so that a process can use it later and modify its own copy
from enterprise import Contract
my= Contract()
#rest of the program
my2 = copy my (or copy on write, similar how qcow format works)
my2.find_contract('abc')
my2.add_name('Lucas')
return (my2.stuff)
I don't seem to be finding the right terminology, or technique to do
this, and where to place it in pyramid?
Thanks
Lucas
--
http://lucasmanual.com/ <http://lucasmanual.com/blog/>
--
You received this message because you are subscribed to the Google
Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com
<https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/6590645a-7af6-480a-a866-ab1af78e0d42%40googlegroups.com
<https://groups.google.com/d/msgid/pylons-discuss/6590645a-7af6-480a-a866-ab1af78e0d42%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwEJbH1qNNZ2-O8D2ie243nAAXj0knXhf8kXskLYfWYqqA%40mail.gmail.com
<https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwEJbH1qNNZ2-O8D2ie243nAAXj0knXhf8kXskLYfWYqqA%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
http://www.imagesdusport.com -- http://pyams.readthedocs.io
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWDAG9-V%3DSibaxztAa%2B8nHdXcrCEqp4Sn7CMjX%3DiQ7gNLA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Bert JW Regeer
2018-10-09 16:05:33 UTC
Permalink
You run the same code at configure time and load it once per process. Even if you do this once at request time or once at configure time you have to do it once per process.
And how do you handle such use case when working in a multi-process/multi-hosts cluster configuration?
If you are doing loading of data at "first run of the function" then you have introduced a race condition in your app where unless you do appropriate locking, two threads (most wsgi servers serve a request per thread) may both consider themselves the first run and load the data. The only way to do this without locks is to do things at config-time like I suggested before. There are hacks that you can do because it's Python with a GIL in which you can do locking in more lightweight ways in the "first run of the function" case but I do not recommend relying on that behavior.
As far as defining an object as read-only, there is nothing specific to Pyramid here and you'll have to find a satisfactory solution in the rest of Python world.
This sounds like an application-global object. These are typically stored on the registry at config-time. For example, in your main you could set config.registry.foo = contract. The registry is available as request.registry and subsequently anything you add to it. You can see lots of examples of this in pyramid addons and things like the dbsession_factory in the alchemy cookiecutter.
Thank you.
request.registry.mycontract = Contract()
mycontract=copy.copy(request.registry.mycontract)
...
mycontract.add_user()
mycontract.update_terms()
Is there a way to force this object to be "read only" or now allow modification, to prevent somebody else in some other sections of the code accidently modifies request.registry.mycontract?
**update run at 0.42755 sec now.
Thanks
Lucas
Hello,
I have an enterprise system that is creating a class but it takes a long time to initiate. About 2 sec, 90K _setitem from pickle. Nothing to profile, since OS cashes the file its as fast as it gets.
#store below at start, of the process, let it initiate,
#then somehow make it read only,
#so that a process can use it later and modify its own copy
from enterprise import Contract
my= Contract()
#rest of the program
my2 = copy my (or copy on write, similar how qcow format works)
my2.find_contract('abc')
my2.add_name('Lucas')
return (my2.stuff)
I don't seem to be finding the right terminology, or technique to do this, and where to place it in pyramid?
Thanks
Lucas
--
http://lucasmanual.com/ <http://lucasmanual.com/blog/>
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com <https://groups.google.com/d/msgid/pylons-discuss/CAKkTUv3qgT%2BUk0-uvLB1owZEA3W%3D-7XA-wkiyZbteWHPAcO6vg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/6590645a-7af6-480a-a866-ab1af78e0d42%40googlegroups.com <https://groups.google.com/d/msgid/pylons-discuss/6590645a-7af6-480a-a866-ab1af78e0d42%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwEJbH1qNNZ2-O8D2ie243nAAXj0knXhf8kXskLYfWYqqA%40mail.gmail.com <https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwEJbH1qNNZ2-O8D2ie243nAAXj0knXhf8kXskLYfWYqqA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
http://www.imagesdusport.com <http://www.imagesdusport.com/> -- http://pyams.readthedocs.io <http://pyams.readthedocs.io/>
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWDAG9-V%3DSibaxztAa%2B8nHdXcrCEqp4Sn7CMjX%3DiQ7gNLA%40mail.gmail.com <https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWDAG9-V%3DSibaxztAa%2B8nHdXcrCEqp4Sn7CMjX%3DiQ7gNLA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/8CF2F822-A0C5-4541-87DE-E066D11431DA%400x58.com.
For more options, visit https://groups.google.com/d/optout.
Bert JW Regeer
2018-10-09 16:08:17 UTC
Permalink
I would disagree, heavily. You want to create your globals once, then fork. This way the memory used by said global can be shared between all of the processes. Instagram even added the ability to freeze items so that they don't go through the normal GC cycle and thus don't accidentally cause COW on those objects:

https://instagram-engineering.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf
And how do you handle such use case when working in a multi-process /
multi-hosts cluster configuration?
Fork first, then run the configure step. Forking before creating stateful
globals is considered best practice for multi-processing.
Tres.
--
===================================================================
Palladion Software "Excellence by Design" http://palladion.com
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/ppij6f%24ol3%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/BBEECD76-6FA7-439D-B33E-0634F5C15E76%400x58.com.
For more options, visit https://groups.google.com/d/optout.
Jonathan Vanasco
2018-10-09 16:19:24 UTC
Permalink
Post by Bert JW Regeer
I would disagree, heavily. You want to create your globals once, then
fork. This way the memory used by said global can be shared between all of
the processes.
+1
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/45cb52dd-a7f3-41d1-8837-cc2bfc225d87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Lukasz Szybalski
2018-11-28 22:40:37 UTC
Permalink
Post by Bert JW Regeer
I would disagree, heavily. You want to create your globals once, then
fork. This way the memory used by said global can be shared between all of
the processes. Instagram even added the ability to freeze items so that
they don't go through the normal GC cycle and thus don't accidentally cause
https://instagram-engineering.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf
And how do you handle such use case when working in a multi-process /
multi-hosts cluster configuration?
Fork first, then run the configure step. Forking before creating
stateful
globals is considered best practice for multi-processing.
Hello,
How would one handle the following?
We have added the initation of the class and establish connection at config
time in
__init__.py

config.registry.MY = MYContract()

This definatelly works, but now we are getting into issues where we get
connection reset by peer. I guess in the other scenerio we connected every
time, so every time we established a new connection. Now we are re-using
the connection which causes below:
How can I try/except this at the __init__.py level in the config,...or what
do I do in views.py to except and redo "config.registry.MY = MYContract()"
to fix the connection issue.



[Thu Nov 01 09:50:28.215262 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033]
[Thu Nov 01 09:50:28.215271 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] Traceback (most recent call
last):
[Thu Nov 01 09:50:28.215277 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] File
"zzzzzzzzzzz/raven/utils/serializer/manager.py", line 76, in transform
[Thu Nov 01 09:50:28.215283 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] return repr(value)
[Thu Nov 01 09:50:28.215295 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] TypeError: __repr__ returned
non-string (type bytes)
[Thu Nov 01 09:50:28.215326 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033]
[Thu Nov 01 09:50:28.244058 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] mod_wsgi (pid=26861): Exception
occurred processing WSGI script '/zzzzzzzzzzzzz.wsgi'.
..........l, headers=headers)
[Thu Nov 01 09:50:28.247647 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] File
"zzzzzzzzzzzz/python3.5/site-packages/httplib2/__init__.py", line 1322, in
request
[Thu Nov 01 09:50:28.247653 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] (response, content) =
self._request(conn, authority, uri, request_uri, method, body, headers,
redirections, cachekey)
[Thu Nov 01 09:50:28.247663 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] File
"/zzzzzzzzz/python3.5/site-packages/httplib2/__init__.py", line 1072, in
_request
[Thu Nov 01 09:50:28.247669 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] (response, content) =
self._conn_request(conn, request_uri, method, body, headers)

[Thu Nov 01 09:50:28.247752 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] self.send(msg)
[Thu Nov 01 09:50:28.247761 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] File
"/usr/lib/python3.5/http/client.py", line 908, in send
[Thu Nov 01 09:50:28.247768 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] self.sock.sendall(data)
[Thu Nov 01 09:50:28.247792 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote 10.74.11.1:25033] ConnectionResetError: [Errno
104] Connection reset by peer

Thank you
Lucas
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/3831bc54-1b7c-4822-b73c-9702fb6cac79%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Lukasz Szybalski
2018-11-29 05:11:55 UTC
Permalink
Post by Michael Merickel
If you are doing loading of data at "first run of the function" then you
have introduced a race condition in your app where unless you do
appropriate locking, two threads (most wsgi servers serve a request per
thread) may both consider themselves the first run and load the data. The
only way to do this without locks is to do things at config-time like I
suggested before.
Hello,
How would one handle the following?
We have added the initation of the class and establish connection at config
time in
__init__.py

config.registry.MY = MYContract()

This definatelly works, but now we are getting into issues where we get
connection reset by peer. I guess in the other scenerio we connected every
time, so every time we established a new connection. Now we are re-using
the connection which causes below:
How can I try/except this at the __init__.py level in the config,...or what
do I do in views.py to except and redo "config.registry.MY = MYContract()"
to fix the connection issue.



[Thu Nov 01 09:50:28.215262 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a]
[Thu Nov 01 09:50:28.215271 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] Traceback (most recent call last):
[Thu Nov 01 09:50:28.215277 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] File
"zzzzzzzzzzz/raven/utils/serializer/manager.py", line 76, in transform
[Thu Nov 01 09:50:28.215283 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] return repr(value)
[Thu Nov 01 09:50:28.215295 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] TypeError: __repr__ returned non-string (type
bytes)
[Thu Nov 01 09:50:28.215326 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a]
[Thu Nov 01 09:50:28.244058 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] mod_wsgi (pid=26861): Exception occurred
processing WSGI script '/zzzzzzzzzzzzz.wsgi'.
..........l, headers=headers)
[Thu Nov 01 09:50:28.247647 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] File
"zzzzzzzzzzzz/python3.5/site-packages/httplib2/__init__.py", line 1322, in
request
[Thu Nov 01 09:50:28.247653 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] (response, content) = self._request(conn,
authority, uri, request_uri, method, body, headers, redirections, cachekey)
[Thu Nov 01 09:50:28.247663 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] File
"/zzzzzzzzz/python3.5/site-packages/httplib2/__init__.py", line 1072, in
_request
[Thu Nov 01 09:50:28.247669 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] (response, content) =
self._conn_request(conn, request_uri, method, body, headers)

[Thu Nov 01 09:50:28.247752 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] self.send(msg)
[Thu Nov 01 09:50:28.247761 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] File "/usr/lib/python3.5/http/client.py",
line 908, in send
[Thu Nov 01 09:50:28.247768 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] self.sock.sendall(data)
[Thu Nov 01 09:50:28.247792 2018] [wsgi:error] [pid 26861:tid
140425465390848] [remote a] ConnectionResetError: [Errno 104] Connection
reset by peer

Thank you
Lucas
--
Post by Michael Merickel
Post by Lukasz Szybalski
http://lucasmanual.com/ <http://lucasmanual.com/blog/>
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/095273c3-4267-4ca4-81ce-945bd86c6ba6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Mike Orr
2018-10-08 17:33:50 UTC
Permalink
Is there a way to force this object to be "read only" or now allow modification, to prevent somebody else in some other sections of the code accidently modifies request.registry.mycontract?
def __setattr__(self, attr, value):
if self.__locked:
raise AttributeError("read-only (locked) object")
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3DurS9B_huwNP0aRefUQY9Z4icKc0ixRNny918LCTE7LTJA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
b***@uniqueinsuranceco.com
2018-10-17 17:55:07 UTC
Permalink
Thanks guys for all your help.. This made our requests go from seconds to
fractions of a second using the method Michael described.
Post by Lukasz Szybalski
Hello,
I have an enterprise system that is creating a class but it takes a long
time to initiate. About 2 sec, 90K _setitem from pickle. Nothing to
profile, since OS cashes the file its as fast as it gets.
#store below at start, of the process, let it initiate,
#then somehow make it read only,
#so that a process can use it later and modify its own copy
from enterprise import Contract
my= Contract()
#rest of the program
my2 = copy my (or copy on write, similar how qcow format works)
my2.find_contract('abc')
my2.add_name('Lucas')
return (my2.stuff)
I don't seem to be finding the right terminology, or technique to do this,
and where to place it in pyramid?
Thanks
Lucas
--
http://lucasmanual.com/ <http://lucasmanual.com/blog/>
--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+***@googlegroups.com.
To post to this group, send email to pylons-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/19ed68da-953b-44d9-989a-4cd0d8ec43d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Continue reading on narkive:
Loading...