gevent.monkey – Make the standard library cooperative¶Make the standard library cooperative.
The primary purpose of this module is to carefully patch, in place, portions of the standard library with gevent-friendly functions that behave in the same way as the original (at least as closely as possible).
The primary interface to this is the patch_all() function, which
performs all the available patches. It accepts arguments to limit the
patching to certain modules, but most programs should use the
default values as they receive the most wide-spread testing, and some monkey
patches have dependencies on others.
Patching should be done as early as possible in the lifecycle of the
program. For example, the main module (the one that tests against
__main__ or is otherwise the first imported) should begin with
this code, ideally before any other imports:
from gevent import monkey
monkey.patch_all()
A corollary of the above is that patching should be done on the main thread and should be done while the program is single-threaded.
Tip
Some frameworks, such as gunicorn, handle monkey-patching for you. Check their documentation to be sure.
Warning
Patching too late can lead to unreliable behaviour (for example, some modules may still use blocking sockets) or even errors.
Tip
Be sure to read the documentation for each patch function to check for known incompatibilities.
Sometimes it is helpful to know if objects have been monkey-patched, and in advanced cases even to have access to the original standard library functions. This module provides functions for that purpose.
Beginning in gevent 1.3, events are emitted during the monkey patching process.
These events are delivered first to gevent.events subscribers, and then
to setuptools entry points.
The following events are defined. They are listed in (roughly) the order
that a call to patch_all() will emit them.
Each event class documents the corresponding setuptools entry point name. The entry points will be called with a single argument, the same instance of the class that was sent to the subscribers.
You can subscribe to the events to monitor the monkey-patching process and
to manipulate it, for example by raising gevent.events.DoNotPatch.
You can also subscribe to the events to provide additional patching beyond what
gevent distributes, either for additional standard library modules, or
for third-party packages. The suggested time to do this patching is in
the subscriber for gevent.events.GeventDidPatchBuiltinModulesEvent.
For example, to automatically patch psycopg2 using psycogreen
when the call to patch_all() is made, you could write code like this:
# mypackage.py
def patch_psycopg(event):
from psycogreen.gevent import patch_psycopg
patch_psycopg()
In your setup.py you would register it like this:
from setuptools import setup
setup(
...
entry_points={
'gevent.plugins.monkey.did_patch_builtins': [
'psycopg2 = mypackage:patch_psycopg',
],
},
...
)
For more complex patching, gevent provides a helper method that you can call to replace attributes of modules with attributes of your own modules. This function also takes care of emitting the appropriate events.
Sometimes it is useful to run existing python scripts or modules that
were not built to be gevent aware under gevent. To do so, this module
can be run as the main module, passing the script and its arguments.
For details, see the main() function.
Changed in version 1.3b1: Added support for plugins and began emitting will/did patch events.
Retrieve the original object from a module.
If the object has not been patched, then that object will still be retrieved.
mod_name (str) – The name of the standard library module,
e.g., 'socket'. Can also be a sequence of standard library
modules giving alternate names to try, e.g., ('thread', '_thread');
the first importable module will supply all item_name items.
item_name – A string or sequence of strings naming the
attribute(s) on the module mod_name to return.
The original value if a string was given for
item_name or a sequence of original values if a
sequence was passed.
Check if a module has been replaced with a cooperative version.
mod_name (str) – The name of the standard library module,
e.g., 'socket'.
Check if an object in a module has been replaced with a cooperative version.
mod_name (str) – The name of the standard library module,
e.g., 'socket'.
item_name (str) – The name of the attribute in the module,
e.g., 'create_connection'.
gevent.monkey - monkey patch the standard modules to use gevent.
USAGE: python -m gevent.monkey [MONKEY OPTIONS] [--module] (script|module) [SCRIPT OPTIONS]
If no MONKEY OPTIONS are present, monkey patches all the modules as if by calling patch_all().
You can exclude a module with –no-<module>, e.g. –no-thread. You can
specify a module to patch with –<module>, e.g. –socket. In the latter
case only the modules specified on the command line will be patched.
The default behavior is to execute the script passed as argument. If you wish
to run a module instead, pass the --module argument before the module name.
Changed in version 1.3b1: The script argument can now be any argument that can be passed to runpy.run_path,
just like the interpreter itself does, for example a package directory containing __main__.py.
Previously it had to be the path to
a .py source file.
Changed in version 1.5: The --module option has been added.
MONKEY OPTIONS: --verbose --[no-]socket, --[no-]dns, --[no-]time, --[no-]select, --[no-]thread, --[no-]os, --[no-]ssl, --[no-]subprocess, --[no-]sys, --[no-]builtins, --[no-]signal, --[no-]queue, --[no-]contextvars
Do all of the default monkey patching (calls every other applicable function in this module).
A true value if patching all modules wasn’t cancelled, a false value if it was.
Changed in version 1.1: Issue a warning if this function is called multiple times
with different arguments. The second and subsequent calls will only add more
patches, they can never remove existing patches by setting an argument to False.
Changed in version 1.1: Issue a warning if this function is called with os=False
and signal=True. This will cause SIGCHLD handlers to not be called. This may
be an error in the future.
Changed in version 1.3a2: Event defaults to True.
Changed in version 1.3b1: Defined the return values.
Changed in version 1.3b1: Add **kwargs for the benefit of event subscribers. CAUTION: gevent may add
and interpret additional arguments in the future, so it is suggested to use prefixes
for kwarg values to be interpreted by plugins, for example, patch_all(mylib_futures=True).
Changed in version 1.3.5: Add queue, defaulting to True, for Python 3.7.
Changed in version 1.5: Remove the httplib argument. Previously, setting it raised a ValueError.
Changed in version 1.5a3: Add the contextvars argument.
Changed in version 1.5: Better handling of patching more than once.
Make the builtin __import__() function greenlet safe under Python 2.
Note
This does nothing under Python 3 as it is not necessary. Python 3 features improved import locks that are per-module, not global.
Replace DNS functions in socket with
cooperative versions.
This is only useful if patch_socket() has been called and is
done automatically by that method if requested.
Replace attributes in target_module with the attributes of the same name in source_module.
The source_module can provide some attributes to customize the process:
__implements__ is a list of attribute names to copy; if not present,
the items keyword argument is mandatory. __implements__ must only have
names from the standard library module in it.
_gevent_will_monkey_patch(target_module, items, warn, **kwargs)
_gevent_did_monkey_patch(target_module, items, warn, **kwargs)
These two functions in the source_module are called if they exist,
before and after copying attributes, respectively. The “will” function
may modify items. The value of warn is a function that should be called
with a single string argument to issue a warning to the user. If the “will”
function raises gevent.events.DoNotPatch, no patching will be done. These functions
are called before any event subscribers or plugins.
items (list) – A list of attribute names to replace. If
not given, this will be taken from the source_module __implements__
attribute.
A true value if patching was done, a false value if patching was canceled.
New in version 1.3b1.
Replace os.fork() with gevent.fork(), and, on POSIX,
os.waitpid() with gevent.os.waitpid() (if the
environment variable GEVENT_NOWAITPID is not defined). Does
nothing if fork is not available.
Caution
This method must be used with patch_signal() to have proper SIGCHLD
handling and thus correct results from waitpid.
patch_all() calls both by default.
Caution
For SIGCHLD handling to work correctly, the event loop must run.
The easiest way to help ensure this is to use patch_all().
On Python 3.7 and above, replace queue.SimpleQueue (implemented
in C) with its Python counterpart.
New in version 1.3.5.
Replace select.select() with gevent.select.select()
and select.poll() with gevent.select.poll (where available).
If aggressive is true (the default), also remove other
blocking functions from select .
select.devpoll() (Python 3.5+)
Make the signal.signal() function work with a monkey-patched os.
Caution
This method must be used with patch_os() to have proper SIGCHLD
handling. patch_all() calls both by default.
Caution
For proper SIGCHLD handling, you must yield to the event loop.
Using patch_all() is the easiest way to ensure this.
See also
Replace the standard socket object with gevent’s cooperative sockets.
dns (bool) – When true (the default), also patch address
resolution functions in socket. See Name Resolution (DNS) for details.
Replace ssl.SSLSocket object and socket wrapping functions in
ssl with cooperative versions.
This is only useful if patch_socket() has been called.
Replace subprocess.call(), subprocess.check_call(),
subprocess.check_output() and subprocess.Popen with
cooperative versions.
Note
On Windows under Python 3, the API support may not completely match the standard library.
Patch sys.std[in,out,err] to use a cooperative IO via a threadpool.
This is relatively dangerous and can have unintended consequences
such as hanging the process or misinterpreting control keys
when input() and raw_input() are used. patch_all()
does not call this function by default.
This method does nothing on Python 3. The Python 3 interpreter wants to flush the TextIOWrapper objects that make up stderr/stdout at shutdown time, but using a threadpool at that time leads to a hang.
Replace the standard thread module to make it greenlet-based.
threading (bool) – When True (the default),
also patch threading.
_threading_local (bool) – When True (the default),
also patch _threading_local.local.
logging (bool) – When True (the default), also patch locks taken if the logging module has been configured.
existing_locks (bool) – When True (the default), and the
process is still single threaded, make sure that any
threading.RLock (and, under Python 3, importlib._bootstrap._ModuleLock)
instances that are currently locked can be properly unlocked. Important: This is a
best-effort attempt and, on certain implementations, may not detect all
locks. It is important to monkey-patch extremely early in the startup process.
Setting this to False is not recommended, especially on Python 2.
Caution
Monkey-patching thread and using
multiprocessing.Queue or
concurrent.futures.ProcessPoolExecutor (which uses a
Queue) will hang the process.
Monkey-patching with this function and using sub-interpreters (and advanced C-level API) and threads may be unstable on certain platforms.
Changed in version 1.1b1: Add logging and existing_locks params.
Changed in version 1.3a2: Event defaults to True.
Replace time.sleep() with gevent.sleep().
Next page: gevent.os – Low-level operating system functions from os