标签：python multiprocessing connection gevent monkey

我有一个应用程序,它使用grequests和multiprocessing.managers来组合IPC通信和HTTP上的异步RESTful通信.

似乎grequests在使用gevent.monkey的patch_all()方法时,会破坏multiprocessing.manager.SyncManager类及其派生类使用的multiprocessing.connection模块.

这显然不是一个孤立的问题,但会影响任何实现multiprocessing.connetion的用例,例如multiprocessing.pool.

深入研究gevent / monkey.py中的代码,我发现使用gevent.socket交换stdlib套接字模块是导致破坏的原因.
这可以在patch_socket()函数下的gevent / monkey.py中的第115行找到：

def patch_socket(dns=True, aggressive=True):
    """Replace the standard socket object with gevent's cooperative sockets.
    ...
    _socket.socket = socket.socket # This line breaks multiprocessing.connection!
    ...

我的问题是为什么这个swappage会破坏multiprocessing.connection,以及使用gevent.socket而不是stdlib的套接字模块会带来什么好处呢？也就是说,如果没有修补套接字模块,会产生什么性能损失？

追溯

Traceback (most recent call last):
  File "clientWithGeventMonkeyPatch.py", line 49, in <module>
    client = GetClient(host, port, authkey)
  File "clientWithGeventMonkeyPatch.py", line 39, in GetClient
    client.connect()
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 500, in connect
    conn = Client(self._address, authkey=self._authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 175, in Client
    answer_challenge(c, authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 414, in answer_challenge
    response = connection.recv_bytes(256)        # reject large message
IOError: [Errno 11] Resource temporarily unavailable

用于重现错误的代码

(在ubuntu服务器11.10,python2.7.3上,安装了gevent,greenlet和grequests)

manager.py

## manager.py
import multiprocessing
import multiprocessing.managers
import datetime


class LocalManager(multiprocessing.managers.SyncManager):
    def __init__(self, *args, **kwargs):
        multiprocessing.managers.SyncManager.__init__(self, *args, **kwargs)
        self.__type__ = 'LocalManager'

def GetManager(host, port, authkey):
    def getdatetime():
        return '{}'.format(datetime.datetime.now())

    LocalManager.register('getdatetime', callable = getdatetime)
    manager = LocalManager(address = (host, port), authkey = authkey)
    manager.start()

    return manager

if __name__ == '__main__':
    # define our manager connection parameters
    port = 55555
    host = 'localhost'
    authkey = 'auth1234'

    # start a manager
    man = GetManager(host, port, authkey)

    # wait for user input to shut down
    raw_input('return to shutdown')
    man.shutdown()

client.py

## client.py -- this one works
import time
import multiprocessing.managers

class RemoteClient(multiprocessing.managers.SyncManager):
    def __init__(self, *args, **kwargs):
        multiprocessing.managers.SyncManager.__init__(self, *args, **kwargs)
        self.__type__ = 'RemoteClient'

def GetClient(host, port, authkey):
    RemoteClient.register('getdatetime')
    client = RemoteClient(address = (host, port), authkey = authkey)
    client.connect()
    return client

if __name__ == '__main__':
    # define our client connection parameters
    port = 55555
    host = 'localhost'
    authkey = 'auth1234'

    # start a manager
    client = GetClient(host, port, authkey)
    print 'connected', client
    print 'client.getdatetime()', client.getdatetime()
    # wait a couple of seconds, then do it again
    time.sleep(2)
    print 'client.getdatetime()', client.getdatetime()

    # exit...

clientWithGeventMonkeyPatch.py

## clientWithGeventMonkeyPatch.py -- breaks, depending on patch_all() parameters        
import time
import multiprocessing.managers


# this part is copied from grequests
# bear in mind that it doesn't actually do anything in this module.
try:
    import gevent
    from gevent import monkey as curious_george
    from gevent.pool import Pool
except ImportError:
    raise RuntimeError('Gevent is required for grequests.')

# this line causes breakage of the multiprocessing.manager connection auth method:
# Monkey-patch. 
# patch_all() parameters with default values:  socket=True, dns=True, time=True, select=True, thread=True, os=True, ssl=True, aggressive=True

curious_george.patch_all(thread=False, select=False) # breaks
#~ curious_george.patch_all(thread=False, select=False, socket = False) # works!
#~ curious_george.patch_all(thread=False, select=False, socket = True, aggressive = True, dns = True) # same as (thread=False, select=False); breaks
#~ curious_george.patch_all(thread=False, select=False, socket = True, aggressive = True, dns = False) # breaks
#~ curious_george.patch_all(thread=False, select=False, socket = True, aggressive = False, dns = True) # breaks
#~ curious_george.patch_all(thread=False, select=False, socket = True, aggressive = False, dns = False) # breaks







class RemoteClient(multiprocessing.managers.SyncManager):
    def __init__(self, *args, **kwargs):
        multiprocessing.managers.SyncManager.__init__(self, *args, **kwargs)
        self.__type__ = 'RemoteClient'

def GetClient(host, port, authkey):
    RemoteClient.register('getdatetime')
    client = RemoteClient(address = (host, port), authkey = authkey)
    client.connect()
    return client

if __name__ == '__main__':
    # define our client connection parameters
    port = 55555
    host = 'localhost'
    authkey = 'auth1234'

    # start a manager
    client = GetClient(host, port, authkey)
    print 'connected', client
    print 'client.getdatetime()', client.getdatetime()
    # wait a couple of seconds, then do it again
    time.sleep(2)
    print 'client.getdatetime()', client.getdatetime()

    # exit...

解决方法:

如果不修补套接字模块,gevent不能阻止网络操作的能力将无法使用,因此首先使用gevent的大部分好处将无法使用.

gevent和多处理并不是真的可以很好地相互配合 – gevent主要假设你通过它进行网络连接,而不是绕过最高级别的Python套接字接口(多处理工作).

标签：python,multiprocessing,connection,gevent,monkey
来源： https://codeday.me/bug/20190625/1286053.html

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。