swift container server 莫名stuck

openstack swift container server的进程经常莫名其妙进入 D Ds等状态

记录一下这个时候 storage.error的log 便于分析

一种情形是下面这种log

Jun  3 16:46:03 host207 container-server: ERROR __call__ error with PUT /sda5/100256/AUTH_3ed1d3f47b4e4794a59b7217bf136eee/atest1/data4M : #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 498, in __call__#012    res = method(req)#012  File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 2217, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 837, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 271, in PUT#012    broker.put_object(obj, timestamp, int(req.headers['x-size']),#012  File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line 219, in __getitem__#012    return self.environ[self._normalize(key)]#012KeyError: 'HTTP_X_SIZE' (txn: tx1b48300503fc454d82e07-0055498a24)
Jun  3 16:51:03 host207 container-server: ERROR __call__ error with PUT /sda5/100256/AUTH_3ed1d3f47b4e4794a59b7217bf136eee/atest1/data4M : #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 498, in __call__#012    res = method(req)#012  File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 2217, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 837, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 271, in PUT#012    broker.put_object(obj, timestamp, int(req.headers['x-size']),#012  File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line 219, in __getitem__#012    return self.environ[self._normalize(key)]#012KeyError: 'HTTP_X_SIZE' (txn: tx1b48300503fc454d82e07-0055498a24)
Jun  3 16:56:03 host207 container-server: ERROR __call__ error with PUT /sda5/100256/AUTH_3ed1d3f47b4e4794a59b7217bf136eee/atest1/data4M : #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 498, in __call__#012    res = method(req)#012  File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 2217, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 837, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 271, in PUT#012    broker.put_object(obj, timestamp, int(req.headers['x-size']),#012  File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line 219, in __getitem__#012    return self.environ[self._normalize(key)]#012KeyError: 'HTTP_X_SIZE' (txn: tx1b48300503fc454d82e07-0055498a24)
Jun  3 17:01:03 host207 container-server: ERROR __call__ error with PUT /sda5/100256/AUTH_3ed1d3f47b4e4794a59b7217bf136eee/atest1/data4M : #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 498, in __call__#012    res = method(req)#012  File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 2217, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 837, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 271, in PUT#012    broker.put_object(obj, timestamp, int(req.headers['x-size']),#012  File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line 219, in __getitem__#012    return self.environ[self._normalize(key)]#012KeyError: 'HTTP_X_SIZE' (txn: tx1b48300503fc454d82e07-0055498a24)
Jun  3 17:59:27 host207 object-replicator: Lockup detected.. killing live coros.
Jun  3 18:23:19 host207 container-server: SIGTERM received
Jun  3 18:23:19 host207 account-server: SIGTERM received
Jun  3 18:23:19 host207 object-server: SIGTERM received