Openstack平台虚拟机疏散失败提示(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')问题
一.问题的表象
过程:创建完windowserver2012r2版本虚拟机，挂载cinder卷，写入一定的数据
问题:关闭虚拟机所在的宿主机，执行疏散操作，虚拟机疏散失败
二.问题日志
- 迁移的目的服务器nova-compute的日志信息如下：
2018-11-28 10:12:01.120 7 ERROR nova.volume.cinder [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Delete attachment failed for attachment e6b83198-8286-4925-b1d8-1b09b0ae9f79. Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6) Code: 500: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Exception during message handling: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 7829, in detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 976, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 5436, in detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server attachment_id=attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 5389, in _detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server attachment_id=attachment_id, destroy_bdm=destroy_bdm)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 415, in detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server attachment_id, destroy_bdm)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 412, in _do_locked_detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server self._do_detach(*args, **_kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 400, in _do_detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server volume_api.attachment_delete(context, self['attachment_id'])
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 379, in wrapper
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 415, in wrapper
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, attachment_id, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 838, in attachment_delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server 'code': getattr(ex, 'code', None)})
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 831, in attachment_delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/v3/attachments.py", line 39, in delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server return self._delete("/attachments/%s" % base.getid(attachment))
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/base.py", line 339, in _delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server resp, body = self.api.client.delete(url)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 209, in delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'DELETE', **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server
- 迁移的目的服务器cinder-volume的日志信息如下：
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server [req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Exception during message handling: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4503, in attachment_delete
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server attachment_id)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/objects/base.py", line 314, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/api.py", line 1777, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return IMPL.get_by_id(context, model, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 6980, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return _GET_METHODS[model](context, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2050, in volume_attachment_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return _attachment_get(context, attachment_id)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 1982, in _attachment_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server .options(joinedload('volume'))
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server ret = list(self[0:1])
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return list(res)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return self._execute_and_instances(context)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server close_with_result=True)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server **kw
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server conn = self.session.connection(**kw)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server execution_options=execution_options)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server conn = engine.contextual_connect(**kw)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server self._wrap_pool_connect(self.pool.connect, None),
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server e, dialect, self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return fn()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return _ConnectionFairy._checkout(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server fairy = _ConnectionRecord.checkout(pool)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server rec = pool._do_get()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server self._dec_overflow()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1183, in _do_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return self._create_connection()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return _ConnectionRecord(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server self.__connect(first_connect_check=True)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server connection = pool._invoke_creator(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return dialect.connect(*cargs, **cparams)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return self.dbapi.connect(*cargs, **cparams)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server return Connection(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 699, in __init__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server self.connect()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 936, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server self._request_authentication()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1156, in _request_authentication
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server auth_packet = self._read_packet()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server packet_header = self._read_bytes(4)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1037, in _read_bytes
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server
日志分析:
1.我们模拟的是虚拟机的疏散过程，在疏散的过程中，我们查看到目的服务器的nova-compute.log和cinder-volume.log的日志存在错误
2.进一步分析得知，我们发现出现问题的原因在cinder-volume，对比虚拟机能够正常疏散时的log日志，我们发现正常疏散时，cinder-volume.log的疏散显示的日志如下：
2018-11-28 14:37:54.772 34 INFO cinder.volume.manager [req-e287af5e-c5c3-4dc3-9cba-ab003ad12704 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Terminate volume connection completed successfully.
2018-11-28 14:37:54.773 34 DEBUG cinder.volume.manager [req-e287af5e-c5c3-4dc3-9cba-ab003ad12704 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Deleting attachment 6e32750e-6134-45cc-942f-c1065826f28b. _do_attachment_delete /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py:4519
3.对比cinder-volume.log问题日志和正常日志，我们分析，在虚拟机的疏散过程，涉及卷的卸载过程，怀疑的cinder卷的卸载存在问题，对此，我模拟windows server2012的虚拟机挂载cinder卷，写大量数据，然后分离卷连接，模拟了大概50次左右，复现过一次cinder卷分离失败的错误，对比log进行分析对比，发现和上面的虚拟机疏散失败问题一致。
4.分析cinder卷的挂载流程，我们发现根据log提示和实际cinder卷的卸载流程存在中断卷连接的操作，也就是第三章节中1.4.6.2步骤。这里我们第三章节从cinder-volume的日志来看，函数在/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1037, in _read_bytes此处提示丢失数据库连接错误，导致了中断卷连接时报错。
三.卷的卸载流程
上述流程图显示，卷在detach()->get vol info by id->terminate connection(call)时报错。
代码分析：
我们看到函数在_read_bytes处抛出错误：
进入cinder-volume容器，查看/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py代码的此函数如下:
def _read_bytes(self, num_bytes):
self._sock.settimeout(self._read_timeout)
while True:
try:
data = self._rfile.read(num_bytes)
break
except (IOError, OSError) as e:
if e.errno == errno.EINTR:
continue
self._force_close()
raise err.OperationalError(
CR.CR_SERVER_LOST,
"Lost connection to MySQL server during query (%s)" % (e,))
#如果MySQL服务器没有根据协议所述发送足够的数据，就会抛出如下错误
if len(data) < num_bytes:
self._force_close()
raise err.OperationalError(
CR.CR_SERVER_LOST, "Lost connection to MySQL server during queryy
")
return data
#当服务器正在关闭(但显然仍在运行，因为这是从服务器发送的数据)时，将在专门为MariaDB记录的情况下抛出
if packet_number == 0:
# MariaDB sends error packet with seqno==0 when shutdown
raise err.OperationalError(
CR.CR_SERVER_LOST,
"Lost connection to MySQL server during query")
raise err.InternalError(
"Packet sequence number wrong - got %d expected %d"
% (packet_number, self._next_seq_id))
四.解决办法
MySQL数据库中，max_allowed_packet允许的server段接受的数据包最大为4M，有的时候大的插入和更新可能会被max_allowed_packet参数限制，导致失败，这里我们修改了max_allowed_packet参数。
[root@hci-test01 templates]# vi /usr/share/kolla-ansible/ansible/roles/mariadb/templates/galera.cnf.j2
[server]
#增加如下行
max_allowed_packet= '32M'
相关知识链接:
数据库知识: https://blog.csdn.net/hxpjava1/article/details/80522201
数据库问题分析：https://bugs.launchpad.net/tripleo/+bug/1691951
cinder的流程图：https://blog.csdn.net/inodes/article/details/24729883
Openstack平台虚拟机疏散失败提示(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')问题的更多相关文章
- 解决python pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
解决python pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') 学习了:ht ...
- pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
pymysql错误: pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') pymy ...
- openstack中数据库连接数太多--pymysql.err.OperationalError，1040, u'Too many connections'
1.出现问题: openstack运行过程中出现如下问题: OperationalError: (pymysql.err.OperationalError) (1040, u'Too many con ...
- pymysql pymysql.err.OperationalError 1045 Access denied最简单解决办法
我使用的是python3.6+pymysql+mysql8.0 在cmd命令行直接输入mysql回车出现:ERROR 1045 (28000): Access denied for user 'ODB ...
- 解决 pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' ([Errno 61] Conne
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' ([Errno 61] ...
- pymysql.err.OperationalError: (1054, "Unknown column 'aa' in 'field list'")(已解决）
错误描述: 今天使用python连接mysql数据库进行数据添加时,出现报错"pymysql.err.OperationalError: (1054, "Unknown colum ...
- 虚拟机中MySQL连接问题：Lost connection to MySQL server at 'reading initial communication packet, system error: 0 以及 host is not allowed to connect mysql
环境:在VirtualBox中安装了Ubuntu虚拟机,网络使用了NAT模式,开启了端口转发. 局域网内其他计算机访问虚拟机中的MySQL Server出现两个问题: Lost connection ...
- Navicat Lite 提示Connection to mysql server on 10065
Navicat Lite 提示Connection to mysql server on 10065 验证过主要是防火墙问题 [root@014136251035 zhop]# vi /etc/sys ...
- MYSQL导入CSV格式文件数据执行提示错误（ERROR 1290）： The MySQL server is running with the --secure-file-priv option so it cannot execute this statement.
MYSQL导入CSV格式文件数据执行提示错误(ERROR 1290): The MySQL server is running with the --secure-file-priv option s ...
- OpenStack平台虚拟机实例在线迁移失败问题
一.在线迁移时提示如下的报错 二.原因分析 通过kolla-ansible部署queens版本时,因为OEM的机器设备的UUID记录的一致,导致迁移时识别的是自身机器的UUID,导致迁移失败 三.问题 ...
随机推荐
- Golang 入门 : 符文
字符串常用语表示一系列文本字符,而Go的符文(rune)则用于表示单个字符. 字符串字面量由双引号(")包围,但rune字面量由单引号(')包围. Go程序几乎可以使用地球上任何语言的任何字 ...
- Django实战项目-学习任务系统-查询列表分页显示
接着上期代码框架,6个主要功能基本实现,剩下的就是细节点的完善优化了. 接着优化查询列表分页显示功能,有很多菜单功能都有查询列表显示页面情况,如果数据量多,不分页显示的话,页面展示效果就不太好. 本次 ...
- 裸金属架构之服务器安装VMWare ESXI虚拟化平台详细流程
目录1. 裸金属架构(Vcent)2. Vcent安装流程2.1 远程控制端(个人电脑)安装VMware-viclien2.2 服务器部署安装VMWare ESXI2.3 VMware-viclien ...
- [源码系列：手写spring] IOC第十节：bean的初始化和销毁方法
内容介绍 在Spring框架中,你可以使用以下几种方法定义bean的初始化和销毁: 使用注解方式: @PostConstruct:在bean的方法上添加@PostConstruct注解,该方法将在be ...
- 【Java】各种代码块的执行顺序
静态代码块:用staitc声明,jvm加载类时执行,仅执行一次 构造代码块:类中直接用{}定义,每一次创建对象时执行. 执行顺序优先级:静态块,main(),构造块,构造方法. 构造函数 public ...
- 【Java】字符串常用操作
字符的常用技巧 char c; 字母的顺序:c - 'A'.c - 'a' 字母大小写转换:c - 'a' + 'A'.c - 'A' + 'a' 数字字符转换为数字:c - '0' String.S ...
- Visual Studio 2017 导出 ASP.NET Core 项目模版项目文件为空
问题重现 VS 2017 针对 ASP.NET Core 导出模版功能有问题 解决办法 visual-studio-2017-templates-and-the-missing-content 目前官 ...
- mybatis-plus.global-config.db-config.id-type=auto 和 @TableId(value = "id", type = IdType.ASSIGN_ID)哪个优先生效
对于id自动生成的方式,有注解和配置两种. 含义相同:不过设置自动增长的时候必须保证数据库中id是自增,assign_id和assign_uuid则不需要. yml配置: mybatis-plus: ...
- EFCore Study(3)——“一”对多关系的设定和插入、查找级联操作
一.建立文章.评论类 /// <summary> /// 文章 /// </summary> public class Artitle { public int Id { ge ...
- 通过IP计算分析归属地
在产品中可能存在不同客户端,请求同一个服务端接口的场景. 例如小程序和App或者浏览器中,如果需要对请求的归属地进行分析,前提是需要先获取请求所在的国家或城市,这种定位通常需要主动授权,而用户一般是不 ...