一.问题的表象

过程:创建完windowserver2012r2版本虚拟机，挂载cinder卷，写入一定的数据

问题:关闭虚拟机所在的宿主机，执行疏散操作，虚拟机疏散失败

二.问题日志

  • 迁移的目的服务器nova-compute的日志信息如下：
2018-11-28 10:12:01.120 7 ERROR nova.volume.cinder [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Delete attachment failed for attachment e6b83198-8286-4925-b1d8-1b09b0ae9f79. Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6) Code: 500: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Exception during message handling: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 7829, in detach_volume

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 976, in decorated_function

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 5436, in detach_volume

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 5389, in _detach_volume

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id, destroy_bdm=destroy_bdm)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 415, in detach

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id, destroy_bdm)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 412, in _do_locked_detach

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self._do_detach(*args, **_kwargs)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 400, in _do_detach

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     volume_api.attachment_delete(context, self['attachment_id'])

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 379, in wrapper

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 415, in wrapper

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 838, in attachment_delete

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     'code': getattr(ex, 'code', None)})

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 831, in attachment_delete

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/v3/attachments.py", line 39, in delete

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/base.py", line 339, in _delete

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 209, in delete

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'DELETE', **kwargs)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)

2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server
  • 迁移的目的服务器cinder-volume的日志信息如下：
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server [req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Exception during message handling: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server Traceback (most recent call last):

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4503, in attachment_delete

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     attachment_id)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/objects/base.py", line 314, in get_by_id

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/api.py", line 1777, in get_by_id

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return IMPL.get_by_id(context, model, id, *args, **kwargs)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 6980, in get_by_id

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _GET_METHODS[model](context, id, *args, **kwargs)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2050, in volume_attachment_get

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _attachment_get(context, attachment_id)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 1982, in _attachment_get

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     .options(joinedload('volume'))

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     ret = list(self[0:1])

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return list(res)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._execute_and_instances(context)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     close_with_result=True)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     **kw

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     conn = self.session.connection(**kw)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     execution_options=execution_options)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     conn = engine.contextual_connect(**kw)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._wrap_pool_connect(self.pool.connect, None),

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     e, dialect, self)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     util.raise_from_cause(newraise, exc_info)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     reraise(type(exception), exception, tb=exc_tb, cause=cause)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return fn()

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _ConnectionFairy._checkout(self)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     fairy = _ConnectionRecord.checkout(pool)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     rec = pool._do_get()

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._dec_overflow()

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     compat.reraise(exc_type, exc_value, exc_tb)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1183, in _do_get

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._create_connection()

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _ConnectionRecord(self)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self.__connect(first_connect_check=True)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     connection = pool._invoke_creator(self)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return dialect.connect(*cargs, **cparams)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self.dbapi.connect(*cargs, **cparams)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return Connection(*args, **kwargs)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 699, in __init__

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self.connect()

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 936, in connect

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._request_authentication()

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1156, in _request_authentication

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     auth_packet = self._read_packet()

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     packet_header = self._read_bytes(4)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1037, in _read_bytes

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)

2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server

日志分析:

1.我们模拟的是虚拟机的疏散过程，在疏散的过程中，我们查看到目的服务器的nova-compute.log和cinder-volume.log的日志存在错误

2.进一步分析得知，我们发现出现问题的原因在cinder-volume，对比虚拟机能够正常疏散时的log日志，我们发现正常疏散时，cinder-volume.log的疏散显示的日志如下：

2018-11-28 14:37:54.772 34 INFO cinder.volume.manager [req-e287af5e-c5c3-4dc3-9cba-ab003ad12704 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Terminate volume connection completed successfully.

2018-11-28 14:37:54.773 34 DEBUG cinder.volume.manager [req-e287af5e-c5c3-4dc3-9cba-ab003ad12704 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Deleting attachment 6e32750e-6134-45cc-942f-c1065826f28b. _do_attachment_delete /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py:4519

3.对比cinder-volume.log问题日志和正常日志，我们分析，在虚拟机的疏散过程，涉及卷的卸载过程，怀疑的cinder卷的卸载存在问题，对此，我模拟windows server2012的虚拟机挂载cinder卷，写大量数据，然后分离卷连接，模拟了大概50次左右，复现过一次cinder卷分离失败的错误，对比log进行分析对比，发现和上面的虚拟机疏散失败问题一致。

4.分析cinder卷的挂载流程，我们发现根据log提示和实际cinder卷的卸载流程存在中断卷连接的操作，也就是第三章节中1.4.6.2步骤。这里我们第三章节从cinder-volume的日志来看，函数在/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1037, in _read_bytes此处提示丢失数据库连接错误，导致了中断卷连接时报错。

三.卷的卸载流程



上述流程图显示，卷在detach()->get vol info by id->terminate connection(call)时报错。

代码分析：

我们看到函数在_read_bytes处抛出错误：

进入cinder-volume容器，查看/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py代码的此函数如下:

def _read_bytes(self, num_bytes):

        self._sock.settimeout(self._read_timeout)

        while True:

            try:

                data = self._rfile.read(num_bytes)

                break

            except (IOError, OSError) as e:

                if e.errno == errno.EINTR:

                    continue

                self._force_close()

                raise err.OperationalError(

                    CR.CR_SERVER_LOST,

                    "Lost connection to MySQL server during query (%s)" % (e,))

        #如果MySQL服务器没有根据协议所述发送足够的数据，就会抛出如下错误

        if len(data) < num_bytes:

            self._force_close()

            raise err.OperationalError(

                CR.CR_SERVER_LOST, "Lost connection to MySQL server during queryy

")

        return data

#当服务器正在关闭(但显然仍在运行，因为这是从服务器发送的数据)时，将在专门为MariaDB记录的情况下抛出

if packet_number == 0:

# MariaDB sends error packet with seqno==0 when shutdown

    raise err.OperationalError(

    CR.CR_SERVER_LOST,

    "Lost connection to MySQL server during query")

raise err.InternalError(

    "Packet sequence number wrong - got %d expected %d"

    % (packet_number, self._next_seq_id))

四.解决办法

MySQL数据库中，max_allowed_packet允许的server段接受的数据包最大为4M，有的时候大的插入和更新可能会被max_allowed_packet参数限制，导致失败，这里我们修改了max_allowed_packet参数。

[root@hci-test01 templates]# vi /usr/share/kolla-ansible/ansible/roles/mariadb/templates/galera.cnf.j2

[server]

#增加如下行

max_allowed_packet= '32M'

相关知识链接:

数据库知识: https://blog.csdn.net/hxpjava1/article/details/80522201

数据库问题分析：https://bugs.launchpad.net/tripleo/+bug/1691951

cinder的流程图：https://blog.csdn.net/inodes/article/details/24729883

Openstack平台虚拟机疏散失败提示(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')问题的更多相关文章

  1. 解决python pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

    解决python pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') 学习了:ht ...

  2. pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

    pymysql错误: pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') pymy ...

  3. openstack中数据库连接数太多--pymysql.err.OperationalError，1040, u'Too many connections'

    1.出现问题: openstack运行过程中出现如下问题: OperationalError: (pymysql.err.OperationalError) (1040, u'Too many con ...

  4. pymysql pymysql.err.OperationalError 1045 Access denied最简单解决办法

    我使用的是python3.6+pymysql+mysql8.0 在cmd命令行直接输入mysql回车出现:ERROR 1045 (28000): Access denied for user 'ODB ...

  5. 解决 pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' ([Errno 61] Conne

    pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' ([Errno 61] ...

  6. pymysql.err.OperationalError: (1054, "Unknown column 'aa' in 'field list'")(已解决）

    错误描述: 今天使用python连接mysql数据库进行数据添加时,出现报错"pymysql.err.OperationalError: (1054, "Unknown colum ...

  7. 虚拟机中MySQL连接问题：Lost connection to MySQL server at 'reading initial communication packet, system error: 0 以及 host is not allowed to connect mysql

    环境:在VirtualBox中安装了Ubuntu虚拟机,网络使用了NAT模式,开启了端口转发. 局域网内其他计算机访问虚拟机中的MySQL Server出现两个问题: Lost connection ...

  8. Navicat Lite 提示Connection to mysql server on 10065

    Navicat Lite 提示Connection to mysql server on 10065 验证过主要是防火墙问题 [root@014136251035 zhop]# vi /etc/sys ...

  9. MYSQL导入CSV格式文件数据执行提示错误（ERROR 1290）： The MySQL server is running with the --secure-file-priv option so it cannot execute this statement.

    MYSQL导入CSV格式文件数据执行提示错误(ERROR 1290): The MySQL server is running with the --secure-file-priv option s ...

  10. OpenStack平台虚拟机实例在线迁移失败问题

    一.在线迁移时提示如下的报错 二.原因分析 通过kolla-ansible部署queens版本时,因为OEM的机器设备的UUID记录的一致,导致迁移时识别的是自身机器的UUID,导致迁移失败 三.问题 ...

随机推荐

  1. Django实战项目-学习任务系统-用户注册

    接着上期代码框架,开发第2个功能,用户注册,在原有用户模型基础上,增加一个学生用户属性表,用来关联学生用户的各种属性值,这个属性表是参考网络小说里系统属性值设计的,方便直观了解用户的能力高低,等级以及 ...

  2. MySQL 事务隔离级别：社交恐惧症的四个阶段

    MySQL 事务隔离级别:社交恐惧症的四个阶段 在数据库的世界里,数据们也有社交问题!事务隔离级别就是控制它们互相看到对方的程度... 什么是事务隔离? 想象一下,数据库是一个繁忙的餐厅,每个事务都是 ...

  3. 自定义异常--java进阶day08

    1.自定义异常 2.自定义异常的格式 看你想要定义哪种异常,对应的继承哪种异常类 以我们之前写的代码举例,Exception类过于庞大,所有的异常子类都可以被它接收,这样就会导致无法精确捕获,所以我们 ...

  4. 【Java】（机考常用）类集

    类集框架(集合框架)是一个用来代表和操纵集合的统一架构.所有的类集框架都包含如下内容: 接口:是代表类集的抽象数据类型.之所以定义多个接口,是为了以不同的方式操作集合对象. 例如:Collection ...

  5. 学习Django【2】把视图和路由连接起来的是URLconfigs的配置文件

    在view.py视图中定义: def detail(request, question_id): return HttpResponse("You're looking at questio ...

  6. JSON Objects Framework（1）

    学习datasnap,json必须掌握.用自身的JSON,就必须熟悉JSON Objects Framework.其中tostring和value区别就是一个坑. The JSON objects f ...

  7. Vscode写Markdown解决图片使用问题

      最近使用Vscode+Markdown写博客,图片不好弄,想了一下办法,有需要的人可以参考,有更方便的方法欢迎提出!   首先为了解决图片粘贴问题,下载一个扩展,Markdown Paste,下载 ...

  8. nodejs集群

    nodejs集群 单个 Node.js 实例运行在单个线程中. 为了充分利用多核系统,有时需要启用一组 Node.js 进程去处理负载任务. 集群中的Master 现在让我们详细了解Master的职责 ...

  9. CH579M 以太网接收死机

    CH579M进行以太网连接服务器时,可以正常连接,收发数据也可以. 但是当接收的数据量到达512左右时,CH579M就会无响应,问题出在工程设置上. 正确的设置↓↓↓↓↓ 错误的设置↓↓↓↓↓

  10. Greenplum数据库索引解析

    以下是对greenplum数据库使用总结. 创建索引 CREATE INDEX i_test_tb_state_az ON test_tb(name_en) WHERE name_en = 'AZ'; ...