Airflow: TypeError can't pickle memoryview objects

apache-airflow1.9.0 + python3 + rabbitmq + librabbitmq2.0.0

相关配置如下:

broker_url = amqp://cord:123456@localhost:5672//

celery_result_backend = rpc://

结果运行的时候抛出如下异常:

TypeError can't pickle memoryview objects XXXX

原因分析:

airflow 1.9.0使用的是celery4.x, 而celery 4.x使用json序列化，而不是用pickle进行序列化。

因此需要将librabbitmq改为pyamqp

broker_url

Default: "amqp://"

Default broker URL. This must be a URL in the form of:

transport://userid:password@hostname:port/virtual_host

Only the scheme part (transport://) is required, the rest is optional, and defaults to the specific transports default values.

The transport part is the broker implementation to use, and the default is amqp, (useslibrabbitmq if installed or falls back to pyamqp). There are also other choices available, including; redis://, sqs://, and qpid://.

http://docs.celeryproject.org/en/latest/userguide/configuration.html#conf-database-result-backend

因此将broker_url配置协议修改为pyamqp即可:

broker_url = pyamqp://cord:123456@localhost:5672//

参考链接:

https://github.com/celery/celery/issues/4693