Celery workers unable to connect to redis on docker instances

匿名 (未验证) 提交于 2019-12-03 02:20:02

问题:

I have a dockerized setup running a Django app within which I use Celery tasks. Celery uses Redis as the broker.

Versioning:

  • Docker version 17.09.0-ce, build afdb6d4
  • docker-compose version 1.15.0, build e12f3b9
  • Django==1.9.6
  • django-celery-beat==1.0.1
  • celery==4.1.0
  • celery[redis]
  • redis==2.10.5

Problem:

My celery workers appear to be unable to connect to the redis container located at localhost:6379. I am able to telnet into the redis server on the specified port. I am able to verify redis-server is running on the container.

When I manually connect to the Celery docker instance and attempt to create a worker using the command celery -A backend worker -l info I get the notice:

[2017-11-13 18:07:50,937: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 99 connecting to localhost:6379. Cannot assign requested address.. Trying again in 4.00 seconds...

Notes:

I am able to telnet in to the redis container on port 6379. On the redis container, redis-server is running.

Is there anything else that I'm missing? I've gone pretty far down the rabbit hole, but feel like I'm missing something really simple.

DOCKER CONFIG FILES:

docker-compose.common.yml here
docker-compose.dev.yml here

回答1:

When you use docker-compose, you aren't going to be using localhost for inter-container communication, you would be using the compose-assigned hostname of the container. In this case, the hostname of your redis container is redis. The top level elements under services: are your default host names.

So for celery to connect to redis, you should try redis://redis:6379/0. Since the protocol and the service name are the same, I'll elaborate a little more: if you named your redis service "butter-pecan-redis" in your docker-compose, you would instead use redis://butter-pecan-redis:6379/0.

Also, docker-compose.dev.yml doesn't appear to have celery and redis on a common network, which might cause them not to be able to see each other. I believe they need to share at least one network in common to be able to resolve their respective host names.

Networking in docker-compose has an example in the first handful of paragraphs, with a docker-compose.yml to look at.



回答2:

You may need to add the link and depends_on sections to your docker compose file, and then reference the containers by their hostname.

Updated docker-compose.yml:

version: '2.1' services:     db:         image: postgres     memcached:         image: memcached     redis:         image: redis         ports:           - '6379:6379'     backend-base:         build:             context: .             dockerfile: backend/Dockerfile-base         image: "/backend:base"     backend:         build:             context: .             dockerfile: backend/Dockerfile         image: "/backend:${ENV:-local}"         command: ./wait-for-it.sh db:5432 -- gunicorn backend.wsgi:application -b 0.0.0.0:8000 -k gevent -w 3         ports:             - 8000         links:             - db             - redis             - memcached         depends_on:             - db             - redis             - memcached     celery:         image: "/backend:${ENV:-local}"         command: ./wait-for-it.sh db:5432 -- celery worker -E -B --loglevel=INFO --concurrency=1         environment:             C_FORCE_ROOT: "yes"         links:             - db             - redis             - memcached         depends_on:             - db             - redis             - memcached     frontend-base:         build:             context: .             dockerfile: frontend/Dockerfile-base             args:                 NPM_REGISTRY: http://.view.build                 PACKAGE_INSTALLER: yarn         image: "/frontend:base"         links:             - db             - redis             - memcached         depends_on:             - db             - redis             - memcached     frontend:         build:             context: .             dockerfile: frontend/Dockerfile         image: "/frontend:${ENV:-local}"         command: 'bash -c ''gulp'''         working_dir: /app/user         environment:             PORT: 3000         links:             - db             - redis             - memcached         depends_on:             - db             - redis             - memcached 

Then configure the urls to redis, postgres, memcached, etc. with:

  • redis://redis:6379/0
  • postgres://user:pass@db:5432/database


易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!