Tensorflow/Keras with django not working correctly with celery

问题

We are building a script for face recognition, mainly with tensorflow for basic recognition functions, from videos.

When we try the soft directly with a python test-reco.py (which take a video path as parameter) it works perfectly.

Now we are trying to integrate it through our website, within a celery task.

Here is the main code:

def extract_labels(self, path_to_video):
    if not os.path.exists(path_to_video):
        print("NO VIDEO!")
        return None
    video = VideoFileClip(path_to_video)
    n_frames = int(video.fps * video.duration)

    out = []
    for i, frame in enumerate(video.iter_frames()):
        if self.verbose > 0:
            print(
                'processing frame:',
                str(i).zfill(len(str(n_frames))),
                '/',
                n_frames
            )

        try:
            rect = face_detector(frame[::2, ::2], 0)[0]
            y0, x0, y1, x1 = np.array([rect.left(), rect.top(), rect.right(), rect.bottom()])*2
            bbox = frame[x0:x1, y0:y1]
            bbox = resize(bbox, [128, 128])
            bbox = rgb2gray(bbox)
            bbox = equalize_hist(bbox)
            y_hat = self.model.predict(bbox[None, :, :, None], verbose=1, batch_size=1)[0]
            # y_hat = np.ones(7)
            out.append(y_hat)
        except IndexError as e:
            print(out)
            print(e)

We need a try catch because sometimes there aren't any face present in the first frames.

But then we have this line: y_hat = self.model.predict(bbox[None, :, :, None], verbose=1, batch_size=1)[0] blocking. Like an endless loop.

The bbox isn't empty.

The celery worker simply blocks on it and you can't exit the process (the warm / cold quit never occurs)

Is there something specific to do with tensorflow with Celery?

回答1:

I had a very similar setup and problem. In my case it helped to simply shift all the imports that referenced Keras stuff into a dedicated initializer function, leading to a setup like this:

from celery import Celery
from celery.signals import worker_process_init

CELERY = ...

@worker_process_init.connect()
def init_worker_process(**kwargs):

    // Load all Keras related imports here
    import ...


@CELERY.task()
def long_running_task(*args, **kwargs):

    // Actual calculation task
    ...

回答2:

tf.Session (Tensorflow session) is not fork-safe. Celery won't work if a package is not fork-safe.

I guess self.model.predict will call tf.Session somewhere and it gets blocked.

来源：https://stackoverflow.com/questions/45017573/tensorflow-keras-with-django-not-working-correctly-with-celery