celery - Tasks that need to run in priority
In my website users can UPDATE they profile (manual) every time he want, or automatic once a day.
This task is being distributed with celery now.
But i have a "problem" :
Every day, in automatic update, a job put ALL users (+-6k users) on queue:
from celery import group from tasks import * import datetime from lastActivityDate.models import UserActivity today = datetime.datetime.today() one_day = datetime.timedelta(days=5) today -= one_day print datetime.datetime.today() user_list = UserActivity.objects.filter(last_activity_date__gte=today) g = group(update_user_profile.s(i.user.auth.username) for i in user_list) print datetime.datetime.today() print g(user_list.count()).get()
If someone try to do the manual update, they will enter on te queue and last forever to be executed.
Is there a way to set this manual task to run in a piority way? Or make a dedicated for each separated queue: manual and automatic?
Celery does not support task priority. (v3.0)
You may solve this problem by routing tasks.
Prepare default and priority_high Queue.
CELERY_DEFAULT_QUEUE = 'default' CELERY_QUEUES = ( Queue('default'), Queue('priority_high'), )
Run two daemon.
user@x:/$ celery worker -Q priority_high user@y:/$ celery worker -Q default,priority_high
And route task.