This document describes the current stable version of Celery (5.2). For development docs, go here.

Change history¶

This document contains change notes for bugfix & new features in the 4.4.x series, please see What’s new in Celery 4.4 (Cliffs) for an overview of what’s new in Celery 4.4.

4.4.7¶

release-date:: 2020-07-31 11.45 P.M UTC+6:00
release-by:: Asif Saif Uddin

Add task_received, task_rejected and task_unknown to signals module.
[ES backend] add 401 as safe for retry.
treat internal errors as failure.
Remove redis fanout caveats.
FIX: -A and –args should behave the same. (#6223)
Class-based tasks autoretry (#6233)
Preserve order of group results with Redis result backend (#6218)
Replace future with celery.five Fixes #6250, and use raise_with_context instead of reraise
Fix REMAP_SIGTERM=SIGQUIT not working
(Fixes#6258) MongoDB: fix for serialization issue (#6259)
Make use of ordered sets in Redis opt-in
Test, CI, Docker & style and minor doc improvements.

4.4.6¶

release-date:: 2020-06-24 2.40 P.M UTC+6:00
release-by:: Asif Saif Uddin

Remove autoscale force_scale methods (#6085).
Fix autoscale test
Pass ping destination to request
chord: merge init options with run options
Put back KeyValueStoreBackend.set method without state
Added –range-prefix option to celery multi (#6180)
Added as_list function to AsyncResult class (#6179)
Fix CassandraBackend error in threads or gevent pool (#6147)
Kombu 4.6.11

4.4.5¶

release-date:: 2020-06-08 12.15 P.M UTC+6:00
release-by:: Asif Saif Uddin

Add missing dependency on future (#6146).
ElasticSearch: Retry index if document was deleted between index
fix windows build
Customize the retry interval of chord_unlock tasks
fix multi tests in local

4.4.4¶

release-date:: 2020-06-03 11.00 A.M UTC+6:00
release-by:: Asif Saif Uddin

Fix autoretry_for with explicit retry (#6138).
Kombu 4.6.10
Use Django DB max age connection setting (fixes #4116).
Add retry on recoverable exception for the backend (#6122).
Fix random distribution of jitter for exponential backoff.
ElasticSearch: add setting to save meta as json.
fix #6136. celery 4.4.3 always trying create /var/run/celery directory.
Add task_internal_error signal (#6049).

4.4.3¶

release-date:: 2020-06-01 4.00 P.M UTC+6:00
release-by:: Asif Saif Uddin

Fix backend utf-8 encoding in s3 backend .
Kombu 4.6.9
Task class definitions can have retry attributes (#5869)
Upgraded pycurl to the latest version that supports wheel.
Add uptime to the stats inspect command
Fixing issue #6019: unable to use mysql SSL parameters when getting
Clean TraceBack to reduce memory leaks for exception task (#6024)
exceptions: NotRegistered: fix up language
Give up sending a worker-offline message if transport is not connected
Add Task to __all__ in celery.__init__.py
Ensure a single chain object in a chain does not raise MaximumRecursion
Fix autoscale when prefetch_multiplier is 1
Allow start_worker to function without ping task
Update celeryd.conf
Fix correctly handle configuring the serializer for always_eager mode.
Remove doubling of prefetch_count increase when prefetch_multiplier
Fix eager function not returning result after retries
return retry result if not throw and is_eager
Always requeue while worker lost regardless of the redelivered flag
Allow relative paths in the filesystem backend (#6070)
[Fixed Issue #6017]
Avoid race condition due to task duplication.
Exceptions must be old-style classes or derived from BaseException
Fix windows build (#6104)
Add encode to meta task in base.py (#5894)
Update time.py to solve the microsecond issues (#5199)
Change backend _ensure_not_eager error to warning
Add priority support for ‘celery.chord_unlock’ task (#5766)
Change eager retry behaviour
Avoid race condition in elasticsearch backend
backends base get_many pass READY_STATES arg
Add integration tests for Elasticsearch and fix _update
feat(backend): Adds cleanup to ArangoDB backend
remove jython check
fix filesystem backend cannot not be serialized by picked

4.4.0¶

release-date:: 2019-12-16 9.45 A.M UTC+6:00
release-by:: Asif Saif Uddin

This version is officially supported on CPython 2.7, 3.5, 3.6, 3.7 & 3.8 and is also supported on PyPy2 & PyPy3.
Kombu 4.6.7
Task class definitions can have retry attributes (#5869)

4.4.0rc5¶

release-date:: 2019-12-07 21.05 A.M UTC+6:00
release-by:: Asif Saif Uddin

Kombu 4.6.7
Events bootstep disabled if no events (#5807)
SQS - Reject on failure (#5843)
Add a concurrency model with ThreadPoolExecutor (#5099)
Add auto expiry for DynamoDB backend (#5805)
Store extending result in all backends (#5661)
Fix a race condition when publishing a very large chord header (#5850)
Improve docs and test matrix

4.4.0rc4¶

release-date:: 2019-11-11 00.45 A.M UTC+6:00
release-by:: Asif Saif Uddin

Kombu 4.6.6
Py-AMQP 2.5.2
Python 3.8
Numerious bug fixes
PyPy 7.2

4.4.0rc3¶

release-date:: 2019-08-14 23.00 P.M UTC+6:00
release-by:: Asif Saif Uddin

Kombu 4.6.4
Billiard 3.6.1
Py-AMQP 2.5.1
Avoid serializing datetime (#5606)
Fix: (group() | group()) not equals single group (#5574)
Revert “Broker connection uses the heartbeat setting from app config.
Additional file descriptor safety checks.
fixed call for null args (#5631)
Added generic path for cache backend.
Fix Nested group(chain(group)) fails (#5638)
Use self.run() when overriding __call__ (#5652)
Fix termination of asyncloop (#5671)
Fix migrate task to work with both v1 and v2 of the message protocol.
Updating task_routes config during runtime now have effect.

4.4.0rc2¶

release-date:: 2019-06-15 4:00 A.M UTC+6:00
release-by:: Asif Saif Uddin

Many bugs and regressions fixed.
Kombu 4.6.3

4.4.0rc1¶

release-date:: 2019-06-06 1:00 P.M UTC+6:00
release-by:: Asif Saif Uddin

Python 3.4 drop
Kombu 4.6.1
Replace deprecated PyMongo methods usage (#5443)
Pass task request when calling update_state (#5474)
Fix bug in remaining time calculation in case of DST time change (#5411)
Fix missing task name when requesting extended result (#5439)
Fix collections import issue on Python 2.7 (#5428)
handle AttributeError in base backend exception deserializer (#5435)
Make AsynPool’s proc_alive_timeout configurable (#5476)
AMQP Support for extended result (#5495)
Fix SQL Alchemy results backend to work with extended result (#5498)
Fix restoring of exceptions with required param (#5500)
Django: Re-raise exception if ImportError not caused by missing tasks module (#5211)
Django: fixed a regression putting DB connections in invalid state when CONN_MAX_AGE != 0 (#5515)
Fixed OSError leading to lost connection to broker (#4457)
Fixed an issue with inspect API unable get details of Request
Fix mogodb backend authentication (#5527)
Change column type for Extended Task Meta args/kwargs to LargeBinary
Handle http_auth in Elasticsearch backend results (#5545)
Fix task serializer being ignored with task_always_eager=True (#5549)
Fix task.replace to work in .apply() as well as `.apply_async() (#5540)
Fix sending of worker_process_init signal for solo worker (#5562)
Fix exception message upacking (#5565)
Add delay parameter function to beat_schedule (#5558)
Multiple documentation updates

4.3.0¶

release-date:: 2019-03-31 7:00 P.M UTC+3:00
release-by:: Omer Katz

Added support for broadcasting using a regular expression pattern or a glob pattern to multiple Pidboxes.

This allows you to inspect or ping multiple workers at once.

Contributed by Dmitry Malinovsky & Jason Held
Added support for PEP 420 namespace packages.

This allows you to load tasks from namespace packages.

Contributed by Colin Watson
Added acks_on_failure_or_timeout as a setting instead of a task only option.

This was missing from the original PR but now added for completeness.

Contributed by Omer Katz
Added the task_received signal.

Contributed by Omer Katz
Fixed a crash of our CLI that occurred for everyone using Python < 3.6.

The crash was introduced in acd6025 by using the ModuleNotFoundError exception which was introduced in Python 3.6.

Contributed by Omer Katz
Fixed a crash that occurred when using the Redis result backend while the result_expires is set to None.

Contributed by Toni Ruža & Omer Katz
Added support the DNS seedlist connection format for the MongoDB result backend.

This requires the dnspython package which will be installed by default when installing the dependencies for the MongoDB result backend.

Contributed by George Psarakis
Bump the minimum eventlet version to 0.24.1.

Contributed by George Psarakis
Replace the msgpack-python package with msgpack.

We’re no longer using the deprecated package. See our important notes for this release for further details on how to upgrade.

Contributed by Daniel Hahler

Allow scheduling error handlers which are not registered tasks in the current worker.

These kind of error handlers are now possible:

from celery import Signature
Signature(
  'bar', args=['foo'],
  link_error=Signature('msg.err', queue='msg')
).apply_async()

Additional fixes and enhancements to the SSL support of the Redis broker and result backend.

Contributed by Jeremy Cohen

Code Cleanups, Test Coverage & CI Improvements by:

Omer Katz

Florian Chardin

Documentation Fixes by:

Omer Katz

Samuel Huang

Amir Hossein Saeid Mehr

Dmytro Litvinov

4.3.0 RC2¶

release-date:: 2019-03-03 9:30 P.M UTC+2:00
release-by:: Omer Katz

Filesystem Backend: Added meaningful error messages for filesystem backend.

Contributed by Lars Rinn
New Result Backend: Added the ArangoDB backend.

Contributed by Dilip Vamsi Moturi
Django: Prepend current working directory instead of appending so that the project directory will have precedence over system modules as expected.

Contributed by Antonin Delpeuch
Bump minimum py-redis version to 3.2.0.

Due to multiple bugs in earlier versions of py-redis that were causing issues for Celery, we were forced to bump the minimum required version to 3.2.0.

Contributed by Omer Katz
Dependencies: Bump minimum required version of Kombu to 4.4

Contributed by Omer Katz

4.3.0 RC1¶

release-date:: 2019-02-20 5:00 PM IST
release-by:: Omer Katz

Canvas: celery.chain.apply() does not ignore keyword arguments anymore when applying the chain.

Contributed by Korijn van Golen
Result Set: Don’t attempt to cache results in a celery.result.ResultSet.

During a join, the results cache was populated using celery.result.ResultSet.get(), if one of the results contains an exception, joining unexpectedly failed.

The results cache is now removed.

Contributed by Derek Harland
Application: celery.Celery.autodiscover_tasks() now attempts to import the package itself when the related_name keyword argument is None.

Contributed by Alex Ioannidis
Windows Support: On Windows 10, stale PID files prevented celery beat to run. We now remove them when a SystemExit is raised.

Contributed by :github_user:`na387`
Task: Added the new task_acks_on_failure_or_timeout setting.

Acknowledging SQS messages on failure or timing out makes it impossible to use dead letter queues.

We introduce the new option acks_on_failure_or_timeout, to ensure we can totally fallback on native SQS message lifecycle, using redeliveries for retries (in case of slow processing or failure) and transitions to dead letter queue after defined number of times.

Contributed by Mario Kostelac
RabbitMQ Broker: Adjust HA headers to work on RabbitMQ 3.x.

This change also means we’re ending official support for RabbitMQ 2.x.

Contributed by Asif Saif Uddin
Command Line: Improve celery update error handling.

Contributed by Federico Bond
Canvas: Support chords with task_always_eager set to True.

Contributed by Axel Haustant
Result Backend: Optionally store task properties in result backend.

Setting the result_extended configuration option to True enables storing additional task properties in the result backend.

Contributed by John Arnold
Couchbase Result Backend: Allow the Couchbase result backend to automatically detect the serialization format.

Contributed by Douglas Rohde
New Result Backend: Added the Azure Block Blob Storage result backend.

The backend is implemented on top of the azure-storage library which uses Azure Blob Storage for a scalable low-cost PaaS backend.

The backend was load tested via a simple nginx/gunicorn/sanic app hosted on a DS4 virtual machine (4 vCores, 16 GB RAM) and was able to handle 600+ concurrent users at ~170 RPS.

The commit also contains a live end-to-end test to facilitate verification of the backend functionality. The test is activated by setting the AZUREBLOCKBLOB_URL environment variable to azureblockblob://{ConnectionString} where the value for ConnectionString can be found in the Access Keys pane of a Storage Account resources in the Azure Portal.

Contributed by Clemens Wolff
Task: celery.app.task.update_state() now accepts keyword arguments.

This allows passing extra fields to the result backend. These fields are unused by default but custom result backends can use them to determine how to store results.

Contributed by Christopher Dignam
Gracefully handle consumer kombu.exceptions.DecodeError.

When using the v2 protocol the worker no longer crashes when the consumer encounters an error while decoding a message.

Contributed by Steven Sklar
Deployment: Fix init.d service stop.

Contributed by Marcus McHale
Django: Drop support for Django < 1.11.

Contributed by Asif Saif Uddin
Django: Remove old djcelery loader.

Contributed by Asif Saif Uddin
Result Backend: celery.worker.request.Request now passes celery.app.task.Context to the backend’s store_result functions.

Since the class currently passes self to these functions, revoking a task resulted in corrupted task result data when django-celery-results was used.

Contributed by Kiyohiro Yamaguchi
Worker: Retry if the heartbeat connection dies.

Previously, we keep trying to write to the broken connection. This results in a memory leak because the event dispatcher will keep appending the message to the outbound buffer.

Contributed by Raf Geens
Celery Beat: Handle microseconds when scheduling.

Contributed by K Davis
Asynpool: Fixed deadlock when closing socket.

Upon attempting to close a socket, celery.concurrency.asynpool.AsynPool only removed the queue writer from the hub but did not remove the reader. This led to a deadlock on the file descriptor and eventually the worker stopped accepting new tasks.

We now close both the reader and the writer file descriptors in a single loop iteration which prevents the deadlock.

Contributed by Joshua Engelman
Celery Beat: Correctly consider timezone when calculating timestamp.

Contributed by :github_user:`yywing`
Celery Beat: celery.beat.Scheduler.schedules_equal() can now handle either arguments being a None value.

Contributed by :github_user:` ratson`
Documentation/Sphinx: Fixed Sphinx support for shared_task decorated functions.

Contributed by Jon Banafato
New Result Backend: Added the CosmosDB result backend.

This change adds a new results backend. The backend is implemented on top of the pydocumentdb library which uses Azure CosmosDB for a scalable, globally replicated, high-performance, low-latency and high-throughput PaaS backend.

Contributed by Clemens Wolff
Application: Added configuration options to allow separate multiple apps to run on a single RabbitMQ vhost.

The newly added event_exchange and control_exchange configuration options allow users to use separate Pidbox exchange and a separate events exchange.

This allow different Celery applications to run separately on the same vhost.

Contributed by Artem Vasilyev
Result Backend: Forget parent result metadata when forgetting a result.

Contributed by :github_user:`tothegump`
Task Store task arguments inside celery.exceptions.MaxRetriesExceededError.

Contributed by Anthony Ruhier
Result Backend: Added the result_accept_content setting.

This feature allows to configure different accepted content for the result backend.

A special serializer (auth) is used for signed messaging, however the result_serializer remains in json, because we don’t want encrypted content in our result backend.

To accept unsigned content from the result backend, we introduced this new configuration option to specify the accepted content from the backend.

Contributed by Benjamin Pereto
Canvas: Fixed error callback processing for class based tasks.

Contributed by Victor Mireyev
New Result Backend: Added the S3 result backend.

Contributed by Florian Chardin
Task: Added support for Cythonized Celery tasks.

Contributed by Andrey Skabelin
Riak Result Backend: Warn Riak backend users for possible Python 3.7 incompatibilities.

Contributed by George Psarakis
Python Runtime: Added Python 3.7 support.

Contributed by Omer Katz & Asif Saif Uddin
Auth Serializer: Revamped the auth serializer.

The auth serializer received a complete overhaul. It was previously horribly broken.

We now depend on cryptography instead of pyOpenSSL for this serializer.

Contributed by Benjamin Pereto
Command Line: celery report now reports kernel version along with other platform details.

Contributed by Omer Katz

Canvas: Fixed chords with chains which include sub chords in a group.

Celery now correctly executes the last task in these types of canvases:

c = chord(
  group([
      chain(
          dummy.si(),
          chord(
              group([dummy.si(), dummy.si()]),
              dummy.si(),
          ),
      ),
      chain(
          dummy.si(),
          chord(
              group([dummy.si(), dummy.si()]),
              dummy.si(),
          ),
      ),
  ]),
  dummy.si()
)

c.delay().get()

Contributed by Maximilien Cuony

Canvas: Complex canvases with error callbacks no longer raises an AttributeError.

Very complex canvases such as this no longer raise an AttributeError which prevents constructing them.

We do not know why this bug occurs yet.

Contributed by Manuel Vázquez Acosta
Command Line: Added proper error messages in cases where app cannot be loaded.

Previously, celery crashed with an exception.

We now print a proper error message.

Contributed by Omer Katz
Task: Added the task_default_priority setting.

You can now set the default priority of a task using the task_default_priority setting. The setting’s value will be used if no priority is provided for a specific task.

Contributed by :github_user:`madprogrammer`
Dependencies: Bump minimum required version of Kombu to 4.3 and Billiard to 3.6.

Contributed by Asif Saif Uddin
Result Backend: Fix memory leak.

We reintroduced weak references to bound methods for AsyncResult callback promises, after adding full weakref support for Python 2 in vine. More details can be found in celery/celery#4839.

Contributed by George Psarakis and :github_user:`monsterxx03`.
Task Execution: Fixed roundtrip serialization for eager tasks.

When doing the roundtrip serialization for eager tasks, the task serializer will always be JSON unless the serializer argument is present in the call to celery.app.task.Task.apply_async(). If the serializer argument is present but is ‘pickle’, an exception will be raised as pickle-serialized objects cannot be deserialized without specifying to serialization.loads what content types should be accepted. The Producer’s serializer seems to be set to None, causing the default to JSON serialization.

We now continue to use (in order) the serializer argument to celery.app.task.Task.apply_async(), if present, or the Producer’s serializer if not None. If the Producer’s serializer is None, it will use the Celery app’s task_serializer configuration entry as the serializer.

Contributed by Brett Jackson
Redis Result Backend: The celery.backends.redis.ResultConsumer class no longer assumes celery.backends.redis.ResultConsumer.start() to be called before celery.backends.redis.ResultConsumer.drain_events().

This fixes a race condition when using the Gevent workers pool.

Contributed by Noam Kush

Task: Added the task_inherit_parent_priority setting.

Setting the task_inherit_parent_priority configuration option to True will make Celery tasks inherit the priority of the previous task linked to it.

Examples:

c = celery.chain(
  add.s(2), # priority=None
  add.s(3).set(priority=5), # priority=5
  add.s(4), # priority=5
  add.s(5).set(priority=3), # priority=3
  add.s(6), # priority=3
)

@app.task(bind=True)
def child_task(self):
  pass

@app.task(bind=True)
def parent_task(self):
  child_task.delay()

# child_task will also have priority=5
parent_task.apply_async(args=[], priority=5)

Contributed by :github_user:`madprogrammer`

Canvas: Added the result_chord_join_timeout setting.

Previously, celery.result.GroupResult.join() had a fixed timeout of 3 seconds.

The result_chord_join_timeout setting now allows you to change it.

Contributed by :github_user:`srafehi`