Celery is used for the following tasks:
Every time an upload is made, a piece of code is triggered that counts how many uploads have been done in that UTC 24 day. This populates the
UploadsCreatedmodel. By querying that you can see how much was uploaded by date rather than having to do heavy aggregates on the main
Every time a symbol URL is attempted to be retrieved but we find out it’s not in our storage backend, we need to write this down in the database. We do that by calling
store_missing_symbolwhich is synchronous. However, if any operational error happens, instead of giving up we send the same parameters to a wrapped function that runs as a Celery task. That task is wrapped with a patient decorator that retries repeatedly. Note that all of this is “guarded” by a memoization wrapper that tries to make sure we only all of this only happens once per 24 per symbol signature.
(NOT enabled as of Dec 2018) If a symbol is missing and its signature looks like we might be able to download it from Microsoft, we attempt to do all this work in a Celery task.
For more information about how to end-to-end test the Celery tasks see the End-to-end testing Celery section.
The ideal pattern is to write tests that test individual tasks directly
rather than testing the views/functions that depend on the task. However,
conftest.py there’s a default
task_always_eager=True which means the tasks run
immediately in the same process as the test.