Configuration

Symbols Service configuration

The Symbols Service covers uploading and downloading symbols.

Gunicorn configuration:

GUNICORN_TIMEOUT
Parser

str

Default

“300”

Required

No

Specifies the timeout value.

https://docs.gunicorn.org/en/stable/settings.html#timeout

Used in bin/run_web.sh.

GUNICORN_WORKERS
Parser

str

Default

“1”

Required

No

Specifies the number of gunicorn workers.

You should set it to (2 x $num_cores) + 1.

https://docs.gunicorn.org/en/stable/settings.html#workers

http://docs.gunicorn.org/en/stable/design.html#how-many-workers

Used in bin/run_web.sh.

Webapp configuration:

Configuration

Configuration summary:

Setting

Parser

Required?

LOCAL_DEV_ENV

bool

TEST_ENV

bool

TOOL_ENV

bool

SENTRY_DSN

str

LOGGING_DEFAULT_LEVEL

str

STATSD_HOST

str

STATSD_PORT

int

STATSD_NAMESPACE

str

FRONTEND_ROOT

str

STATIC_ROOT

str

WHITENOISE_MAX_AGE

int

OIDC_RP_CLIENT_ID

str

Yes

OIDC_RP_CLIENT_SECRET

str

Yes

OIDC_OP_AUTHORIZATION_ENDPOINT

str

Yes

OIDC_OP_TOKEN_ENDPOINT

str

Yes

OIDC_OP_USER_ENDPOINT

str

Yes

OIDC_VERIFY_SSL

bool

ENABLE_TOKENS_AUTHENTICATION

bool

TOKENS_DEFAULT_EXPIRATION_DAYS

int

REDIS_URL

str

Yes

REDIS_SOCKET_CONNECT_TIMEOUT

int

REDIS_SOCKET_TIMEOUT

int

AWS_ACCESS_KEY_ID

str

AWS_SECRET_ACCESS_KEY

str

AWS_DEFAULT_REGION

str

MEMOIZE_LOG_MISSING_SYMBOLS_SECONDS

int

MEMOIZE_KEY_EXISTING_SIZE_SECONDS

int

UPLOAD_FILE_UPLOAD_MAX_WORKERS

int

ENABLE_STORE_MISSING_SYMBOLS

bool

UPLOAD_TEMPDIR_PREFIX

str

ALLOW_UPLOAD_BY_ANY_DOMAIN

bool

SYNCHRONOUS_UPLOAD_FILE_UPLOAD

bool

SECRET_KEY

str

Yes

DEBUG

bool

ALLOWED_HOSTS

ListOf(str)

DATABASE_URL

dj_database_url.parse

Yes

CONN_MAX_AGE

int

SESSION_COOKIE_AGE

int

SYMBOL_URLS

ListOf(str)

Yes

UPLOAD_DEFAULT_URL

str

Yes

UPLOAD_TRY_SYMBOLS_URL

str

Yes

UPLOAD_URL_EXCEPTIONS

dict_parser

SYMBOL_FILE_PREFIX

str

COMPRESS_EXTENSIONS

ListOf(str)

MIME_OVERRIDES

dict_parser

SYMBOLS_GET_TIMEOUT

int

DISALLOWED_SYMBOLS_SNIPPETS

ListOf(str)

SYMBOLDOWNLOAD_EXISTS_TTL_SECONDS

int

UPLOAD_REATTEMPT_LIMIT_SECONDS

int

ALLOW_UPLOAD_BY_DOWNLOAD_DOMAINS

ListOf(str)

DOWNLOAD_FILE_EXTENSIONS_ALLOWED

ListOf(str)

Configuration options:

LOCAL_DEV_ENV
Parser

bool

Default

“false”

Required

No

Set to true if you’re running in a local dev environment; false otherwise

TEST_ENV
Parser

bool

Default

“false”

Required

No

Set to true if you’re running tests; false otherwise.

TOOL_ENV
Parser

bool

Default

“false”

Required

No

Set to true if you’re running manage.py in a tool context. For example, for collectstatic.

SENTRY_DSN
Parser

str

Default

“”

Required

No

Sentry DSN or empty string

LOGGING_DEFAULT_LEVEL
Parser

str

Default

“INFO”

Required

No

Default level for logging. Should be one of INFO, DEBUG, WARNING, ERROR.

STATSD_HOST
Parser

str

Default

“localhost”

Required

No

statsd host.

STATSD_PORT
Parser

int

Default

“8125”

Required

No

statsd port.

STATSD_NAMESPACE
Parser

str

Default

“”

Required

No

Namespace for statsd keys.

FRONTEND_ROOT
Parser

str

Default

“os.path.join(BASE_DIR, “frontend/build/”)”

Required

No

Root directory for frontend files like index.html

STATIC_ROOT
Parser

str

Default

“os.path.join(BASE_DIR, “frontend/build/static/”)”

Required

No

Root directory for static files.

WHITENOISE_MAX_AGE
Parser

int

Default

“str(60 * 60)”

Required

No

Maximum age for cache control for whitenoise served static files.

OIDC_RP_CLIENT_ID
Parser

str

Required

Yes

OIDC RP client id.

OIDC_RP_CLIENT_SECRET
Parser

str

Required

Yes

OIDC RP client secret.

OIDC_OP_AUTHORIZATION_ENDPOINT
Parser

str

Required

Yes

OIDC OP authorization endpoint.

OIDC_OP_TOKEN_ENDPOINT
Parser

str

Required

Yes

OIDC OP token endpoint.

OIDC_OP_USER_ENDPOINT
Parser

str

Required

Yes

OIDC OP user endpoint.

OIDC_VERIFY_SSL
Parser

bool

Default

“true”

Required

No

Whether or not to verify SSL. This should always be True unless in a local dev environment.

ENABLE_TOKENS_AUTHENTICATION
Parser

bool

Default

“true”

Required

No

True if API token authentication is enabled; false otherwise.

TOKENS_DEFAULT_EXPIRATION_DAYS
Parser

int

Default

“365”

Required

No

Default expiration in days for tokens.

REDIS_URL
Parser

str

Required

Yes

URL for Redis.

REDIS_SOCKET_CONNECT_TIMEOUT
Parser

int

Default

“1”

Required

No

Connection timeout to use for Redis connections.

REDIS_SOCKET_TIMEOUT
Parser

int

Default

“2”

Required

No

Connection timeout for socket operations.

AWS_ACCESS_KEY_ID
Parser

str

Default

“”

Required

No

AWS access key id.

AWS_SECRET_ACCESS_KEY
Parser

str

Default

“”

Required

No

AWS secret access key.

AWS_DEFAULT_REGION
Parser

str

Default

“”

Required

No

AWS default region.

MEMOIZE_LOG_MISSING_SYMBOLS_SECONDS
Parser

int

Default

“str(60 * 60 * 24)”

Required

No

When a symbol is tried to be downloaded, and it turns out the symbol does not exist in S3, we write this down so all missing symbols can be post-processed after.

But we only need to write it down once per symbol. There’s a memoizing guard and this defines how long it should cache that it memoized.

MEMOIZE_KEY_EXISTING_SIZE_SECONDS
Parser

int

Default

“str(60 * 60 * 24)”

Required

No

When we ask S3 for the size (if it exists) of a symbol already in S3 this can be cached. This value determines how long we do that caching.

UPLOAD_FILE_UPLOAD_MAX_WORKERS
Parser

int

Default

“0”

Required

No

When we upload a .zip file, we iterate over the content and for each file within (that isn’t immediately ignorable) we kick off a function which figures out what (and how) to process the file. That function involves doing a S3 GET (technically ListObjectsV2), (possible) gzipping the payload and (possibly) a S3 PUT. All of these function calls get put in a concurrent.futures.ThreadPoolExecutor pool. This setting is about how many of these to start, max.

ENABLE_STORE_MISSING_SYMBOLS
Parser

bool

Default

“true”

Required

No

Whether to store the missing symbols in Postgres or not. If you disable this, at the time of writing, missing symbols will be stored in the Redis default cache.

UPLOAD_TEMPDIR_PREFIX
Parser

str

Default

“raw-uploads”

Required

No

The prefix used when generating directories in the temp directory.

ALLOW_UPLOAD_BY_ANY_DOMAIN
Parser

bool

Default

“false”

Required

No

When doing local development, especially load testing, it’s sometimes useful to be able to bypass all URL checks for Upload by Download.

SYNCHRONOUS_UPLOAD_FILE_UPLOAD
Parser

bool

Default

“false”

Required

No

This is only really meant for the sake of being overrideable by other setting classes; in particular when running tests.

SECRET_KEY
Parser

str

Required

Yes

Django’s secret key for signing things.

DEBUG
Parser

bool

Default

“false”

Required

No

Whether or not to enable debug mode. Don’t set this to True in server environments

ALLOWED_HOSTS
Parser

ListOf(str)

Default

“”

Required

No

Comma-delimited list of strings of host/domain names for this site.

DATABASE_URL
Parser

dj_database_url.parse

Required

Yes

The database_url to use. This gets parsed into DATABASES configuration.

CONN_MAX_AGE
Parser

int

Default

“60”

Required

No

Maximum age in minutes for connections.

Parser

int

Default

“str(60 * 60 * 24 * 365)”

Required

No

Age in seconds for cookies. Keep it quite short because we don’t have a practical way to do OIDC ID token renewal for this AJAX and curl heavy app.

SYMBOL_URLS
Parser

ListOf(str)

Required

Yes

Comma-separated list of urls for symbol lookups.

The order here matters. Symbol download goes through these one at a time. Ideally you want the one most commonly hit first unless there’s a cascading reason you want other buckets first.

By default, each URL is assumed to be private!

If there’s a bucket you want to include that should be accessed by HTTP only, add ‘?access=public’ to the URL.

UPLOAD_DEFAULT_URL
Parser

str

Required

Yes

The default url to use for symbols. This must be a public bucket and one of the items in SYMBOL_URLS.

UPLOAD_TRY_SYMBOLS_URL
Parser

str

Required

Yes

When an upload comes in with symbols from a Try build, these symbols mustn’t be uploaded with the regular symbols.

You could set this to UPLOAD_DEFAULT_URL with a ‘/try’ prefix.

For example:

UPLOAD_DEFAULT_URL=http://s3.example.com/publicbucket/
UPLOAD_TRY_SYMBOLS_URL=http://s3.example.com/publicbucket/try/
UPLOAD_URL_EXCEPTIONS
Parser

dict_parser

Default

“{}”

Required

No

This is a config that, typed as a Python dictionary, specifies specific email addresses or patterns to custom URLs.

For example:

UPLOAD_URL_EXCEPTIONS={"peter@example.com":"https://s3.amazonaws.com/bucket"}

or

UPLOAD_URL_EXCEPTIONS={"*@example.com": "https://s3.amazonaws.com/bucket"}

anybody uploading with an @example.com email address.

SYMBOL_FILE_PREFIX
Parser

str

Default

“v1”

Required

No

Prefix in the bucket for all symbol files. This allows us to change the file path template.

COMPRESS_EXTENSIONS
Parser

ListOf(str)

Default

“sym”

Required

No

During upload, for each file in the archive, if the extension matches this list, the file gets gzip compressed before uploading.

MIME_OVERRIDES
Parser

dict_parser

Default

“{“sym”:”text/plain”}”

Required

No

For specific file uploads, override the mimetype.

For .sym files, for example, if S3 knows them as ‘text/plain’ they become really handy to open in a browser and view directly.

SYMBOLS_GET_TIMEOUT
Parser

int

Default

“5”

Required

No

Number of seconds to wait for a symbol download. If this trips, no error will be raised and we’ll just skip using it as a known symbol file. The value gets cached as an empty dict for one hour.

DISALLOWED_SYMBOLS_SNIPPETS
Parser

ListOf(str)

Default

“qcom/proprietary”

Required

No

Individual strings that can’t be allowed in any of the lines in the content of a symbols archive file.

SYMBOLDOWNLOAD_EXISTS_TTL_SECONDS
Parser

int

Default

“str(60 * 60 * 6)”

Required

No

We can cache quite aggressively here because the SymbolDownloader has chance to invalidate certain keys. Also, any time a symbol archive file is upload, for each file within that we end up uploading to S3 we also cache invalidate.

UPLOAD_REATTEMPT_LIMIT_SECONDS
Parser

int

Default

“str(60 * 60 * 12)”

Required

No

Every time we do a symbol upload, we also take a look to see if there are incomplete uploads that could have failed due to some unlucky temporary glitch.

When we do the reattempt, we need to wait sufficiently long because the upload might just be incomplete because it’s in the queue, not because it failed.

Note also, if the job is put back into a celery job, we also log this in the cache so that it doesn’t add it more than once. That caching uses this same timeout.

ALLOW_UPLOAD_BY_DOWNLOAD_DOMAINS
Parser

ListOf(str)

Default

“queue.taskcluster.net,firefox-ci-tc.services.mozilla.com,stage.taskcluster.nonprod.cloudops.mozgcp.net”

Required

No

When you “upload by download”, the URL’s domain needs to be in this allow list. This is to double-check that we don’t allow downloads from domains we don’t fully trust.

DOWNLOAD_FILE_EXTENSIONS_ALLOWED
Parser

ListOf(str)

Default

“.sym,.dl_,.ex_,.pd_,.dbg.gz,.tar.bz2”

Required

No

A list of file extensions that if a file is NOT one of these extensions we can immediately return 404 and not bother to process for anything else.

It’s case sensitive and has to be lower case. As a way to get marginal optimization of this, make sure ‘.sym’ is first in the list since it’s the most common.

Symbolication Service configuration (Eliot)

Webapp

The Symbolication Service (aka Eliot) is run as worker processes by Gunicorn which is run by Honcho.

Gunicorn configuration:

ELIOT_GUNICORN_WORKERS
Parser

str

Default

“1”

Required

No

Specifies the number of gunicorn workers.

Gunicorn docs suggest to set it to (2 x $num_cores) + 1.

https://docs.gunicorn.org/en/stable/settings.html#workers

https://docs.gunicorn.org/en/stable/design.html#how-many-workers

Used in bin/run_eliot_web.sh.

ELIOT_GUNICORN_TIMEOUT
Parser

str

Default

“300”

Required

No

Specifies the timeout value.

https://docs.gunicorn.org/en/stable/settings.html#timeout

Used in bin/run_eliot_web.sh.

ELIOT_GUNICORN_PORT
Parser

str

Default

“8000”

Required

No

Specifies the port to listen to.

Used in bin/run_eliot_web.sh.

ELIOT_GUNICORN_CMD_PREFIX
Parser

str

Default

“”

Required

No

Specifies a command prefix to run the gunicorn process in.

Used in bin/run_eliot_web.sh.

Webapp configuration:

Configuration

Configuration summary:

Setting

Parser

Required?

ELIOT_LOCAL_DEV_ENV

bool

ELIOT_HOST_ID

str

ELIOT_LOGGING_LEVEL

str

ELIOT_STATSD_HOST

str

ELIOT_STATSD_PORT

int

ELIOT_STATSD_NAMESPACE

str

ELIOT_SECRET_SENTRY_DSN

str

ELIOT_SYMBOLS_CACHE_DIR

str

ELIOT_SYMBOLS_URLS

<ListOf(str)>

Configuration options:

ELIOT_LOCAL_DEV_ENV
Parser

bool

Default

“False”

Required

No

Whether or not this is a local development environment.

ELIOT_HOST_ID
Parser

str

Default

“”

Required

No

Identifier for the host that is running Eliot. This identifies this Eliot instance in the logs and makes it easier to correlate Eliot logs with other data. For example, the value could be a public hostname, an instance id, or something like that. If you do not set this, then socket.gethostname() is used instead.

ELIOT_LOGGING_LEVEL
Parser

str

Default

“INFO”

Required

No

The logging level to use. DEBUG, INFO, WARNING, ERROR or CRITICAL

ELIOT_STATSD_HOST
Parser

str

Default

“localhost”

Required

No

Hostname for statsd server.

ELIOT_STATSD_PORT
Parser

int

Default

“8124”

Required

No

Port for statsd server.

ELIOT_STATSD_NAMESPACE
Parser

str

Default

“”

Required

No

Namespace for statsd metrics.

ELIOT_SECRET_SENTRY_DSN
Parser

str

Default

“”

Required

No

Sentry DSN to use. If this is not set an unhandled exception logging middleware will be used instead. See https://docs.sentry.io/quickstart/#configure-the-dsn for details.

ELIOT_SYMBOLS_CACHE_DIR
Parser

str

Default

“/tmp/cache”

Required

No

Location for caching symcache files.

ELIOT_SYMBOLS_URLS
Parser

<ListOf(str)>

Default

https://symbols.mozilla.org/try/

Required

No

Comma-separated list of urls to pull symbols files from.

Disk cache manager

The disk cache manager is run as a single process by Honcho.

Configuration

Configuration summary:

Setting

Parser

Required?

ELIOT_LOCAL_DEV_ENV

bool

ELIOT_HOST_ID

str

ELIOT_LOGGING_LEVEL

str

ELIOT_STATSD_HOST

str

ELIOT_STATSD_PORT

int

ELIOT_STATSD_NAMESPACE

str

ELIOT_SECRET_SENTRY_DSN

str

ELIOT_SYMBOLS_CACHE_DIR

str

ELIOT_SYMBOLS_CACHE_MAX_SIZE

int

Configuration options:

ELIOT_LOCAL_DEV_ENV
Parser

bool

Default

“False”

Required

No

Whether or not this is a local development environment.

ELIOT_HOST_ID
Parser

str

Default

“”

Required

No

Identifier for the host that is running Eliot. This identifies this Eliot instance in the logs and makes it easier to correlate Eliot logs with other data. For example, the value could be a public hostname, an instance id, or something like that. If you do not set this, then socket.gethostname() is used instead.

ELIOT_LOGGING_LEVEL
Parser

str

Default

“INFO”

Required

No

The logging level to use. DEBUG, INFO, WARNING, ERROR or CRITICAL

ELIOT_STATSD_HOST
Parser

str

Default

“localhost”

Required

No

Hostname for statsd server.

ELIOT_STATSD_PORT
Parser

int

Default

“8124”

Required

No

Port for statsd server.

ELIOT_STATSD_NAMESPACE
Parser

str

Default

“”

Required

No

Namespace for statsd metrics.

ELIOT_SECRET_SENTRY_DSN
Parser

str

Default

“”

Required

No

Sentry DSN to use. If this is not set an unhandled exception logging middleware will be used instead.

See https://docs.sentry.io/quickstart/#configure-the-dsn for details.

ELIOT_SYMBOLS_CACHE_DIR
Parser

str

Default

“/tmp/cache”

Required

No

Location for caching symcache files.

ELIOT_SYMBOLS_CACHE_MAX_SIZE
Parser

int

Default

“1073741824”

Required

No

Max size (bytes) of symbols cache. You can use _ to group digits for legibility.