Uploading symbols¶

Basics ¶

Tecken lets you upload symbols files. It stores these symbols files allowing others to use the download API to access them for symbolicating stacks, profiling, debugging, etc.

When building software, you can run dump_syms to extract debugging information and generate symbols files.

You can upload your symbols ZIP file to Tecken using the upload API.

Symbols ZIP file structure ¶

To upload symbols files, you must first put them in a ZIP file. The ZIP file must consist of symbols files in this structure:

<module>/<debug_id>/<file>

For example, here’s the contents of a symbols ZIP file from a Firefox build:

certutil/1AFC71D5AB0BEBD3C38BE53162EFA29A0/certutil.sym
crashreporter/509BEAD7F29FDF02EC309704A35C48960/crashreporter.sym
firefox-bin/4B736D100CA6655C9D41A138408EF4480/firefox-bin.sym
firefox/F18AA48D06849232EE436621609EB9030/firefox.sym
js/05817021DC072736D84A3D6F6800C09F0/js.sym
libclearkey.so/6F805460FA945EDAC1FEB2229E7E5D310/libclearkey.so.sym
libfakeopenh264.so/4E4B3E441ACA34CEC07655B61DBDCF880/libfakeopenh264.so.sym
libfake.so/F50D7C64975965FD70357FF58DD0615C0/libfake.so.sym
libfreebl3.so/D23A341A8B2614640D094F0CC429A8A80/libfreebl3.so.sym
libfreeblpriv3.so/3D7CE9A569ED46B0C77DBD3A4B1EA0470/libfreeblpriv3.so.sym
libipcclientcerts.so/8C9A9FCF80F9CE0797E7746FF900765F0/libipcclientcerts.so.sym
liblgpllibs.so/80AE7BD11E2532CFFC217154968DA6A00/liblgpllibs.so.sym
libmozavcodec.so/555BF9F90CE583DF21FB525B7E6243FF0/libmozavcodec.so.sym
libmozavutil.so/57AC5DFA71D0C5880517CF195A69A1860/libmozavutil.so.sym
libmozgtk.so/21FB2BB8FD80120D6C131AFE2EC88A080/libmozgtk.so.sym
libmozsandbox.so/CBBC8F7B7794A777E2F7BCAD64EE03780/libmozsandbox.so.sym
libmozsqlite3.so/F913FF352E918F683FAF4FAD5C0182BD0/libmozsqlite3.so.sym
libmozwayland.so/2226E0AEA0C87E748BA72A6BF258F3550/libmozwayland.so.sym
libnspr4.so/13530EC7CAB3F71EE091E41721F13CC30/libnspr4.so.sym
libnss3.so/9D8322087BEBED332EAEED0F5D6975870/libnss3.so.sym
libnssckbi.so/163C00046AA9E6198B61A6045C38E3650/libnssckbi.so.sym
libnssutil3.so/6A9E332D4559D83FFF6C725DCB7A98870/libnssutil3.so.sym
libplc4.so/BCD4DD0CF3614CB57D4AD189633B820F0/libplc4.so.sym
libplds4.so/58120BEB46A8D9B05C89DF8124D753400/libplds4.so.sym
libsmime3.so/E0897F4BD6626CE490F4974A5494B7000/libsmime3.so.sym
libsoftokn3.so/C01F5710501A0528AC50D472C754254E0/libsoftokn3.so.sym
libssl3.so/84283CDA8EBD0F78E6118BC869E6DE150/libssl3.so.sym
libxul.so/5C63DDEA1326BB8DADFCC7D606633D1E0/libxul.so.sym
logalloc-replay/DA0BFC167FB27B4A50763355964523DE0/logalloc-replay.sym
minidump-analyzer/9F3842DFF1F104156D658A5E4493539F0/minidump-analyzer.sym
modutil/847194C9942BB8D079E43A1D8FD077280/modutil.sym
pingsender/6129FD5C877F2C22EAC832C26678E8A40/pingsender.sym
pk12util/C119F9093FCBD813277074CBCD35154B0/pk12util.sym
plugin-container/45A20FE024EAD3B49CDF292851C2DD4B0/plugin-container.sym
shlibsign/835151A0E5F29FF71AD93C37F0C1E30A0/shlibsign.sym
signmar/53F4637B906F2E00353A8B451A754B7A0/signmar.sym
updater/3D7533ED47669F8A207B93E4512C35870/updater.sym
xpcshell/B95F899A59E247055C0ED7883D0235A40/xpcshell.sym

Try symbols ¶

Tecken differentiates “release builds” from “try builds”.

Try builds can be kicked off by anyone and are typically used to debug or test changes. There are many more Try builds than there are release builds. Try builds tend to be more ephemeral and not very interesting after a week or two.

For this reason, Tecken lets you differentiate between “release builds” and “try builds”. Try build symbols won’t override release build symbols. Try build symbols files expire from our system after 28 days.

There are two ways to upload symbols for try builds:

Use an auth token with “Upload Try Symbols” permission.
Include try=1 in the HTTP POST data.

Example with curl:

$ curl --user-agent "example/1.0" -X POST -H 'auth-token: xxx' \
    --form try=1 \
    --form myfile.zip=@myfile.zip \
    https://symbols.mozilla.org/upload/

See the Try builds documentation under Download.

Two ways to upload symbols ¶

While there is one API endpoint, there are two different ways to upload symbols.

Including the symbols ZIP file in the HTTP POST.
Uploading the ZIP file to a publicly available URL and then specifying that URL in the HTTP POST.

Upload by HTTP POST (payload < 2gb size)¶

Include the ZIP file in the HTTP POST to /upload/ as a multipart/form-data payload.

Here’s a curl example:

$ curl --user-agent "example/1.0" -X POST -H 'auth-token: xxx' \
    --form myfile.zip=@myfile.zip \
    https://symbols.mozilla.org/upload/

Here’s a Python example using the requests library:

>>> import requests
>>> files = {"myfile.zip": open("path/to/myfile.zip", "rb")}
>>> url = "https://symbols.mozilla.org/upload/"
>>> headers = {"User-Agent": "example/1.0", "Auth-token": "xxx"}
>>> response = requests.post(url, files=files, headers=headers)
>>> response.status_code
201

This works if the HTTP POST is less than 2gb. If the HTTP POST request is larger than 2gb, then you’ll need to use upload by download url.

Upload by download url (payload > 2gb size)¶

Upload the symbols file to some publicly available URL at an approved domain.

Then do an HTTP POST to /upload/ as a application/x-www-form-urlencoded payload and specify the url to the symbols file as a value to url.

Domains that Tecken will download from is specified in the DJANGO_ALLOW_UPLOAD_BY_DOWNLOAD_DOMAINS environment variable and at the time of this writing is set to:

queue.taskcluster.net
public-artifacts.taskcluster.net

If you need another domain supported, file a bug.

Tecken will check redirects. At first a HEAD request is made with the URL and Tecken will check both the original URL and the redirected URL against the list of allowed URLs.

An example with curl:

$ curl --user-agent "example/1.0" -X POST -H 'auth-token: xxx' \
   -d url="https://queue.taskcluster.net/YC0FgOlE/artifacts/symbols.zip" \
   https://symbols.mozilla.org/upload/

An example with Python and the requests library:

>>> import requests
>>> url = "https://symbols.mozilla.org/upload/"
>>> headers = {"User-Agent": "example/1.0", "Auth-token": "xxx"}
>>> data = {"url": "https://queue.taskcluster.net/YC0FgOlE/artifacts/symbols.zip"}
>>> response = requests.post(url, data=data, headers=headers)
>>> response.status_code
201

Permissions and auth tokens ¶

production:: https://symbols.mozilla.org/
create a bug:: https://bugzilla.mozilla.org/enter_bug.cgi?product=Tecken&component=General
create a token:: https://symbols.mozilla.org/tokens

Uploading symbols to Tecken requires special permission. The process for requesting access to upload symbols is roughly the following:

Log into Mozilla Symbols Server. When you log in, an account will be created automatically.
Create a bug requesting access to upload symbols.
A Tecken admin will process the request.

If you are a Mozilla employee, your manager will be needinfo’d to verify you need upload access.

If you are not a Mozilla employee, we’ll need to find someone to vouch for you.
After that’s been worked out, the Tecken admin will give you permission to upload symbols.

Once you have permission to upload symbols, you will additionally need an auth token. Once you log in, you can create an API token. It needs to have the “Upload Symbols” or “Upload Try Symbols” permission.

The auth token is sent as an Auth-Token HTTP header in the HTTP POST.

Note

Auth tokens support labels to make it easier to know which auth token has which permissions. A - and anything after that in the auth token is considered a label and ignored.

For example, if you had an auth token for “Upload Try Symbols”:

E468C3D4BBDA43DEBC0B856983895835

you could use:

E468C3D4BBDA43DEBC0B856983895835-uploadtry-20230913

Testing symbol uploads with our stage environment ¶

stage:: https://symbols.stage.mozaws.net/
create a token:: https://symbols.stage.mozaws.net/tokens

If you’re testing symbol uploads out, testing something that uses symbol files, testing a symbol upload script, or something like that, you might want to use our staging server. Then the tests you’re doing won’t affect production and potentially everyone using production.

To get access to our stage server:

Log into Mozilla Symbols Server (stage). When you log in, an account will be created automatically.
Ask a Tecken admin (currently willkg) to grant you upload permissions.

We hang out in #crashreporting matrix channel.

You can also find us on Slack or send us an email–whatever works best for you.

Once you have permission to upload symbols, you will additionally need an auth token. Once you log in, you can create an API token. It needs to have the “Upload Symbols” or “Upload Try Symbols” permission.

The auth token is sent as an Auth-Token HTTP header in the HTTP POST.

Note

Auth tokens created in production won’t work on stage and auth tokens created on stage won’t work in production.

Upload: /upload/¶

POST /upload/¶

Upload symbols files as a ZIP file.

Request Headers:

Content-Type –
the content type of the payload
- use multipart/form-data for Upload by HTTP POST
- use application/x-www-form-urlencoded for Upload by Download URL
Auth-Token – the value of the auth token you’re using
User-Agent – please provide a unique user agent to make it easier for us to help you debug problems

Form Parameters:

<FILENAME> –
the key is the name of the file and the value is the contents of the file; for example symbols.zip=<BINARY>

Use this for HTTP POST.

Set this or url–don’t set both.
url –
the url for the symbols file

Use this for Upload by Download URL

Set this or <FILENAME>–don’t set both.
try – use try=1 if this is an upload of try symbols

Status Codes:

201 Created – successful upload of symbols
400 Bad Request – if the specified url can’t be downloaded; verify that the url can be downloaded and retry
403 Forbidden – your auth token is invalid and you need to get a new one
413 Request Entity Too Large – your upload is too large; split it into smaller files or switch to upload by download url
429 Too Many Requests – sleep for a bit and retry
500 Internal Server Error – sleep for a bit and retry; if retrying doesn’t work, then please file a bug report
503 Service Unavailable – sleep for a bit and retry

Symbols processing ¶

Tecken processes ZIP files in a couple of steps.

First, it validates the ZIP file. See section below on “Checks and Validation”.

Once the ZIP file is validated, Tecken uploads the files in the ZIP file. For files that are already in AWS S3, it skips the uploading step and just logs the filename.

Records of the upload and what files were in it are available on the website.

Note

Symbols files can not be overwritten. Once a file is uploaded, any future upload attempts for that debug_filename/debug_id combination will be skipped.

Which S3 Bucket ¶

The S3 bucket for symbols is configured by DJANGO_UPLOAD_DEFAULT_URL. For example: https://s3-us-west-2.amazonaws.com/org-mozilla-symbols-public. From the URL the bucket name is deduced and that’s the default S3 bucket used.

Checks and Validations ¶

First, Tecken checks the ZIP file to see if it’s a valid ZIP file that contains at least one file.

Then, Tecken iterates over the files in the ZIP file and checks if any file contains the list of strings in settings.DISALLOWED_SYMBOLS_SNIPPETS. This check is a block list check to make sure proprietary files are never uploaded in S3 buckets that might be exposed publicly.

To override this amend the DJANGO_DISALLOWED_SYMBOLS_SNIPPETS environment variable as a comma separated list. But be aware to include the existing defaults which can be seen in settings.py.

The final check is to make sure that each file in the ZIP file is either:

<module>/<debug_id>/<file> for symbols files.

Example:

firefox/F18AA48D06849232EE436621609EB9030/firefox.sym

<name>-symbols.txt for file listings relative to the root of the zip file.

While these files can exist in your ZIP file, they’re silently ignored.

Gzip ¶

Certain files get gzipped before being uploaded into S3. At the time of writing that list is all .sym files. S3, unlike something like Nginx, doesn’t do content encoding on the fly based on the client’s capabilities. Instead, we manually gzip the file in memory in Tecken and set the additional ContentEncoding header to gzip. Since these .sym files are always text based, it saves a lot of memory in the S3 storage.

Additionally, the .sym files get their content type (aka. mime type) set when uploading to S3 to text/plain. Because S3 can’t know in advance that the files are actually ASCII plain text, if you try to open them in a browser it will set the Content-Type to application/octet-stream which makes it hard to quickly look at its content in a browser.

Both the gzip and the mimetype overrides can be changed by setting the DJANGO_COMPRESS_EXTENSIONS and DJANGO_MIME_OVERRIDES environment variables. See settings.py for the current defaults.

Metadata and Optimization ¶

For every gzipped file we upload, we attach 2 pieces of metadata to the key:

Original size
Original MD5 checksum

The reasons for doing this is to be able to quickly skip a file if it’s uploaded a second time.

A similar approach is done for files that don’t need to be compressed. In the case of those files, we skip uploading, again, simply if the file size of an existing file hasn’t changed. However, that approach is too expensive for compressed files. If we don’t store and retrieve the original size and original MD5 checksum, we have to locally compress the file to be able to make that final size comparison. By instead checking the original size (and hash) we can skip early without having to do the compression again.