Symbolication¶

Background
Symbolication: /symbolicate/v5
Symbolication: /symbolicate/v4 (deprecated)

Background ¶

The Mozilla Symbolication Server (https://symbolication.services.mozilla.com/) has a symbolication API for converting memory addresses to symbols using debug information generated by builds of Mozilla products that were uploaded to the Mozilla Symbols Server.

Note

2021-10-07

There are two different implementations of the Symbolication API:

Status	URL
Old and deprecated	https://symbols.mozilla.org/symbolicate/v5
New	https://symbolication.services.mozilla.com/symbolicate/v5

Please migrate to using the new server.

Stacks ¶

The stack is a list of lists. Each frame in the stack is composed of two things:

the index in the modules list for the module for this frame
the memory offset relative to the base memory address for that module as an base-10 integer

For example, [1, 65802] is the base memory address 65,802 for the 1-index module in the memoryMap.

Modules ¶

A module is something loaded into memory. It can be a library, binary, font, etc. We need to get the debug information for a given module in order to know what symbols correspond to which addresses.

For Breakpad SYM files, modules are denoted by a debug filename and debug id.

Each debug filename and debug id pair corresponds to a path in the symbols store. The full URL comes from taking the base url, appending the debug filename, appending the debug id, and then appending the debug filename again with a .sym extension.

For example:

["wntdll.pdb", "D74F79EB1F8D4A45ABCD2F476CCABACC2"]

becomes:

https://example.com/v1/wntdll.pdb/D74F79EB1F8D4A45ABCD2F476CCABACC2/wntdll.sym
                       ^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^
                       |          |                                 debug filename with .sym extension
                       |          debug id
                       debug filename

Symbolication ¶

Given a set of modules and a set of stacks, symbolication is the act of fetching the debug symbols files for the modules and then looking up the symbols corresponding to the module offsets.

For eaxmple, given this set of modules:

["firefox.pdb", "5F84ACF1D63667F44C4C44205044422E1"],
["mozavcodec.pdb", "9A8AF7836EE6141F4C4C44205044422E1"],
["Windows.Media.pdb", "01B7C51B62E95FD9C8CD73A45B4446C71"],
["xul.pdb", "09F9D7ECF31F60E34C4C44205044422E1"],
...

and this stack:

[3, 6516407],
[3, 12856365],
[3, 12899916],
[3, 13034426],
...

you might end up with something like this:

xul.pdb  mozilla::ConsoleReportCollector::FlushReportsToConsole(unsigned long long, nsIConsoleReportCollector::ReportAction)
xul.pdb  mozilla::net::HttpBaseChannel::MaybeFlushConsoleReports()",
xul.pdb  mozilla::net::HttpChannelChild::OnStopRequest(nsresult const&, mozilla::net::ResourceTimingStructArgs const&, mozilla::net::nsHttpHeaderArray const&, nsTArray<mozilla::net::ConsoleReportCollected> const&)
xul.pdb  std::_Func_impl_no_alloc<`lambda at /builds/worker/checkouts/gecko/netwerk/protocol/http/HttpChannelChild.cpp:1001:11',void>::_Do_call()
...

Symbolication: /symbolicate/v5 ¶

POST /symbolicate/v5¶

Symbolicate one or more jobs. Each job consists of one or more stacks and a memory map.

Send an HTTP POST request with a JSON payload.

Set a User-Agent. For example:

my-script/1.0 (+https://example.com/path-to-code/)

For documentation on setting useful user agents: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent#crawler_and_bot_ua_strings

Example request:

POST /symbolicate/v5 HTTP/1.1
User-Agent: my-script/1.0 (+https://example.com/path-to-code)

{
  "jobs": [
    {
      "stacks": [
        [
          [
            1,
            7556668
          ],
          [
            1,
            7509754
          ]
        ]
      ],
      "memoryMap": [
        [
          "firefox.pdb",
          "C0A5F7D110D262364C4C44205044422E1"
        ],
        [
          "xul.pdb",
          "0FBE970321AB8CF14C4C44205044422E1"
        ]
      ]
    }
  ],
  "version": 5
}

Example response: 1

HTTP/1.1 200 OK
Content-Type: application/json

{
  "results": [
    {
      "stacks": [
        [
          {
            "frame": 0,
            "module": "xul.dll",
            "module_offset": "0x734e3c",
            "function": "mozilla::dom::JSActorManager::ReceiveRawMessage(mozilla::dom::JSActorMessageMeta const&, mozilla::Maybe<mozilla::dom::ipc::StructuredCloneData>&&, mozilla::Maybe<mozilla::dom::ipc::StructuredCloneData>&&)",
            "function_offset": "0x7dc",
            "file": "hg:hg.mozilla.org/mozilla-central:dom/ipc/jsactor/JSActorManager.cpp:55c63a6c547f1fecd412505a064f21fd1e1ec48e",
            "line": 172
          },
          {
            "frame": 1,
            "module": "xul.dll",
            "module_offset": "0x7296fa",
            "function": "mozilla::dom::WindowGlobalChild::RecvRawMessage(mozilla::dom::JSActorMessageMeta const&, mozilla::Maybe<mozilla::dom::ClonedMessageData> const&, mozilla::Maybe<mozilla::dom::ClonedMessageData> const&)",
            "function_offset": "0x1ba",
            "file": "hg:hg.mozilla.org/mozilla-central:dom/ipc/WindowGlobalChild.cpp:55c63a6c547f1fecd412505a064f21fd1e1ec48e",
            "line": 586
          }
        ]
      ],
      "found_modules": {
        "firefox.pdb/C0A5F7D110D262364C4C44205044422E1": null,
        "xul.pdb/0FBE970321AB8CF14C4C44205044422E1": true
      }
    }
  ]
}

1: The example response is indented for readability.

Here’s an example you can copy and paste–though symbols do age out of our system, so this may not be all that exciting:

curl --user-agent "my-script/1.0 (+https://example.com/)" \
    -d '{"jobs": [{"stacks":[[[0,11723767],[1, 65802]]],"memoryMap":[["xul.pdb","44E4EC8C2F41492B9369D6B9A059577C2"],["wntdll.pdb","D74F79EB1F8D4A45ABCD2F476CCABACC2"]]}]}' \
    https://symbolication.services.mozilla.com/symbolicate/v5

Note

Helpful tips!

Try to batch symbolication so a single request contains multiple jobs. That’ll reduce the HTTP request/response overhead.

If you can, batch requests for the same build of the same product.
If you get an HTTP 429 or HTTP 5xx response, wait and retry again. You could use a backoff of (0s, 1s, 2s, 3s, 4s, 5s).
You should always get back a JSON response. If you don’t, treat that like a temporary failure, wait a bit and try again.
If you’re getting a 200 response, but some frames aren’t symbolicated, then either Eliot doesn’t have debugging symbols for that module or the debugging symbols for that module are malformed.

Request JSON Object

jobs –
array of json objects each specifying a job to symbolicate

[].memoryMap

array of [debug name (str), debug id (str)] arrays

[].stacks

array of stacks where each stack is an array of [module index (int), memory offset (int)] arrays

Response JSON Object

results –
array of result objects–one for every job

[].stacks

array of symbolicated stacks where each stack is an array of JSON objects

frame (int)

frame index; 0-based

module (str)

the module name

module_offset (str)

the module offset in hex

function (str)

(optional) the function name

function_offset (str)

(optional) the function offset in hex

file (str)

(optional) the source file

line (int)

(optional) the line number in the source file

[].found_modules

json object indicating which modules we had symbols for and which ones we didn’t

<debug_filename>/<debug_id> (str)

true if we found symbols, false if we didn’t, and null if we didn’t need to look up symbols because it’s not referenced in the stacks

Request Headers

User-Agent –
please provide a unique user agent to make it easier for us to help you debug problems

For example:
```
my-script/1.0 (+https://example.com/)
```
See the MDN docs on good user agent practices:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent#crawler_and_bot_ua_strings
Debug – if you add Debug: true to the headers, then symbolication will also return debug information about cache lookups, how many downloads, timings, and some other things

Status Codes

200 OK – success symbolicating stacks
429 Too Many Requests – your request was rate-limited; wait a bit and retry 2
500 Internal Server Error – something bad happened–please open up a bug
503 Service Unavailable – temporary problem with the server; wait a bit and retry

2: Eliot isn’t currently rate-limiting, but it’s something we could add in the future, so it’s best for you to handle it now.

Symbolication: /symbolicate/v4 (deprecated)¶

POST /symbolicate/v4¶

Symbolicate one or more stacks.

Send an HTTP POST request with a JSON payload.

Warning

Don’t use Symbolication v4 for anything new. Please migrate to v5 as soon as you can. v4 will be removed in the near future.

Follow this bug for status:

https://bugzilla.mozilla.org/show_bug.cgi?id=1475334

Example request:

POST /symbolicate/v4 HTTP/1.1

{
  "memoryMap": [
    [
      "xul.pdb",
      "44E4EC8C2F41492B9369D6B9A059577C2"
    ],
    [
      "wntdll.pdb",
      "D74F79EB1F8D4A45ABCD2F476CCABACC2"
    ]
  ],
  "stacks": [
    [
      [0, 11723767],
      [1, 65802]
    ]
  ],
  "version": 4
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "symbolicatedStacks": [
    [
      "XREMain::XRE_mainRun() (in xul.pdb)",
      "KiUserCallbackDispatcher (in wntdll.pdb)"
    ]
  ],
  "knownModules": [
    true,
    true
  ]
}

Symbolication¶

Background¶

Stacks¶

Modules¶

Symbolication¶

Symbolication: /symbolicate/v5¶

Symbolication: /symbolicate/v4 (deprecated)¶

Background ¶

Stacks ¶

Modules ¶

Symbolication ¶

Symbolication: /symbolicate/v5 ¶