Symbolication

Background

The Mozilla Symbolication Server (https://symbolication.services.mozilla.com/) has a symbolication API for converting memory addresses to symbols using debug information generated by builds of Mozilla products that were uploaded to the Mozilla Symbols Server.

Note

2021-10-07

There are two different implementations of the Symbolication API:

Status

URL

Old and deprecated

https://symbols.mozilla.org/symbolicate/v5

New

https://symbolication.services.mozilla.com/symbolicate/v5

Please migrate to using the new server.

Stacks

The stack is a list of lists. Each frame in the stack is composed of two things:

  1. the index in the modules list for the module for this frame

  2. the memory offset relative to the base memory address for that module as an base-10 integer

For example, [1, 65802] is the base memory address 65,802 for the 1-index module in the memoryMap.

Modules

A module is something loaded into memory. It can be a library, binary, font, etc. We need to get the debug information for a given module in order to know what symbols correspond to which addresses.

For Breakpad SYM files, modules are denoted by a debug filename and debug id.

Each debug filename and debug id pair corresponds to a path in the symbols store. The full URL comes from taking the base url, appending the debug filename, appending the debug id, and then appending the debug filename again with a .sym extension.

For example:

["wntdll.pdb", "D74F79EB1F8D4A45ABCD2F476CCABACC2"]

becomes:

https://example.com/v1/wntdll.pdb/D74F79EB1F8D4A45ABCD2F476CCABACC2/wntdll.sym
                       ^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^
                       |          |                                 debug filename with .sym extension
                       |          debug id
                       debug filename

Symbolication

Given a set of modules and a set of stacks, symbolication is the act of fetching the debug symbols files for the modules and then looking up the symbols corresponding to the module offsets.

For eaxmple, given this set of modules:

["firefox.pdb", "5F84ACF1D63667F44C4C44205044422E1"],
["mozavcodec.pdb", "9A8AF7836EE6141F4C4C44205044422E1"],
["Windows.Media.pdb", "01B7C51B62E95FD9C8CD73A45B4446C71"],
["xul.pdb", "09F9D7ECF31F60E34C4C44205044422E1"],
...

and this stack:

[3, 6516407],
[3, 12856365],
[3, 12899916],
[3, 13034426],
...

you might end up with something like this:

0  xul.pdb  mozilla::ConsoleReportCollector::FlushReportsToConsole(unsigned long long, nsIConsoleReportCollector::ReportAction)
1  xul.pdb  mozilla::net::HttpBaseChannel::MaybeFlushConsoleReports()",
2  xul.pdb  mozilla::net::HttpChannelChild::OnStopRequest(nsresult const&, mozilla::net::ResourceTimingStructArgs const&, mozilla::net::nsHttpHeaderArray const&, nsTArray<mozilla::net::ConsoleReportCollected> const&)
3  xul.pdb  std::_Func_impl_no_alloc<`lambda at /builds/worker/checkouts/gecko/netwerk/protocol/http/HttpChannelChild.cpp:1001:11',void>::_Do_call()
...

Symbolication: /symbolicate/v5

POST /symbolicate/v5

Symbolicate one or more jobs. Each job consists of one or more stacks and a memory map.

Send an HTTP POST request with a JSON payload.

Set a User-Agent. For example:

my-script/1.0 (+https://example.com/path-to-code/)

For documentation on setting useful user agents: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent#crawler_and_bot_ua_strings

Example request:

POST /symbolicate/v5 HTTP/1.1
User-Agent: my-script/1.0 (+https://example.com/path-to-code)

{
  "jobs": [
    {
      "stacks": [
        [
          [
            1,
            7556668
          ],
          [
            1,
            7509754
          ]
        ]
      ],
      "memoryMap": [
        [
          "firefox.pdb",
          "C0A5F7D110D262364C4C44205044422E1"
        ],
        [
          "xul.pdb",
          "0FBE970321AB8CF14C4C44205044422E1"
        ]
      ]
    }
  ],
  "version": 5
}

Example response: 1

HTTP/1.1 200 OK
Content-Type: application/json

{
  "results": [
    {
      "stacks": [
        [
          {
            "frame": 0,
            "module": "xul.dll",
            "module_offset": "0x734e3c",
            "function": "mozilla::dom::JSActorManager::ReceiveRawMessage(mozilla::dom::JSActorMessageMeta const&, mozilla::Maybe<mozilla::dom::ipc::StructuredCloneData>&&, mozilla::Maybe<mozilla::dom::ipc::StructuredCloneData>&&)",
            "function_offset": "0x7dc",
            "file": "hg:hg.mozilla.org/mozilla-central:dom/ipc/jsactor/JSActorManager.cpp:55c63a6c547f1fecd412505a064f21fd1e1ec48e",
            "line": 172
          },
          {
            "frame": 1,
            "module": "xul.dll",
            "module_offset": "0x7296fa",
            "function": "mozilla::dom::WindowGlobalChild::RecvRawMessage(mozilla::dom::JSActorMessageMeta const&, mozilla::Maybe<mozilla::dom::ClonedMessageData> const&, mozilla::Maybe<mozilla::dom::ClonedMessageData> const&)",
            "function_offset": "0x1ba",
            "file": "hg:hg.mozilla.org/mozilla-central:dom/ipc/WindowGlobalChild.cpp:55c63a6c547f1fecd412505a064f21fd1e1ec48e",
            "line": 586
          }
        ]
      ],
      "found_modules": {
        "firefox.pdb/C0A5F7D110D262364C4C44205044422E1": null,
        "xul.pdb/0FBE970321AB8CF14C4C44205044422E1": true
      }
    }
  ]
}
1

The example response is indented for readability.

Here’s an example you can copy and paste–though symbols do age out of our system, so this may not be all that exciting:

curl --user-agent "my-script/1.0 (+https://example.com/)" \
    -d '{"jobs": [{"stacks":[[[0,11723767],[1, 65802]]],"memoryMap":[["xul.pdb","44E4EC8C2F41492B9369D6B9A059577C2"],["wntdll.pdb","D74F79EB1F8D4A45ABCD2F476CCABACC2"]]}]}' \
    https://symbolication.services.mozilla.com/symbolicate/v5

Note

Helpful tips!

  1. Try to batch symbolication so a single request contains multiple jobs. That’ll reduce the HTTP request/response overhead.

    If you can, batch requests for the same build of the same product.

  2. If you get an HTTP 429 or HTTP 5xx response, wait and retry again. You could use a backoff of (0s, 1s, 2s, 3s, 4s, 5s).

  3. You should always get back a JSON response. If you don’t, treat that like a temporary failure, wait a bit and try again.

  4. If you’re getting a 200 response, but some frames aren’t symbolicated, then either Eliot doesn’t have debugging symbols for that module or the debugging symbols for that module are malformed.

Request JSON Object
  • jobs

    array of json objects each specifying a job to symbolicate

    [].memoryMap

    array of [debug name (str), debug id (str)] arrays

    [].stacks

    array of stacks where each stack is an array of [module index (int), memory offset (int)] arrays

Response JSON Object
  • results

    array of result objects–one for every job

    [].stacks

    array of symbolicated stacks where each stack is an array of JSON objects

    frame (int)

    frame index; 0-based

    module (str)

    the module name

    module_offset (str)

    the module offset in hex

    function (str)

    (optional) the function name

    function_offset (str)

    (optional) the function offset in hex

    file (str)

    (optional) the source file

    line (int)

    (optional) the line number in the source file

    [].found_modules

    json object indicating which modules we had symbols for and which ones we didn’t

    <debug_filename>/<debug_id> (str)

    true if we found symbols, false if we didn’t, and null if we didn’t need to look up symbols because it’s not referenced in the stacks

Request Headers
Status Codes
2

Eliot isn’t currently rate-limiting, but it’s something we could add in the future, so it’s best for you to handle it now.

Symbolication: /symbolicate/v4 (deprecated)

POST /symbolicate/v4

Symbolicate one or more stacks.

Send an HTTP POST request with a JSON payload.

Warning

Don’t use Symbolication v4 for anything new. Please migrate to v5 as soon as you can. v4 will be removed in the near future.

Follow this bug for status:

https://bugzilla.mozilla.org/show_bug.cgi?id=1475334

Example request:

POST /symbolicate/v4 HTTP/1.1

{
  "memoryMap": [
    [
      "xul.pdb",
      "44E4EC8C2F41492B9369D6B9A059577C2"
    ],
    [
      "wntdll.pdb",
      "D74F79EB1F8D4A45ABCD2F476CCABACC2"
    ]
  ],
  "stacks": [
    [
      [0, 11723767],
      [1, 65802]
    ]
  ],
  "version": 4
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "symbolicatedStacks": [
    [
      "XREMain::XRE_mainRun() (in xul.pdb)",
      "KiUserCallbackDispatcher (in wntdll.pdb)"
    ]
  ],
  "knownModules": [
    true,
    true
  ]
}