Symbolication¶
Background¶
The Mozilla Symbolication Server (https://symbolication.services.mozilla.com/) has a symbolication API for converting memory addresses to symbols using debug information generated by builds of Mozilla products that were uploaded to the Mozilla Symbols Server.
Note
2021-10-07
The old symbolication API service hosted on symbols.mozilla.org is deprecated.
Status |
URL |
---|---|
Old and deprecated |
|
New |
Please update to the new service.
Stacks¶
The stack is a list of lists. Each frame in the stack is composed of two things:
the index in the modules list for the module for this frame
the memory offset relative to the base memory address for that module as an base-10 integer
For example, [1, 65802]
is the base memory address 65,802 for
the 1-index module in the memoryMap
.
Modules¶
A module is something loaded into memory. It can be a library, binary, font, etc. We need to get the debug information for a given module in order to know what symbols correspond to which addresses.
For Breakpad SYM files, modules are denoted by a debug filename and debug id.
Each debug filename and debug id pair corresponds to a path in the symbols
store. The full URL comes from taking the base url, appending the debug
filename, appending the debug id, and then appending the debug filename again
with a .sym
extension.
For example:
["wntdll.pdb", "D74F79EB1F8D4A45ABCD2F476CCABACC2"]
becomes:
https://example.com/v1/wntdll.pdb/D74F79EB1F8D4A45ABCD2F476CCABACC2/wntdll.sym
^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^
| | debug filename with .sym extension
| debug id
debug filename
Symbolication¶
Given a set of modules and a set of stacks, symbolication is the act of fetching the debug symbols files for the modules and then looking up the symbols corresponding to the module offsets.
For eaxmple, given this set of modules:
["firefox.pdb", "5F84ACF1D63667F44C4C44205044422E1"],
["mozavcodec.pdb", "9A8AF7836EE6141F4C4C44205044422E1"],
["Windows.Media.pdb", "01B7C51B62E95FD9C8CD73A45B4446C71"],
["xul.pdb", "09F9D7ECF31F60E34C4C44205044422E1"],
...
and this stack:
[3, 6516407],
[3, 12856365],
[3, 12899916],
[3, 13034426],
...
you might end up with something like this:
0 xul.pdb mozilla::ConsoleReportCollector::FlushReportsToConsole(unsigned long long, nsIConsoleReportCollector::ReportAction)
1 xul.pdb mozilla::net::HttpBaseChannel::MaybeFlushConsoleReports()",
2 xul.pdb mozilla::net::HttpChannelChild::OnStopRequest(nsresult const&, mozilla::net::ResourceTimingStructArgs const&, mozilla::net::nsHttpHeaderArray const&, nsTArray<mozilla::net::ConsoleReportCollected> const&)
3 xul.pdb std::_Func_impl_no_alloc<`lambda at /builds/worker/checkouts/gecko/netwerk/protocol/http/HttpChannelChild.cpp:1001:11',void>::_Do_call()
...
Symbolication: /symbolicate/v5¶
- POST /symbolicate/v5¶
Symbolicate one or more jobs. Each job consists of one or more stacks and a memory map.
Send an HTTP POST request with a JSON payload.
Set a User-Agent. For example:
my-script/1.0 (+https://example.com/path-to-code/)
For documentation on setting useful user agents: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent#crawler_and_bot_ua_strings
Example request:
POST /symbolicate/v5 HTTP/1.1 User-Agent: my-script/1.0 (+https://example.com/path-to-code) { "jobs": [ { "stacks": [ [ [ 1, 7556668 ], [ 1, 7509754 ] ] ], "memoryMap": [ [ "firefox.pdb", "C0A5F7D110D262364C4C44205044422E1" ], [ "xul.pdb", "0FBE970321AB8CF14C4C44205044422E1" ] ] } ], "version": 5 }
Example response: 1
HTTP/1.1 200 OK Content-Type: application/json { "results": [ { "stacks": [ [ { "frame": 0, "module": "xul.dll", "module_offset": "0x734e3c", "function": "mozilla::dom::JSActorManager::ReceiveRawMessage(mozilla::dom::JSActorMessageMeta const&, mozilla::Maybe<mozilla::dom::ipc::StructuredCloneData>&&, mozilla::Maybe<mozilla::dom::ipc::StructuredCloneData>&&)", "function_offset": "0x7dc", "file": "hg:hg.mozilla.org/mozilla-central:dom/ipc/jsactor/JSActorManager.cpp:55c63a6c547f1fecd412505a064f21fd1e1ec48e", "line": 172 }, { "frame": 1, "module": "xul.dll", "module_offset": "0x7296fa", "function": "mozilla::dom::WindowGlobalChild::RecvRawMessage(mozilla::dom::JSActorMessageMeta const&, mozilla::Maybe<mozilla::dom::ClonedMessageData> const&, mozilla::Maybe<mozilla::dom::ClonedMessageData> const&)", "function_offset": "0x1ba", "file": "hg:hg.mozilla.org/mozilla-central:dom/ipc/WindowGlobalChild.cpp:55c63a6c547f1fecd412505a064f21fd1e1ec48e", "line": 586 } ] ], "found_modules": { "firefox.pdb/C0A5F7D110D262364C4C44205044422E1": null, "xul.pdb/0FBE970321AB8CF14C4C44205044422E1": true } } ] }
- 1
The example response is indented for readability.
Here’s an example you can copy and paste–though symbols do age out of our system, so this may not be all that exciting:
curl --user-agent "my-script/1.0 (+https://example.com/)" \ -d '{"jobs": [{"stacks":[[[0,11723767],[1, 65802]]],"memoryMap":[["xul.pdb","44E4EC8C2F41492B9369D6B9A059577C2"],["wntdll.pdb","D74F79EB1F8D4A45ABCD2F476CCABACC2"]]}]}' \ https://symbolication.services.mozilla.com/symbolicate/v5
Note
Helpful tips!
Try to batch symbolication so a single request contains multiple jobs. That’ll reduce the HTTP request/response overhead.
If you can, batch requests for the same build of the same product.
If you get an HTTP 429 or HTTP 5xx response, wait and retry again. You could use a backoff of (0s, 1s, 2s, 3s, 4s, 5s).
You should always get back a JSON response. If you don’t, treat that like a temporary failure, wait a bit and try again.
If you’re getting a 200 response, but some frames aren’t symbolicated, then either Eliot doesn’t have debugging symbols for that module or the debugging symbols for that module are malformed.
- Request JSON Object
jobs –
array of json objects each specifying a job to symbolicate
- [].memoryMap
array of
[debug name (str), debug id (str)]
arrays- [].stacks
array of stacks where each stack is an array of
[module index (int), memory offset (int)]
arrays
- Response JSON Object
results –
array of result objects–one for every job
- [].stacks
array of symbolicated stacks where each stack is an array of JSON objects
- frame (int)
frame index; 0-based
- module (str)
the module name
- module_offset (str)
the module offset in hex
- function (str)
(optional) the function name
- function_offset (str)
(optional) the function offset in hex
- file (str)
(optional) the source file
- line (int)
(optional) the line number in the source file
- [].found_modules
json object indicating which modules we had symbols for and which ones we didn’t
- <debug_filename>/<debug_id> (str)
true if we found symbols, false if we didn’t, and null if we didn’t need to look up symbols because it’s not referenced in the stacks
- Request Headers
please provide a unique user agent to make it easier for us to help you debug problems
For example:
my-script/1.0 (+https://example.com/)
See the MDN docs on good user agent practices:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent#crawler_and_bot_ua_strings
Debug – if you add
Debug: true
to the headers, then symbolication will also return debug information about cache lookups, how many downloads, timings, and some other things
- Status Codes
200 OK – success symbolicating stacks
429 Too Many Requests – your request was rate-limited; wait a bit and retry 2
500 Internal Server Error – something bad happened–please open up a bug
503 Service Unavailable – temporary problem with the server; wait a bit and retry
- 2
Eliot isn’t currently rate-limiting, but it’s something we could add in the future, so it’s best for you to handle it now.
Symbolication: /symbolicate/v4 (deprecated)¶
- POST /symbolicate/v4¶
Symbolicate one or more stacks.
Send an HTTP POST request with a JSON payload.
Warning
Don’t use Symbolication v4 for anything new. Please migrate to v5 as soon as you can. v4 will be removed in the near future.
Follow this bug for status:
Example request:
POST /symbolicate/v4 HTTP/1.1 { "memoryMap": [ [ "xul.pdb", "44E4EC8C2F41492B9369D6B9A059577C2" ], [ "wntdll.pdb", "D74F79EB1F8D4A45ABCD2F476CCABACC2" ] ], "stacks": [ [ [0, 11723767], [1, 65802] ] ], "version": 4 }
Example response:
HTTP/1.1 200 OK Content-Type: application/json { "symbolicatedStacks": [ [ "XREMain::XRE_mainRun() (in xul.pdb)", "KiUserCallbackDispatcher (in wntdll.pdb)" ] ], "knownModules": [ true, true ] }