Dynamic Sampling Context (DSC)

All data sent to Sentry will end up in a Trace. Traces can be sampled by the tracesSampleRate or tracesSampler options in the SDKs.

Changing those options has quite a few consequences for users of Sentry SDKs:

• Changing the sampling rate involved either redeploying applications (which is problematic in case of applications that are not updated automatically, i.e., mobile apps or physically distributed software) or building complex systems to dynamically fetch a sample rate.
• The sampling strategy leverages head-based simple random sampling.
• Employing sampling rules, for example, based on event parameters, is very complex. Sometimes this even requires users to have an in-depth understanding of the event schema.
• While writing rules for singular transactions is possible, enforcing them on entire traces is infeasible.

The solution for this is Dynamic Sampling. Dynamic Sampling allows Sentry to automatically adjust the amount of data retained based on the value of the data. For detailed information see Dynamic Sampling.

High-Level Problem Statement

Ingest

Implementing Dynamic Sampling comes with challenges, especially on the ingestion side of things. For Dynamic Sampling, we want to make sampling decisions for entire traces. However, to keep ingestion speedy, Relay only looks at singular transactions in isolation (as opposed to looking at whole traces). This means that we need the exact same decision basis for all transactions belonging to a trace. In other words, all transactions of a trace need to hold all of the information to make a sampling decision, and that information needs to be the same across all transactions of the trace. We call the information we base sampling decisions on "Dynamic Sampling Context" or "DSC".

Currently, we can dynamically sample in two ways. First, we can do dynamic sampling on single transactions. For this process, Relay looks at the incoming event payload to make decisions. Second, we can do dynamic sampling across an entire trace. For this process, Relay relies on a Dynamic Sampling Context.

As a mental model: The head transaction in a trace determines the Dynamic Sampling Context for all following transactions in that trace. No information can be changed, added or deleted after the first propagation. Dynamic Sampling Context is bound to only one particular trace, and all the transactions that are part of this trace. Multiple different traces can and should have different Dynamic Sampling Contexts.

SDKs

SDKs are responsible for propagating a "Dynamic Sampling Context" or "DSC" across all applications that are part of a trace. This involves:

1. If there is an incoming request, extracting the DSC from incoming requests.
2. If there is no incoming request, creating a new DSC by collecting the data that makes up the DSC.
3. Propagating DSC to downstream SDKs (via the baggage header).
4. Sending the DSC to Sentry (via the trace envelope header).

To align DSC propagation over all our SDKs, we defined a unified propagation mechanism (step-by-step instructions) that all SDK implementations should be able to follow.

DSC Specification

All of the attributes in the table below are required (non-optional) in a sense, that when they are known to an SDK at the time an envelope with an event (transaction or error) is sent to Sentry, or at the time a baggage header is propagated, they must also be included in said envelope or baggage.

At the moment, only release, environment and transaction are used by the product for dynamic sampling functionality. The rest of the context attributes, trace_id, public_key, sampled and sample_rate, are used by Relay for internal decisions (like transaction sample rate smoothing). Additional entries such as replay_id, org and sample_rand are only using the DSC as a mean of transport.

0: In any case, trace_id, public_key, and sample_rate should always be known to an SDK, so these values are strictly required.

1: UUID V4 encoded as a hexadecimal sequence with no dashes that is a sequence of 32 hexadecimal digits.

2: It allows Sentry to sample traces spanning multiple projects, by resolving the same set of rules based on the starting project.

3: This string should always be a number between (and including) 0 and 1 in a notation that is supported by the JSON specification. If a tracesSampler callback was used for the sampling decision, its result should be used for sample_rate instead of the tracesSampleRate from SentryOptions. In case tracesSampler returns True it should be sent as 1.0, False should be sent as 0.0.

DSC Propagation

The DSC is sent to two destinations:

• to Sentry via the trace envelope header and
• to downstream services via the baggage header.

Envelope Header

Dynamic Sampling Context is transferred to Sentry through the trace envelope header. The value of this envelope header is a JSON object containing the fields specified in the DSC Specification.

Baggage Header

We chose baggage as the propagation mechanism for DSC. (w3c baggage spec). Baggage is a standard HTTP header with URI encoded key-value pairs.

For the propagation of DSC, SDKs first read the DSC from the baggage header of incoming requests/messages.

To propagate DSC to downstream SDKs/services, we create a baggage header (or modify an existing one) through HTTP request instrumentation.

The baggage header should only be attached to an outgoing request if the request's URL matches at least one entry of the tracePropagationTargets SDK option or this option is set to null or not set.

SDKs must set all of the keys in the form of "sentry-[name]". Where [name] is the attribute in the DSC Specification. The prefix "sentry-" acts to identify key-value pairs set by Sentry SDKs.

The following is an example of what a baggage header containing Dynamic Sampling Context might look like:

baggage: other-vendor-value-1=foo;bar;baz, sentry-trace_id=771a43a4192642f0b136d5159a501700, sentry-public_key=49d0f7386ad645858ae85020e393bef3, sentry-sample_rate=0.01337, sentry-user_id=Am%C3%A9lie, other-vendor-value-2=foo;bar;

The sentry- prefix allows SDKs to put all of the sentry key-value pairs from the baggage directly onto the envelope header, after stripping away the sentry- prefix.

Being able to simply copy key-value pairs from the baggage header onto the trace envelope header gives us the flexibility to provide dedicated API methods to propagate additional values using Dynamic Sampling Context. This, in return, allows users to define their own values in the Dynamic Sampling Context so they can sample by those in the Sentry interface.

Freezing Dynamic Sampling Context

As mentioned above, in order to be able to make sampling decisions for entire traces, Dynamic Sampling Context must be the same across all transactions of a trace.

What does this mean for SDKs?

When starting a new trace, SDKs are no longer allowed to alter the DSC for that trace as soon as this DSC leaves the boundaries of the SDK for the first time. The DSC is then considered "frozen". DSC leaves SDKs in two situations:

• When an outgoing request with a baggage header, containing the DSC, is made.
• When a transaction envelope containing the DSC is sent to Sentry

When an SDK receives an HTTP request that was "instrumented" or "traced" by a Sentry SDK, the receiving SDK should consider the incoming DSC as instantly frozen. Any values on the DSC should be propagated "as is" - this includes values like "environment" or "release".

SDKs should recognize incoming requests as "instrumented" or "traced" when at least one of the following applies:

• The incoming request has a sentry-trace and/or traceparent header
• The incoming request has a baggage header containing one or more keys starting with "sentry-"

After the DSC of a particular trace has been frozen, API calls like set_user should have no effect on the DSC.

Unified Propagation Mechanism

SDKs should follow these steps for any incoming and outgoing requests (in python pseudo-code for illustrative purposes):

def collect_dynamic_sampling_context():
  # Placeholder function that collects as many values for Dynamic Sampling Context
  # as possible and returns a dict

def has_sentry_value_in_baggage_header(request):
  # Placeholder function that returns True when there is at least one key-value pair in the baggage
  # header of `request`, for which the key starts with "sentry-". Otherwise, it returns False.

def on_incoming_request(request):
  if (request.has_header("sentry-trace") or request.has_header("traceparent")) and (not request.has_header("baggage") or not has_sentry_value_in_baggage_header(request)):
    # Request comes from an old SDK which doesn't support Dynamic Sampling Context yet
    # --> we don't propagate baggage for this trace
    current_transaction.dynamic_sampling_context_frozen = True
  elif request.has_header("baggage") and has_sentry_value_in_baggage_header(request):
    current_transaction.dynamic_sampling_context_frozen = True
    current_transaction.dynamic_sampling_context = baggage_header_to_dict(request.headers.baggage)

def on_outgoing_request(request):
  if not current_transaction.dynamic_sampling_context_frozen:
    current_transaction.dynamic_sampling_context_frozen = True
    current_transaction.dynamic_sampling_context = merge_dicts(collect_dynamic_sampling_context(), current_transaction.dynamic_sampling_context)

  if not current_transaction.dynamic_sampling_context:
    # Make sure there is at least an empty DSC set on transaction
    # This is independent of whether it is locked or not
    current_transaction.dynamic_sampling_context = {}

  if request.has_header("baggage"):
    outgoing_baggage_dict = baggage_header_to_dict(request.headers.baggage)
    merged_baggage_dict = merge_dicts(outgoing_baggage_dict, current_transaction.dynamic_sampling_context)
    request.set_header("baggage", dict_to_baggage_header(merged_baggage_dict))
  else:
    request.set_header("baggage", dict_to_baggage_header(current_transaction.dynamic_sampling_context))

While there is no strict necessity for the current_transaction.dynamic_sampling_context_frozen flag yet, there is a future use case where we need it: We might want users to be able to set Dynamic Sampling Context values themselves. The flag becomes relevant after the first propagation, where Dynamic Sampling Context becomes immutable. When users attempt to set DSC afterwards, our SDKs should make this operation a noop.

Considerations and Challenges

This section details some open questions and considerations that need to be addressed for dynamic sampling and the usage of the baggage propagation mechanism. These are not blockers to the adoption of the spec, but instead are here as context for future developments of the dynamic sampling product and spec.

The Temporal Problem

Unlike environment or release, which should always be known to an SDK at initialization time, user_segment, and transaction (name) are only known after SDK initialization time. This means that if a trace is propagated from a running transaction BEFORE the user/transaction attributes are set, you'll get a portion of transactions in a trace that have different Dynamic Sampling Context than other portions, leading to dynamic sampling across a trace not working as expected for users.

Let's say we want to dynamically sample a browser application based on the user_segment. In a typical single page application (SPA), the user information has to be requested from some backend service before it can be set with Sentry.setUser on the frontend.

Here's an example of that flow:

• Page starts loading
• Sentry initializes and starts pageload transaction
• Page makes HTTP request to user service to get user (propagates sentry-trace/traceparent/baggage to user service)
  • user service continues trace by automatically creating sampling transaction
  • user service pings database service (propagates sentry-trace/traceparent/baggage to database service)
  • database service continues trace by automatically creating sampling transaction
• Page gets data from user service, calls Sentry.setUser and sets user_segment
• Page makes HTTP requests to service A, service B, and service C (propagates sentry-trace/traceparent/baggage to services A, B and C)
  • DSC is propagated with baggage to service A, service B, and service C, so 3 child transactions
• Page finishes loading, finishing pageload transaction, which is sent to Sentry

In this case, the baggage that is propagated to the user service and the downstream database service does not have the user_segment value in it, because it was not yet set on the browser SDK. Therefore, when Relay tries to dynamically sample the user services and database services transactions based on user_segment, it will not be able to. In addition, since the DSC is frozen after it's been sent, the DSC sent to service A, service B, and service C will not have user_segment on it either. This means it also will not be dynamically sampled properly if there is a trace-wide DS rule on user_segment.

For transaction name, the problem is similar, but it is because of parameterization. As much as we can, the SDKs will try to parameterize transaction names (for ex, turn /teams/123/user/456 into /teams/:id/user/:id) so that similar transactions are grouped together in the UI. This improves both aggregate statistics for transactions and the general UX of using the product (setting alerts, checking measurements like web vitals, etc.). For some frameworks, for example React Router v4 - v6, we parameterize the transaction name after the transaction has been started due to constraints with the framework itself. To illustrate this, let's look at another example:

• Page starts loading
• Sentry initializes and starts pageload transaction (with transaction name /teams/123/user/456 - based on window URL)
• Page makes HTTP request to service A (propagates sentry-trace/traceparent/baggage to user service)
• Page renders with react router, triggering parameterization of transaction name (/teams/123/user/456 -> /teams/:id/user/:id).
• Page finishes loading, finishing pageload transaction, which is sent to Sentry

When the pageload transaction shows up in Sentry, it'll be called /teams/:id/user/:id, but the Dynamic Sampling Context has /teams/123/user/456 propagated, which means that the transaction from service A will not be affected by any trace-wide dynamic sampling rules put on the transaction name. This will be very confusing to users, as effectively the transaction name they thinking that works will not.

Choosing Baggage as the Propagation Mechanism

For more information on this, check the DACI around using baggage (Sentry employees only).

Previously, we were using the w3c Trace Context spec as the mechanism to propagate dynamic sampling context between SDKs. We switched to the w3c Baggage spec though because it was easier to use, required less code on the SDK side, and had much more liberal size limits than the trace context spec. To make sure that we were not colliding with user defined keys as baggage is an open standard, we decided to prefix all sentry related keys with sentry- as a namespace. This allowed us to use baggage as an open standard while only having to worry about Sentry data.

Other Items

TODO - Add some sort of Q&A section on the following questions, after evaluating if they still need to be answered:

• Why must baggage be immutable before the second transaction has been started?
• What are the consequences and impacts of the immutability of baggage on Dynamic Sampling UX?
• Why can't we just make the decision for the whole trace in Relay after the trace is complete?
• What is sample rate smoothing and how does it use sample_rate from the Dynamic Sampling Context?
• What are the differences between Dynamic Sampling on traces vs. transactions?