Datadog APM monitor trace count discrepancy

You may run into a case when your Datadog APM monitor, configured for trace counting, shows weird, usually low number of traces, compared to Trace Overview section. Look at this monitor:

It shows 4 spans:

However, Trace Overview shows 28 spans/traces for the same time range:

The main question here: why?

There is a tool named Datadog Notebooks. You can create a new one and construct your query:

I encourage you to select Retained By option when grouping results (1). You see, that we have only one retention reason available: retention_filter (2). And our trace count is small (3).

This is the answer! You can impersonate APM monitor logic by yourself in Trace Overview page by choosing “All Spans” scope (1), adding explicit retained_by:retention_filter to the search query (2) and boom – you see the same amount of traces/spans that APM monitor shows:

If you wonder where this thing is located in the Datadog docs, here you go: https://docs.datadoghq.com/tracing/trace_pipeline/trace_retention/#in-dashboards-notebooks-and-monitors.

Quote: “spans indexed by the intelligent retention filter are excluded from APM queries that appear in dashboards and notebooks and also excluded from trace analytics monitor evaluation.”

Telegram channel

If you still have any questions, feel free to ask me in the comments under this article or write me at promark33@gmail.com.

If I saved your day, you can support me 🤝

One thought on “Datadog APM monitor trace count discrepancy

  1. Ежегодно в течение сентября организовывается Тюменский инновационный форум «[u][b]НЕФТЬГАЗТЭК[/b][/u]».
    Форум посвящен определению способов инновационного развития областей [b]топливно-энергетического комплекса[/b], рассмотрению а также поиску заключений, организации наилучших обстоятельств для формирования инноваторских проектов. Ежегодный тюменский форум представляетсобой авторитетной дискуссионной площадкой по продвижению нефтегазовой сферы в России, содержит большой статус и своевременность, созвучен общей стратегии продвижения инноваторского курса в [u][b]Российской Федерации[/b][/u]
    -https://neftgaztek.ru/

Leave a Reply

Your email address will not be published. Required fields are marked *