Labels vs annotations: how one YAML indent made Prometheus scrape Pyroscope twice

A Prometheus ServiceMonitor selects Services by labels. It never looks at annotations. In Grafana Pyroscope’s Helm chart, a refactor nested prometheus.io/service-monitor: "false" under annotations instead of labels on the headless Service, so the exclusion silently stopped working. Prometheus scraped every pod twice, as job="pyroscope" and job="pyroscope-headless". My fix, grafana/pyroscope#5377 (in review), moves the key back under metadata.labels.

What’s the difference between labels and annotations in Kubernetes?

Both are key-value metadata, but they do different jobs:

Aspect	Labels	Annotations
Purpose	Identify and select objects	Attach non-identifying information
Used by selectors (Services, ServiceMonitors, kubectl `-l`)	Yes	No
Typical content	`app`, `component`, `tier`	build info, tool configuration, long text

Some tools read annotations for configuration, like the old prometheus.io/scrape convention for annotation-based scraping. That’s exactly why the two get mixed up. But the Prometheus Operator’s ServiceMonitor uses a label selector.

How did the chart exclude the headless Service?

Pyroscope runs a normal Service and a headless one pointing at the same pods. Scraping both would duplicate every metric, so the chart’s ServiceMonitor excludes Services marked with a label:

ServiceMonitor selector

selector:
  matchExpressions:
    - key: prometheus.io/service-monitor
      operator: NotIn
      values:
        - "false"

The headless Service carries prometheus.io/service-monitor: "false", so it’s left out. That only works if the key is a label.

What broke?

A change that added support for custom headlessAnnotations (#4451) changed the YAML nesting in services.yaml. The prometheus.io/service-monitor: "false" line ended up under metadata.annotations.

Nothing failed. helm template still rendered valid YAML, the Service still worked, and Prometheus still scraped. It just scraped the same pod IPs twice under two job names (issue #5215): duplicate series, double the storage for those metrics, and dashboards that sum by job quietly counting double.

How did I fix it?

I moved the prometheus.io/service-monitor: "false" block so it renders under metadata.labels for the headless Service.
I kept the headlessAnnotations feature from #4451 working.
I bumped the chart version (2.1.2 → 2.1.3) and regenerated the rendered golden fixtures with make helm/check.

You can verify it with:

Check the rendered headless Service

helm template . -s templates/services.yaml --set serviceMonitor.enabled=true

and check that the key now sits under metadata.labels on the headless Service.

How do you catch this class of bug?

Test the selection, not just the rendering. A golden-file test that asserts “the headless Service has label X” would have failed when the key moved.
Watch for duplicate series. The same instance showing up under two job values is a classic sign of double scraping.
Review YAML nesting changes like logic changes. In Helm templates, indentation is the logic.

More upstream fixes, from LiteLLM to Node.js, are on my projects page.