Prometheus Interview Questions (2026): Most Asked

A time series is a stream of timestamped samples belonging to one metric name plus a specific set of label key-value pairs. Prometheus's own docs describe the notation as <metric_name>{<label>="<value>",...}, so http_requests_total{method="POST", handler="/messages"} is one distinct series (Prometheus documentation, data model). Change any label value and you're not updating that series, you've created a brand new one.

A counter only goes up, or resets to zero on a restart, and is meant for things you'd never want to decrease, requests served, errors seen, bytes sent. A gauge can move in either direction and represents a current value, memory usage, queue depth, number of active connections. The practical tell: if you ever find yourself calling .dec() or subtracting from a metric in your instrumentation code, it should have been a gauge, not a counter.

http_requests_total     # counter, monotonically increasing
process_resident_memory_bytes # gauge, goes up and down

As of recent Prometheus releases, yes, both metric names and label names can technically contain UTF-8 characters. But the documentation still recommends sticking to the conventional pattern, [a-zA-Z_:][a-zA-Z0-9_:]*, for compatibility with older exporters, client libraries, and PromQL tooling that assumes the classic character set (Prometheus documentation, data model). Names that use the reserved colon are meant only for user-defined recording rules, not raw instrumented metrics.

Label names beginning with __ are reserved for Prometheus's own internal use, and by convention they're dropped before a series ever gets stored. The one nearly every user runs into is __name__, which actually holds the metric name itself internally, meaning a metric name is really just a special label at the implementation level. Relabeling configs are the other common place these show up, __address__ and __meta_* labels carry service discovery metadata that gets rewritten or dropped before scraping.

An instant vector selector returns one sample per matching series, all sharing the same timestamp, the most recent value at or before the query's evaluation time. A range vector selector, written with a duration in square brackets, returns every sample within that time window per series, not just the latest one. That's exactly why functions like rate() require a range vector as input, they need more than one point to compute a rate.

http_requests_total        # instant vector, one value per series
http_requests_total[5m]       # range vector, every sample from the last 5 minutes

= matches an exact label value, != excludes an exact value, =~ matches a regular expression against the label value, and !~ excludes anything matching that regex. Regex matches in PromQL are always fully anchored, so method=~"GET" only matches the exact string "GET", not any string containing it, you'd need method=~".*GET.*" for a substring match.

http_requests_total{status_code="500"}
http_requests_total{status_code!="200"}
http_requests_total{handler=~"/api/.*"}

rate() computes the per-second average rate of increase across the entire range you give it, extrapolating slightly to cover the range's endpoints, which makes it stable and the right choice for alerting rules and dashboards tracking slower-moving counters. irate() only looks at the last two data points inside the range and computes the rate between just those, which makes it react instantly to spikes but also makes it noisy and unsuitable for alerting, since a single scrape hiccup can make it swing wildly (Prometheus documentation, query functions).

Prometheus periodically reaches out to each configured target over HTTP and scrapes a plaintext or protobuf-encoded metrics endpoint, typically /metrics, rather than waiting for applications to push data to it. That's the pull model. It means Prometheus itself controls the scrape interval and can detect a target that's unreachable, versus a push model where a silent, crashed sender might never be noticed at all.

static_configs is a fixed, hardcoded list of targets in the config file, fine for a handful of stable servers but unworkable once instances come and go dynamically. Service discovery integrations query an external system, Kubernetes' API, Consul, EC2, DNS, on a regular basis and update the scrape target list automatically as instances appear and disappear, without anyone editing a config file by hand.

No, and the project is explicit about this. Local on-disk storage is meant for a few weeks of retention at most, useful for alerting and recent dashboards, but it's not built as a durable, long-term archive on its own, it's a single node with no built-in replication of that historical data. For genuinely long-term retention or multi-cluster aggregation, the standard approach is remote_write to a separate long-term storage system built for it.

A recording rule pre-computes an expression on a schedule and saves the result as a brand new time series, purely to make a dashboard or a downstream query faster and cheaper than re-running an expensive expression on every page load. An alerting rule evaluates an expression on a schedule too, but instead of storing a new series, it decides whether that expression's result should fire an alert, which then gets pushed to Alertmanager.

Grouping bundles related firing alerts into a single notification instead of paging someone once per affected instance. Alertmanager's own docs give the canonical example: if hundreds of instances in a cluster lose connectivity to the same database at once, grouping by cluster and alert name turns hundreds of individual pages into one notification listing every affected instance (Prometheus documentation, Alertmanager).

An exporter is a small standalone process that translates metrics from a system that doesn't natively speak Prometheus's format into a /metrics endpoint Prometheus can scrape directly. mysqld_exporter, for example, connects to a MySQL instance, runs its own status queries against it, and re-exposes the results as standard Prometheus metrics, letting you monitor systems you can't or don't want to modify directly.

Because the label set is part of the series identity, not metadata attached to it. Internally, Prometheus keys everything off the full combination of metric name and labels. If a pod's instance label changes on redeploy, the old series doesn't get renamed, it just stops receiving samples and a new series starts from zero. This is exactly why counters that reset to zero on restart look like a dip in a dashboard rather than a clean handoff, and why long-lived label values like pod IDs quietly leave behind thousands of dead series that still cost memory until they're compacted out.

Both observe a distribution, typically request durations or response sizes, but they store that distribution differently. A histogram buckets observations into configurable ranges and exposes a count per bucket, letting you compute quantiles after the fact, at query time, across any dimension, and even aggregate quantiles across multiple instances. A summary calculates quantiles client-side over a sliding window and exposes those pre-computed φ-quantiles directly, which is cheaper to query but those quantiles can't be aggregated across instances, averaging two p99s doesn't give you a real p99.

The Prometheus docs describe histograms as counting observations "in configurable buckets" and note native histograms now offer dynamic bucketing that's more storage-efficient than the classic fixed-bucket approach (Prometheus documentation, metric types). My default answer in an interview: reach for a histogram unless you have a specific reason not to, because you can always compute a quantile later, but you can't un-collapse a summary's pre-aggregated number.

Because splitting a metric by status code into separate metric names throws away the ability to query across them together. http_responses_total{code="500"} and http_responses_total{code="403"} can be summed, filtered, or graphed side by side with one expression; five separately-named metrics can't be combined without hardcoding every one of them into a query. The guidance is to keep the metric name describing the general thing being measured and push anything that varies into labels instead (Prometheus documentation, instrumentation practices).

A classic histogram isn't actually one time series, it's a family. _bucket with a le label exposes a cumulative count of observations less than or equal to each bucket boundary, one series per boundary plus a +Inf bucket that equals the total count. _sum tracks the running total of all observed values, and _count tracks the total number of observations, which not coincidentally equals the +Inf bucket.

http_request_duration_seconds_bucket{le="0.1"}
http_request_duration_seconds_bucket{le="0.5"}
http_request_duration_seconds_bucket{le="+Inf"}
http_request_duration_seconds_sum
http_request_duration_seconds_count

offset shifts the time a selector is evaluated relative to the query's own evaluation timestamp, without changing the query time itself. It's the standard tool for week-over-week or day-over-day comparisons in one expression instead of running two separate queries and diffing them by hand.

http_requests_total offset 1w  # value from exactly one week before now
(rate(http_requests_total[5m]) - rate(http_requests_total[5m] offset 1w))

A scalar is a single numeric floating point value with no labels attached at all, it's not tied to any time series. An instant vector, even one that happens to have exactly one result, still carries a label set and is conceptually a set of series. This matters for arithmetic: dividing a vector by a scalar applies that scalar to every element, but dividing two vectors requires PromQL to match up their label sets first, which is where on() and ignoring() come in.

http_requests_total / 100            # vector divided by a scalar, fine everywhere
http_requests_total / sum(http_requests_total) # vector divided by vector, needs label matching

Nothing, these are two equivalent syntaxes for the same aggregation, PromQL accepts the grouping clause either right after the aggregation keyword or right after the parenthesized expression. What actually matters is what's inside the sum, this pattern of applying rate() first and aggregating second, not the other way around.

rate() automatically detects counter resets by watching for any drop in value between consecutive samples, and it adjusts for that break in monotonicity instead of treating the drop as a real negative rate. The docs put it plainly: "breaks in monotonicity, such as counter resets due to target restarts, are automatically adjusted for" (Prometheus documentation, query functions). You still want at least two scrape intervals of data inside the range so rate() has enough points to detect and correct for the reset confidently.

increase() is effectively rate() multiplied by the number of seconds in the range, which the docs describe directly, it's meant to make a rate more human-readable as a total count over a window rather than a per-second figure. It is not simply last sample minus first sample. Because rate() extrapolates slightly to the exact edges of the requested range, increase() can return a number a little higher or lower than the naive raw difference, especially over a short window with few samples.

delta() calculates the simple difference between the first and last value in a range, with no counter-reset correction and no rate-of-change normalization to per-second. It's meant for gauges, where a raw drop from one sample to the next is a real, meaningful decrease, memory freed, a queue draining. Running delta() against a counter would treat a legitimate restart-driven reset to zero as an enormous negative delta, which is exactly the wrong answer.

histogram_quantile() takes a φ value between 0 and 1 as its first argument and a set of bucketed counter series (the _bucket family with the le label) as its second, and it interpolates within the bucket boundaries to estimate that quantile. Because it interpolates linearly inside buckets, the precision of the resulting quantile is only as good as how tightly your bucket boundaries are chosen around the values you actually expect to see.

histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Relabeling happens before a scrape target is finalized (in relabel_configs) or before a scraped sample is written to storage (in metric_relabel_configs), and it can act on service discovery metadata, the __meta_* labels, that never becomes a real, stored label unless you copy it there yourself. That's the entire mechanism Kubernetes service discovery relies on, deciding which pods to scrape and what labels to attach based on pod annotations, all before a single sample is ever written to disk.

relabel_configs:
 - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
  action: keep
  regex: "true"
 - source_labels: [__meta_kubernetes_namespace]
  target_label: namespace

metric_relabel_configs with an action of drop, matched against __name__, is the standard way. This runs post-scrape but pre-storage, so the metric was still fetched over the network but never gets written into the TSDB, which is exactly what you want when a third-party exporter emits something you don't control and can't fix at the source.

metric_relabel_configs:
 - source_labels: [__name__]
  regex: "some_noisy_high_cardinality_metric"
  action: drop

The head block holds the most recent data, roughly the last couple of hours, entirely in memory and backed by a write-ahead log for crash recovery. On a fixed interval, Prometheus compacts the head block down into an immutable, on-disk block, then eventually merges several small blocks into larger ones during background compaction, similar in spirit to how an LSM-tree database compacts levels. Older blocks past the configured retention window get deleted entirely.

The WAL records every incoming sample to disk immediately, before it's fully indexed into the in-memory head block, purely so that data isn't lost if the process crashes or is killed before the head block gets compacted. On restart, Prometheus replays the WAL to rebuild the head block's in-memory state from where it left off. This is why a Prometheus restart can take noticeably longer after an unclean shutdown, replaying a large WAL takes real time.

Consistency across consumers is the underrated reason, beyond raw speed. If five different dashboards and two alerting rules all need "5-minute error rate by service," writing that expression out five separate times risks someone getting the aggregation or the label matching subtly wrong in one of them. A single recording rule, referenced everywhere by name, guarantees every consumer sees the exact same computed number.

groups:
 - name: example
  rules:
   - record: job:http_errors:rate5m
    expr: sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (job)

for requires the alert's condition to stay true continuously across multiple rule evaluations before it transitions from pending to firing, which filters out brief, self-resolving blips that don't deserve a page. Without a for clause, any single bad evaluation, one slow scrape, one momentary spike, immediately fires and pages someone, and flapping conditions can fire and resolve repeatedly within minutes, training the on-call rotation to ignore that alert.

- alert: HighErrorRate
 expr: job:http_errors:rate5m > 0.05
 for: 10m
 labels:
  severity: page

An alert enters the pending state the first time its expression evaluates true, and stays there until the for duration has elapsed with the condition remaining true the entire time, at which point it flips to firing. Alertmanager only ever receives alerts once they've reached the firing state, the pending state is purely internal bookkeeping inside Prometheus itself and never gets forwarded anywhere.

Inhibition is a rule you configure ahead of time, it suppresses certain alerts automatically whenever another, related alert is already firing, the classic case is muting every individual service alert for a cluster once a "cluster unreachable" alert is already active, since those downstream alerts add no new information. Silencing is a manual, time-boxed action taken through the Alertmanager UI or API, typically used for planned maintenance, matching alerts by label and muting them for a set duration that someone has to explicitly configure in the moment.

Routing is a tree of matchers, each route can match on alert labels (severity, team, service) and either handle the alert itself by sending it to a specific receiver, or continue matching against child routes nested underneath it. An alert falls through to the first matching route unless that route's continue flag is explicitly set to true, in which case matching keeps evaluating sibling and child routes too, letting one alert notify multiple receivers when that's genuinely the intent.

route:
 receiver: default
 routes:
  - match:
    severity: page
   receiver: pagerduty
  - match:
    team: data
   receiver: data-team-slack

Federation is one Prometheus server scraping a subset of another Prometheus server's own metrics, typically already-aggregated recording rule results, through a special /federate endpoint. It's meant for hierarchical setups, a global Prometheus pulling high-level, pre-aggregated numbers from several per-cluster Prometheus instances, giving a cross-cluster view without every single raw metric from every cluster flowing into one central instance.

Pushgateway exists for short-lived batch jobs that finish and exit before Prometheus's next scheduled scrape would ever catch them, it lets a job push its final metric values to an intermediary that Prometheus then scrapes normally. The project's own guidance is that it's not meant as a general push-based alternative to normal scraping, since it breaks Prometheus's usual up-detection (a target being unreachable no longer means a service is actually down, it might just mean nothing has pushed recently), and metrics pushed there persist and get re-scraped indefinitely until explicitly deleted, which can leave stale values looking current long after the job that pushed them has stopped running entirely.

Every unique label combination is a fully separate time series with its own chunk of memory in the head block, its own entry in the inverted index, and its own line in every scrape. The classic mistake is putting something like a raw user_id or a full URL path with query parameters on a metric that's scraped from thousands of instances. The docs' own worked example: adding per-user filesystem quotas across 10,000 machines and 10,000 users would multiply one sane metric into potentially millions of series (Prometheus documentation, instrumentation practices).

To spot one before it's a problem, I look at prometheus_tsdb_head_series for a sudden jump, and I use topk(10, count by (__name__)({__name__=~".+"})) against the API to find which metric name owns the most series. Catching it in code review is cheaper: any label value pulled from user input, a raw error string, a request path, a session token, should be a red flag the moment it's proposed.

PromQL's default matching behavior for binary operators between two vectors requires every label to match exactly on both sides, and it expects exactly one match per element. The error fires when the label sets don't line up one-to-one, usually because the right side has fewer distinguishing labels than the left, so one series on the right would otherwise need to match multiple series on the left.

The fix is to be explicit with on() to restrict which labels must match, combined with group_left or group_right to tell PromQL which side is allowed to have the extra, unmatched labels.

http_requests_total * on(instance) group_left(job) up

offset shifts a query relative to its own evaluation time, but the @ modifier pins a selector to a fixed Unix timestamp regardless of when the overall query is evaluated. That makes it useful for building a query that always compares against one specific historical moment, say, the exact instant a deployment happened, no matter what time window you're currently looking at on a dashboard.

http_requests_total @ 1700000000
rate(http_requests_total[5m] @ 1700000000)

rate() needs to see each individual series' own raw counter history to correctly detect and correct for that series' own resets. If you sum the raw counters across instances first and then take the rate of the sum, one instance restarting and resetting to zero gets buried inside the combined total, and depending on the shape of the other series, the reset-detection logic can either silently mask the dip or, worse, misattribute a real drop as a legitimate decrease. Applying rate() per-series first, then summing the resulting rates, keeps reset handling correct on each series independently before they're ever combined.

sum(rate(http_requests_total[5m])) by (job)  # correct
sum(http_requests_total) by (job)       # then take a derivative, incorrect for resets

rate() needs at least two samples inside the range to compute anything, and a scrape can occasionally be delayed or dropped for one interval. A range of one minute against a 30-second scrape interval leaves almost no margin, a single missed or slightly late scrape can leave only one sample in the window, which produces no result at all. The standard guidance is to size the range at roughly four times the scrape interval as a safety margin, so a 30-second scrape interval pairs comfortably with a range closer to two or four minutes.

Each scraped instance produces its own full set of le bucket series. If you feed histogram_quantile() the raw, unaggregated bucket series across ten instances, it sees ten overlapping copies of each bucket boundary instead of one combined distribution, and the interpolation math breaks down. Summing by le first (after applying rate() to each bucket series) merges all instances into one combined cumulative distribution, which is the shape histogram_quantile() actually expects.

relabel_configs runs during target discovery, before Prometheus decides whether to scrape a target at all and what labels that target carries. metric_relabel_configs runs after the scrape has already happened, on each individual metric sample, and it's meant for dropping or rewriting metrics you don't want stored, not for deciding which targets to scrape in the first place. Using metric_relabel_configs to drop noisy metrics still costs the network and CPU of scraping them, it just saves storage; using relabel_configs to filter targets entirely avoids the scrape cost too.

By default, if a scraped metric already carries a label that would collide with a label Prometheus wants to attach itself (like job or instance), Prometheus renames its own version with an exported_ prefix and keeps the target's original value under the original name. Setting honor_labels true flips that priority, the scraped label wins outright and Prometheus's own value is dropped instead. It's mostly used for federation, scraping one Prometheus from another, where you specifically want the original instance's labels preserved rather than overwritten by the scraping server's identity.

remote_write streams every sample Prometheus scrapes to an external endpoint in near real time, typically a long-term storage backend designed for horizontal scale and durability. It doesn't replace local storage because Prometheus still needs its own local TSDB to evaluate alerting and recording rules quickly and reliably, remote_write is fundamentally a one-way, best-effort forward, not a synchronous write, so a slow or unreachable remote endpoint shouldn't be allowed to block or degrade local rule evaluation and alerting.

By default, Prometheus's TSDB is append-only per series and expects samples to arrive in increasing timestamp order. A sample that arrives with the exact same value at the exact same timestamp as one already stored is silently treated as a duplicate and dropped without error, but a sample with a genuinely older timestamp than the most recent one already stored for that series gets rejected outright unless out-of-order ingestion has been explicitly enabled in the TSDB configuration, since it wasn't designed to insert into the middle of an existing block by default.

Cardinality growth inside the head block. Memory usage tracks the number of active series in the current head block far more closely than it tracks scrape count or target count, ten targets each emitting a hundred thousand unique series will exhaust memory long before a thousand well-behaved targets each emitting a few dozen series would. This is the same root cause as most cardinality horror stories, a new label value on an existing metric quietly doubles or triples the series count without anyone touching the scrape config at all.

Colons in metric names are reserved specifically for recording rules, and the loose convention is level:metric:operations, the aggregation level first, the base metric name second, and the operations applied (rate, sum, and so on) last. It's not enforced by the engine, nothing breaks if you skip it, but it's the one place a naming convention actually helps a team scan a rules file and immediately understand what a rule computes without opening its expression.

Yes, and it's a common pattern, referencing a pre-aggregated recording rule keeps the alerting expression itself simple and fast. Evaluation order within a single rule group does matter here, rules inside one group are evaluated sequentially in the order they're written, so a recording rule has to be defined before an alerting rule in the same group if that alert depends on it, or the alert would be evaluating against a series that doesn't exist yet on that pass. Rules in separate groups evaluate independently and their relative order isn't guaranteed the same way.

A single Alertmanager instance is a single point of failure for notifications, if it's down when something breaks, nobody gets paged even though Prometheus is correctly evaluating the alert as firing. Running a cluster with gossip-based state sharing between instances means multiple Prometheus servers can send the same alert to any Alertmanager in the cluster, and the cluster deduplicates and coordinates so the same alert doesn't fire a separate, duplicate notification from each member.

No, and this is a real operational gotcha. A silence matches on the exact label matchers configured when it was created, it doesn't track an alert's identity across a label change, it re-evaluates the match criteria fresh on every incoming alert. If a relabeling change alters a label the silence was matching on, even a value that still logically refers to the same underlying condition, the silence stops applying and the alert starts notifying again, which has caught more than one team off guard during a maintenance window right after an unrelated config deploy.

Federation still scrapes over HTTP on a schedule, so it inherits the same scaling ceiling any single Prometheus instance has, one process, one set of local storage, one memory footprint for the head block. Pulling raw metrics from many clusters defeats the entire purpose, you'd be recreating the original cardinality problem one level up, now concentrated onto a single global instance instead of distributed across per-cluster instances. It's specifically designed for pre-aggregated, already-reduced data, not as a general-purpose replication mechanism.

A single large instance is simpler operationally, one config, one storage volume, one set of rules, but it caps out on the memory and CPU limits of one machine and becomes a single point of failure for every team relying on it. Sharding by service or team distributes that load and blast radius, a bad deploy from one team's high-cardinality metric can't take down monitoring for every other team, at the cost of real complexity: you now need federation or a remote-write aggregation layer to get any cross-cutting view, and every team needs to actually own and maintain their own instance instead of relying on a shared platform team to do it once.