Prometheus, and what it is not¶
Not the Bezos startup
Since late 2025 there is also Project Prometheus, Jeff Bezos' industrial AI company building an "artificial general engineer" for physical product design. It has nothing to do with this. This page is about the open-source monitoring system.
Most descriptions of Prometheus open with "time-series database", which helps nobody. Here it is in working terms:
A target runs a lightweight HTTP endpoint exposing numeric metrics with labels. Prometheus scrapes that endpoint on a schedule and buffers the raw samples for a limited retention window. Those samples are then queried with PromQL, with all arithmetic done at read time, by three consumers: dashboards, alerting rules, and someone investigating after something broke.
Five parts of that sentence carry the whole design.
1. The target must speak HTTP¶
Every target Prometheus reads is an HTTP endpoint, no exceptions. It is a trivial server, not a web stack — a client library opens a socket on a spare port and prints text:
# HELP jobs_processed_total Jobs completed
# TYPE jobs_processed_total counter
jobs_processed_total{status="ok"} 41822
jobs_processed_total{status="failed",reason="corrupt_input"} 1936
The name is the measurement. The braces hold labels, the dimensions you slice by later.
In Python that endpoint is about two lines:
from prometheus_client import start_http_server, Counter
processed = Counter('jobs_processed_total', 'Jobs completed', ['status'])
start_http_server(8000) # that is the entire "web server"
2. Prometheus pulls — the target never pushes¶
Prometheus opens the connection. The target does not need to know Prometheus exists, hold credentials for it, or understand PromQL.
PromQL never reaches the target. This tripped me up. The query language lives only between you (or Grafana) and Prometheus, after the data is stored:
you / Grafana --PromQL--> Prometheus --plain HTTP GET--> target /metrics
(stores it) <--text response---
3. Only numbers¶
Every value is a number. No strings, no logs, no events, no JSON payloads. A failure reason cannot be stored as a sentence — it becomes a label, as reason="corrupt_input" above.
That constraint is why Prometheus is fast, and also why it is wrong for anything log-shaped.
Never label with an ID
Every distinct label combination is a separate stored series, and active-series bookkeeping lives in memory. A user_id or file_id label creates a series per value and will degrade the server. Per-item detail belongs in a database or a log. Prometheus should only see aggregates: totals, rates, and breakdowns across a small fixed set of categories.
4. It buffers, it does not archive¶
Retention is bounded, typically a few weeks. Prometheus is a rolling window on recent behaviour, not a system of record. Long history needs Thanos, Mimir or VictoriaMetrics behind it.
5. The maths happens on read¶
Raw counter values are stored exactly as scraped. Rates and percentiles are computed when the query runs:
rate(jobs_processed_total[5m]) # per second
sum by (reason) (rate(jobs_processed_total[5m])) # split by failure reason
histogram_quantile(0.95, rate(job_seconds_bucket[5m]))
Because nothing is pre-aggregated at ingestion, you can ask questions you had not thought of when the data was collected.
The moving parts¶
flowchart LR
T["Application<br/>exposes /metrics"] -->|scraped every 15s| P
X["Exporter<br/>speaks for things<br/>with no HTTP"] -->|scraped every 15s| P
P[("Prometheus<br/>buffers samples<br/>for weeks")] --> D["Dashboards<br/>Grafana"]
P --> A["Alert rules<br/>then Alertmanager"]
P --> I["Ad-hoc PromQL<br/>investigation"]Does every monitored thing need to be a web server? Effectively yes, but the thing serving is usually not the thing you care about. Four ways to bridge that:
| Bridge | Use it when |
|---|---|
Local exporter (node_exporter) | You need host internals: CPU, RAM, disk. Must run on that machine. |
Remote exporter (postgres_exporter, redis_exporter) | The service is reachable over the network. The exporter can live on the monitoring box. |
Multi-target exporter (blackbox_exporter, snmp_exporter) | One process probes many devices via a ?target= parameter. Nothing installed on them. |
| Textfile collector | Lightest of all: a cron script writes a .prom file into a watched directory and node_exporter publishes it. A shell script with echo produces metrics. |
Versus Uptime Kuma¶
Different questions, and the split is black-box vs white-box.
Uptime Kuma asks "is it up?" from outside: HTTP status, TCP port, ping, DNS, keyword match, cert expiry. Binary, plus a response-time graph. Prometheus asks "what is it doing, and how much?" from inside.
A fair equation:
Uptime Kuma ≈ Prometheus + blackbox_exporter + Alertmanager + Grafana + a status page, collapsed into one small app with a web UI and no config files.
Which is why Kuma is the right tool for what it does. Reproducing it in the Prometheus stack means four components and YAML for each. The trade is that Kuma cannot go deeper — no query language, no labels, short retention, no "requests per second by endpoint" or "memory growth over 30 days".
Keep both. Kuma answers is it up, Prometheus answers is it healthy. Reach for Prometheus when you hit a question Kuma structurally cannot express, such as "will this disk fill up?":
- alert: DiskWillFillIn4Hours
expr: predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0
for: 15m
Versus ThingSpeak and friends¶
If you have ever pushed sensor readings to a hosted site to look at later, that is the fastest way in: same job, opposite direction.
| ThingSpeak and similar | Prometheus | |
|---|---|---|
| Direction | Device pushes to the service | Server pulls from the target |
| Hosting | Hosted service with an account | Self-hosted, you run it |
| Subject | Field and hobby sensors | Servers and applications you operate |
| Behind NAT or on cellular | Yes, naturally | No, the target must be reachable |
| Retention | Long, that is the point | Short, a rolling window |
| Query | Fixed charts plus export | A full query language, at read time |
| Alerting | Limited | First class |
| Tens of thousands of endpoints | Yes | No |
So Prometheus is bad for IoT?¶
For the devices, yes. The pull model cannot reach devices behind NAT, on dynamic cellular IPs, or asleep on battery. And one series per device per metric across many thousands of devices inverts the shape it was tuned for. Scraping is also uncompressed HTTP on a fixed interval, which is wasteful on metered links.
For the IoT backend, it is exactly right — the MQTT broker, Kafka, the ingestion services, the databases, the cluster running it all. Most serious IoT platforms do both: MQTT into a store built for device data, and a normal Prometheus stack for infrastructure health, with Grafana over both.
The push bridges, each narrow:
- Pushgateway — for short-lived batch jobs that die before a scrape arrives. The project's own docs discourage it as a general ingestion point.
- OpenTelemetry — pushes from the edge, batched and compressed. The modern answer.
- remote write from a per-site gateway that aggregates many devices into one upstream connection. The right pattern for a factory or a remote installation.
Why it does not belong on a shop floor¶
Worth writing down, because it looks superficially like a fit. Industrial machines publish temperatures, pressures and cycle times over OPC UA, so why not scrape them?
| What the shop floor needs | Why Prometheus cannot |
|---|---|
| Years of history for warranty and quality investigations | Retention is weeks. A buffer, not an archive. |
| Fast sampling of a physical signal, e.g. a pressure curve within one cycle | Scrapes are seconds apart. The curve's shape is lost. |
| Data identified per cycle, per cavity, per serial number | High-cardinality per-event data — exactly what labels must not carry. |
| An auditable system of record, usable as quality evidence | Prometheus makes no such guarantee about its samples. |
| Engineering units, asset models, a plant hierarchy | It has metric names and labels, and nothing else. |
That job belongs to process historians and MES products, which keep long history, sample fast, and model the plant as equipment rather than as endpoints. Prometheus is not competing there. Where it does belong in a plant is behind that boundary: the servers running the MES, the message broker, the simulation cluster, the licence servers.
Who actually uses it¶
Written at SoundCloud in 2012, modelled on an internal Google monitoring system, then released as open source. It became the second project accepted by the CNCF, after Kubernetes, and later graduated.
The predominant users are SRE, DevOps and platform teams running cloud-native infrastructure. In practice it is the default metrics layer for Kubernetes, usually with Grafana in front. Its text format became a de facto standard, so a great many open-source server products now ship a /metrics endpoint unasked.
Licence, and why Amazon sells it¶
Prometheus is Apache License 2.0 — permissive. Anyone may use it commercially, modify it, keep changes private, and sell a hosted service built on it, with no obligation to publish anything back. Attribution and a patent grant are the main conditions.
That is the direct reason Amazon Managed Service for Prometheus can exist: data arrives by remote write, queried with PromQL. Some other open-source data products responded to exactly this pattern by relicensing away from permissive terms specifically to stop cloud vendors reselling them. Prometheus did not, and CNCF governance means no single company could.
Is it there so AWS can watch its own EC2 fleet? No. AWS already sells CloudWatch for that, and its internal fleet telemetry is proprietary and predates Prometheus. The managed service exists because customers running Kubernetes had already standardised on Prometheus, and operating it reliably at scale with long retention and high availability is genuinely hard. Their alternatives were to do that work themselves or buy it from Grafana Cloud or Datadog. Offering a managed version keeps the workload, the data and the spend inside AWS. The licence permits it; customer demand motivated it.