Blog / 137 Million Metrics and Counting: Tamin…
Every monitoring agent in our fleet reports CPU, RAM, disk, network and service health every 5 seconds. Do that across 32+ servers for a few years and the numbers get serious fast. Today KIFAA’s metrics database holds 137 million rows — and the queries that power the dashboards still return in milliseconds.
That did not happen by accident. It took one architecture decision and one humbling bug fix. Here is what we learned.
When we designed KIFAA, the obvious choice for time-series data was InfluxDB — it is the default answer to “where do metrics live?”. We rejected it for a boring but important reason: we already run PostgreSQL.
TimescaleDB is an extension that turns a normal PostgreSQL table into a hypertable — the same SQL, the same tools, the same backups, but with automatic partitioning by time. That means:
The trade-off is that you are responsible for tuning. InfluxDB hides the details; TimescaleDB hands you the engine and expects you to know how to drive it. That is exactly the kind of trade we like — we understand every part of the stack.
Our first “latest value per server” query looked innocent enough. Something like: for every metric type and every host, give me the most recent reading. The natural SQL is a DISTINCT ON across the whole table:
SELECT DISTINCT ON (host_id, metric) *
FROM metrics
ORDER BY host_id, metric, ts DESC;
With 1.7 million rows scanned, that query hung for five minutes. Every dashboard that used it was effectively broken. We had two options: throw a bigger machine at it, or understand the data.
We chose the second.
The insight is almost embarrassingly simple: the latest value only needs the most recent data. Nobody cares what the CPU was doing six months ago when they want to know what it is doing now.
We rewrote the query to narrow the time window to the last 2 hours and aggregate per host and metric:
SELECT host_id, metric,
(array_agg(value ORDER BY ts DESC))[1] AS latest
FROM metrics
WHERE ts > now() - INTERVAL '2 hours'
GROUP BY host_id, metric;
Because TimescaleDB partitions data into chunks by time, PostgreSQL only touches the one or two chunks that cover the last two hours — not 1.7 million rows across the whole history.
The result: 150 milliseconds. A 2,000× speedup from a query rewrite, not a hardware upgrade.
One fix does not make a system fast forever. Three practices keep KIFAA quick as it grows:
(host_id, metric, ts) rather than on columns nobody filters by. Every index is a write penalty; we keep only the ones the queries actually use.
No performance post would be honest without mentioning the bug that made us question everything. When we ran multiple uvicorn workers on startup, they all executed CREATE TABLE IF NOT EXISTS at the same moment — and PostgreSQL threw pg_type duplicate-key errors.
The fix was defensive and boring:
try:
await conn.execute("CREATE TABLE IF NOT EXISTS ...")
except Exception:
await conn.rollback() # another worker won the race
It is not elegant. It is correct, and in production, correct beats elegant every day.
137 million rows is not a ceiling — it is a warm-up. The platform that handles it is the same one we deploy for clients, and the same discipline applies whether you have 3 servers or 300.
Want to see a 137M-row monitoring platform in action? Request a KIFAA demo.
Tell us what you are trying to run — we will tell you the honest way to run it. Free consultation, no obligation.
Talk to the team