Why every daemon should phone home — and how to make it

01The silent-failure problem

Most monitoring answers the question "is this thing up?" — it pokes an endpoint and waits for a reply. That works for things that are supposed to answer: a web server, a load balancer, an API. But a huge share of what keeps a business running never answers anything. It's scheduled and silent:

• The nightly Postgres backup that runs at 03:00 and exits.
• The cron that emails invoices on the 1st of the month.
• The worker that drains a queue, the ETL that syncs a warehouse, the cert-renewal timer, the log-rotation job, the cache warmer.

When one of these dies, nothing pages you. There's no 500, no failed health check, no red light — because the symptom isn't an error, it's an absence. The backup simply didn't happen. You find out three weeks later, when you reach for the backup and it isn't there. By industry lore this is the most expensive class of outage precisely because time-to-detect is measured in days, not seconds.

02What a heartbeat actually is

A heartbeat — also called a dead man's switch or deadman's snitch — inverts the direction of monitoring. Instead of the monitor reaching into your job, the job reaches out to the monitor on every successful run:

• The job checks in — an HTTP ping, an email, a DNS lookup — each time it runs or stays alive.
• The monitor expects that check-in on a schedule — "every night," "every 60 seconds," "by 04:10."
• Silence is the alert. Miss the window plus a grace period, and the monitor fires — because the only thing that stops the check-ins is the job dying.

It's the same logic as a railway driver's safety pedal: the system is healthy only as long as the signal keeps coming. The moment it stops, someone gets paged. With cron.watch a check-in is literally one line — curl -fsS https://cron.watch/s/<slug> — and the absence-detection, grace windows, and escalation are handled for you.

The two are complements, not rivals. Uptime checks watch the things that answer; heartbeats watch the things that act. A complete picture needs both.

03Why it matters for startups

A small team has no NOC, no follow-the-sun on-call, and no spare attention. The founder is the on-call rotation. That makes silent failures disproportionately dangerous — there's nobody whose job is to notice — and it makes heartbeats disproportionately valuable, because they convert "someone has to remember to check" into "we get told."

For a startup the calculation is simple: the engineering cost of adding a heartbeat is one line; the cost of not having one is a silent failure you find out about from the people you can least afford to disappoint.

04Why it matters for enterprises

At scale the problem changes shape. It's no longer "did the one backup run" — it's thousands of scheduled jobs across fleets, teams, and regions, each an unwatched corner where a silent failure can hide. Heartbeats turn that sprawl into something measurable and provable.

• SLAs & error budgets. A contract that says "the export lands by 06:00" needs evidence it did. Check-ins with timestamps turn a promise into a measured SLO with burn-down — not a vibe.
• Compliance & audit. Regulators and auditors ask you to prove the nightly backup, the reconciliation, the data-retention purge actually ran. A check-in history is that audit trail.
• Mean-time-to-detect. Across thousands of jobs, MTTD is the metric that bounds blast radius. Absence-detection makes a missed run a page in minutes instead of a postmortem finding.
• Fleet & quorum health. "At least 5 of 8 ingest workers must report in each minute" is a single quorum monitor, not eight brittle ones — degraded capacity pages before it becomes an outage.
• Separation of duties. The team that runs a job and the platform that verifies it are decoupled. The job just dials out; nobody needs inbound access into production to watch it.

05One daemon, one monitor, one socket heartbeat

The right granularity is one monitor per worker or daemon — not a single check for the whole box, and not one shared monitor behind a middleware that fires for every job type. Each long-lived process owns its monitor and emits its own heartbeat from inside its work loop, so an alert names the exact process that went quiet. Create the monitor once and hand the slug to that one daemon:

# one monitor for this worker → one check-in identity
curl -X POST https://api.cron.watch/v1/monitors \
  -H 'Authorization: Bearer cw_live_xxx' \
  -d '{"name":"api-worker","type":"heartbeat","schedule":"every 60s","grace":"3m"}'
# → { "checkin_url": "https://cron.watch/s/api-worker" }

The parallel-poller trap

A tempting shortcut is a sidecar timer that runs systemctl is-active && curl alongside the service. Don't. A poller that only checks is-active has no reference to the app's real work — the unit stays "active" while the daemon is deadlocked, spinning on a dead upstream, or wedged mid-batch. It proves the process hasn't exited, which is not the same as alive. Bolting on a second process to guess at the first one's health just adds a thing that can lie to you. The heartbeat has to originate from the work itself.

Socket-based liveness, the systemd way

The main app proves it's alive by sending a datagram over a socket — systemd's notify socket, a local AF_UNIX endpoint at $NOTIFY_SOCKET. With Type=notify and WatchdogSec=, the app pings WATCHDOG=1 from its real loop; miss the deadline and systemd restarts it. The same loop forwards the beat to cron.watch — one signal source driving local restart and remote alerting, with no extra process.

/etc/systemd/system/api-worker.service

[Unit]
Description=API worker
[Service]
ExecStart=/opt/api-worker
Type=notify            # app reports readiness + liveness over $NOTIFY_SOCKET
WatchdogSec=45s        # systemd restarts if no WATCHDOG=1 arrives within 45s
Restart=on-failure
NotifyAccess=main
[Install]
WantedBy=multi-user.target

…and the app, from the loop that does the work — local watchdog and remote heartbeat in the same place:

# Python — one socket beat to systemd, one to cron.watch, from the work loop
import socket, os

def sd_notify(msg):                       # local AF_UNIX datagram to systemd
    addr = os.environ.get("NOTIFY_SOCKET")
    if not addr: return
    s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
    s.connect("\0" + addr[1:] if addr[0] == "@" else addr)
    s.sendall(msg.encode())

sd_notify("READY=1")                       # primary initialization
while running:
    do_one_unit_of_work()
    sd_notify("WATCHDOG=1")                # local: systemd restarts on a miss
    cw_beat()                              # remote: cron.watch, over a cheap channel (§06)

Run-to-completion jobs are the other shape — one monitor each, signalling start and exit code instead of a continuous beat: curl …/s/nightly-backup/start; ./run.sh; curl …/s/nightly-backup/$?. Identical replicas are the one deliberate exception to one-per-process: point the pool at a single fleet monitor with a quorum rule (type=fleet, dimension=host, quorum=5) instead of one monitor per replica.

06Keep it cheap: tier your transport

An HTTPS request every few seconds from every worker is the expensive way to stay alive — a TLS handshake and a fresh TCP connection per beat, multiplied across a fleet, taxes both ends for no extra signal. A heartbeat should get cheaper the more often it fires. Tier it in three layers, each used at the frequency it's priced for:

• 1 · Initialize once, over HTTPS. At startup the worker makes a single authenticated call — create or attach the monitor, take back the slug and an ingest token. One handshake for the life of the process.
• 2 · Beat on a connectionless channel. After init, frequent liveness and intermediate progress ride a path with no per-beat handshake:

• 3 · Reconcile every 5 minutes, over HTTPS. A periodic durable POST carries the full structured state — counts, durations, last result — and squares the record with whatever the cheap channel may have dropped. Because UDP and DNS are best-effort, this 5-minute sync is what keeps the truth accurate even when individual datagrams are lost.

Why UDP for the frequent beat: no connection setup, no per-packet ACK, no retransmit, no server-side connection state — a datagram is the cheapest "I'm alive, and here's where I am" you can put on the wire. The cost of occasional loss is absorbed entirely by the 5-minute HTTPS reconcile, so you buy TCP-grade accuracy at UDP-grade cost.

# after init (HTTPS handed us the slug) — cheap UDP beat carries an intermediate marker
printf 'api-worker stage=index pct=64' | nc -u -w0 ping.cron.watch 4738   # fire-and-forget

# DNS-only box? a lookup is a UDP check-in too (no +tcp ⇒ UDP):
dig +short index-64.api-worker.dns.cron.watch

# long job with live progress? one WebSocket, no per-frame handshake:
#   wss://cron.watch/s/api-worker   →   {"stage":"index","pct":64}

07Recipes for your job framework

Same model in every stack: one monitor per worker, initialized once over HTTPS, kept alive by a cheap UDP beat that carries an intermediate marker, and reconciled by an HTTPS sync every 5 minutes (stats() is your own snapshot — processed count, queue depth, last result). Pick a language — code shows only for the tab you choose:

# celery_app.py — one monitor for THIS worker process
import socket, time, threading, urllib.request, json
from celery.signals import worker_ready, task_postrun

SLUG = "billing-worker"
udp  = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

def beat(marker=""):                       # cheap: fire-and-forget UDP
    udp.sendto(f"{SLUG} {marker}".encode(), ("ping.cron.watch", 4738))

def sync():                                # durable: full state every 5 min
    while True:
        urllib.request.urlopen(f"https://cron.watch/s/{SLUG}",
            data=json.dumps({"metrics": stats()}).encode(), timeout=10)
        time.sleep(300)

@worker_ready.connect
def _init(**_):                            # primary initialization, over HTTPS
    urllib.request.urlopen(f"https://cron.watch/s/{SLUG}/start")
    threading.Thread(target=sync, daemon=True).start()

@task_postrun.connect
def _beat(task=None, **_): beat(task.name) # intermediate marker per task

# config/initializers/cronwatch.rb — one monitor for this Sidekiq process
require "socket"; require "net/http"
SLUG = "billing-worker"
UDP  = UDPSocket.new

def beat(m = "")                         # cheap UDP beat
  UDP.send("#{SLUG} #{m}", 0, "ping.cron.watch", 4738)
end

Sidekiq.configure_server do |c|
  c.on(:startup) do                       # init (HTTPS) + 5-min sync thread
    Net::HTTP.get(URI("https://cron.watch/s/#{SLUG}/start"))
    Thread.new { loop { sleep 300
      Net::HTTP.post_form(URI("https://cron.watch/s/#{SLUG}"), stats) } }
  end
end

class BillingJob
  include Sidekiq::Job
  def perform(*args)
    do_work(*args)
    beat("done")                          # intermediate marker per run
  end
end

const slug = "billing-worker"
udp, _ := net.Dial("udp", "ping.cron.watch:4738")

func beat(m string) { fmt.Fprintf(udp, "%s %s", slug, m) } // fire-and-forget

func main() {
    http.Get("https://cron.watch/s/" + slug + "/start")        // init, over HTTPS
    go func() {                                            // 5-min HTTPS sync
        for range time.Tick(5 * time.Minute) {
            http.Post("https://cron.watch/s/"+slug, "application/json", stats())
        }
    }()
    srv.Run(asynq.HandlerFunc(func(ctx context.Context, t *asynq.Task) error {
        err := work(ctx, t)
        beat(t.Type())                                       // intermediate marker per task
        return err
    }))
}

const SLUG: &str = "billing-worker";

#[tokio::main]
async fn main() {
    reqwest::get(format!("https://cron.watch/s/{SLUG}/start")).await.ok();  // init
    let sock = tokio::net::UdpSocket::bind("0.0.0.0:0").await.unwrap();
    sock.connect("ping.cron.watch:4738").await.unwrap();

    tokio::spawn(async {                                             // 5-min HTTPS sync
        let mut t = tokio::time::interval(Duration::from_secs(300));
        loop { t.tick().await;
            reqwest::Client::new().post(format!("https://cron.watch/s/{SLUG}"))
                .json(&stats()).send().await.ok(); }
    });

    while let Some(job) = queue.next().await {
        work(job).await;
        let _ = sock.send(format!("{SLUG} done").as_bytes()).await;         // cheap UDP beat
    }
}

@Component
public class BillingWorker {
    static final String SLUG = "billing-worker";
    final HttpClient http = HttpClient.newHttpClient();
    DatagramSocket udp;  InetAddress host;

    @PostConstruct void init() throws Exception {              // primary init, over HTTPS
        http.send(HttpRequest.newBuilder(URI.create(
            "https://cron.watch/s/" + SLUG + "/start")).build(),
            HttpResponse.BodyHandlers.discarding());
        udp  = new DatagramSocket();
        host = InetAddress.getByName("ping.cron.watch");
    }

    void beat(String m) throws Exception {                     // cheap UDP beat
        byte[] b = (SLUG + " " + m).getBytes();
        udp.send(new DatagramPacket(b, b.length, host, 4738));
    }

    @Scheduled(fixedRate = 300_000)                              // 5-min HTTPS sync
    void sync() throws Exception {
        http.send(HttpRequest.newBuilder(URI.create("https://cron.watch/s/" + SLUG))
            .POST(HttpRequest.BodyPublishers.ofString(stats())).build(),
            HttpResponse.BodyHandlers.discarding());
    }
}

import { Worker } from "bullmq";
import dgram from "node:dgram";

const SLUG = "billing-worker";
const udp  = dgram.createSocket("udp4");
const beat = (m = "") => udp.send(`${SLUG} ${m}`, 4738, "ping.cron.watch"); // cheap UDP

await fetch(`https://cron.watch/s/${SLUG}/start`);            // primary init, HTTPS
setInterval(() =>                                            // 5-min HTTPS sync
  fetch(`https://cron.watch/s/${SLUG}`, { method: "POST", body: JSON.stringify(stats()) }),
  300_000);

new Worker("billing", async job => {
  await doWork(job);
  beat(job.name);                                            // intermediate marker per job
});

08AI-native: just ask your agent

cron.watch ships an MCP server, so if your editor or assistant (Claude Code, Cursor, Claude Desktop) is connected, you don't have to write any of the above by hand — describe the job and let the agent create the monitor and wire the check-in. The tools it drives: create_monitor, list_monitors, check_in, push_metric. Prompts that work today:

The same flow works from the HTTP API if you'd rather script it: POST /v1/monitors to create, then hand the returned checkin_url to the job.

09Picking an interval and grace window

The two numbers that decide signal-to-noise are the interval (how often the job checks in) and the grace (how late it's allowed to be before you're paged). Get them wrong and you either miss outages or train yourself to ignore the alerts.

• Grace ≥ interval + worst-case jitter. A 60s heartbeat with a 3m grace tolerates two missed pings before paging — so a single hiccup, GC pause, or deploy blip doesn't wake anyone.
• Match grace to the job's variance, not its average. A backup that usually finishes in 8 minutes but occasionally takes 20 needs grace past the 20, or you'll page on every slow-but-fine night.
• Tighten where it's worth it. The revenue-critical billing run earns a short grace and a loud escalation; the cache warmer can be relaxed. Don't give everything the same urgency.

10What good looks like

• Check in on success, at the end (or start + end) — never just at launch, or a job that dies mid-run looks healthy.
• Send the exit code so "ran but failed" is distinct from "ran clean" and from "never ran."
• Attach the metrics that define correct — row counts, durations, queue depth — so you catch the run that exited 0 but processed nothing.
• Route and escalate — Slack first, PagerDuty if it's still missing in 15 minutes — and mute on a schedule for known maintenance so planned work doesn't page.
• One monitor per job, one fleet monitor per pool — named clearly, tagged by team and environment, so an alert says exactly what's broken.

Do this across your daemons and the unwatched corners disappear: every scheduled thing either checks in on time or tells you the moment it doesn't.