← All essays

Vojvodina property list

Coverage is a verdict, not a setting

A crawl that stopped early and a source with nothing left produce identical output. The pipeline has to tell them apart before it deletes anything.

The pipeline ranks every house and plot around one city, cheapest first, across portals that cross-post the same property at different prices. To do that it has to know when something has been sold or withdrawn, because a list full of properties that no longer exist is worse than no list.

The obvious rule is: if a listing was there yesterday and is not there today, it is gone. That rule is wrong, and the reason is the whole essay.

Absence is not evidence unless you looked

a run that stopped at `--limit 700` did not fail to find the 701st listing, it never looked.
README · delisting and coverage

A crawl that stops early produces exactly the same observation as a crawl that finished and found nothing: the listing is not in the results. The two mean opposite things. One is evidence of removal. The other is evidence of nothing at all, and treating it as removal quietly deletes real properties from the list.

So the rule needs two conditions, not one, and the second was missing for six sprints:

A listing is marked inactive only after **two consecutive complete runs that covered it** failed to find it.
README · delisting and coverage

Nobody is allowed to declare completeness

The fix that matters is not the rule, it is who gets to apply it. A flag an operator can set means completeness is an opinion, and it will be set to true on the day someone is in a hurry.

Nothing sets `coverage` by hand — the collector reports a `termination` and the verdict follows from it:
README · delisting and coverage

A collector does not report whether it succeeded. It reports how it stopped — pages ran out, the item limit was reached, the page limit was reached, the URL list ended, a fetch failed — and the coverage verdict is derived from that. Only one termination produces a complete verdict, and only a complete run counts as having looked.

Only `complete` is evidence that a missing listing is gone.
README · delisting and coverage

What it costs, which is the interesting half

An exhausted run that returned zero records is deliberately not counted as complete. From outside, a source that genuinely has nothing and a parser that has stopped matching produce identical output: no rows, no error, pages exhausted. Refusing to trust that case costs re-crawls, and it slows every delisting down.

It buys one thing. A single broken selector cannot quietly retire an entire source — which is the failure that would be invisible, permanent, and discovered months later by someone wondering why one portal's properties stopped appearing.

The same shape shows up elsewhere: firmware that records only that a remote attempted to transmit, and a trading interface that reports the ticks it missed.