How I Recover Stale Crawler Jobs
When a crawler worker dies mid-job, the rows it claimed sit in a started state forever. Here is the timeout, retry count, and alert that fixed that.
A worker that claims jobs and then dies leaves those jobs marked as started, and nothing in the system moves them. Without a recovery step, they sit there forever and nobody notices until the data is missing.
This is how the distributed crawler I built handled it. The full design is in the crawler case study.
How the work was claimed
Each job was a set of parcels, and a worker claimed a batch of roughly 10 to 20 at a time. The claim used a row lock with SKIP LOCKED, so two workers never took the same rows and never waited on each other.
When a worker claimed a batch, the rows moved to a started state. That state is the problem, because “started” only means a worker once began the work, not that one is still doing it.
Detecting a stale claim
The rule was simple. If a row was marked started and had gone more than 10 minutes without an update, I treated the worker as gone.
That number was chosen against how long the work really took. A batch of 10 to 20 parcels never ran for even 3 minutes, so a 10 minute gap was a safe signal and not a guess about a slow page.
What happens to the interrupted job
The recovery step set the stalled rows back to pending and added one to a retry count. Another worker then claimed them like any new job.
The work was not resumed from the middle. It was restarted from the beginning of each parcel.
Restarting works because a parcel is a small, self-contained unit. If the unit were large, I would need to checkpoint progress inside it.
Giving up on purpose
A row that failed three times was marked failed and a notification went out. Retrying forever is its own failure, because it hides a real problem behind constant activity.
Three failures in a row most likely meant something outside the worker had changed, usually a site updating its markup. That needs a person to look at it, and a notification gets one there.
What I would tell someone building this
Keep the claim, the stale timeout, the retry count, and the failure alert as separate, simple pieces. Each one is easy to reason about alone, and together they cover a worker that crashes, a parcel that cannot be read, and a site that changed.
I also would not trust a worker to report its own death. The timeout works because it only needs the database to notice that nothing has happened.
Related
For how this fits into the whole system, read the distributed crawler case study. The data pipelines and automation service covers work like this.