Skip to content

Worker Runtime Troubleshooting

Why are messages stale although the tenant is active?

Active tenant status alone is not enough. Check this full chain:

  1. tenant enabled = true
  2. jobs.messages.config.enabled = true
  3. package dependency already satisfied
  4. last_run_hot and last_run_cold are not too old
  5. lock acquisition is not failing repeatedly

Typical stale-data patterns

Symptom Likely cause
messages stale, packages fresh messages loop disabled, delayed, or lock-blocked
alerts stale, messages fresh alerts loop lagging behind
payloads stale, messages present payload loop disabled or blocked
everything stale tenant disabled, upstream connectivity problem, or broad worker issue

Important runtime keys to inspect

Key Why it matters
last_run generic loop recency
last_run_hot most recent fast message sync
last_run_cold most recent slower history sync
last_seen_completed alert logic progress over completed messages
active_since_completed continuity marker for message alert processing

Common dependency trap

Multiple loops explicitly wait for package completion first:

  • payloads
  • messages
  • alerts

So if packages are not considered done, later runtime layers may all appear stale at once.

Common lock symptoms

When the code cannot acquire a lock, the loop may skip a dispatch cycle. Repeated lock contention can make data look randomly delayed even though the system is technically alive.

Typical examples in current code:

  • messages hot lock
  • messages cold lock
  • alerts lock
  • archive lock
  • package lock

Why is archive data missing?

Archive scheduling is plan-based rather than frequent-interval based.

Check:

  • jobs.archive.config.enabled
  • plan_run
  • archive_older_than
  • delete_older_than
  • whether current time has passed the planned daily execution time

Why can alerts lag behind message updates?

Alerting is downstream of both package dependency and message or artifact freshness. It can therefore lag even when the popup already shows newer message rows.

Use:

Quick diagnosis ladder

  1. packages fresh?
  2. messages hot fresh?
  3. messages cold fresh?
  4. payloads fresh?
  5. alerts fresh?
  6. UI still stale after all of that?

If steps 1 through 5 are healthy, the next suspicion is UI caching or view-specific query behavior rather than runtime collection.