Skip to content

Add health checks and readiness/liveness probes for core services #1553

Open

Description

@ahmed-arb

None of the templates in tutor/templates/local, dev or k8s define a Docker Compose healthcheck or a Kubernetes readinessProbe / livenessProbe / startupProbe.

Consequences:

  • On Kubernetes, LMS/CMS pods receive traffic as soon as the container starts, while Django is still booting (10-20s), so rolling deploys briefly return 502s.
  • A hung process is never restarted automatically.
  • With Compose, depends_on can only wait for "started", not "healthy".

Proposal:

  • LMS/CMS: a readinessProbe (and Compose healthcheck) on GET /heartbeat, which both services already expose. A startupProbe with a generous threshold covers slow cold boots.
  • Keep livenessProbe cheap, e.g. a TCP check on port 8000, because /heartbeat queries MySQL/Mongo/cache, and a backend outage should not restart every pod.
  • MySQL, Redis, Mongo and Meilisearch: their standard health commands (mysqladmin ping, redis-cli ping, ...), which would also let Compose depends_on: condition: service_healthy replace some of the wait loops in init jobs.
  • Expose the probe blocks through patches or config so operators can tune timings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    AIIssue drafted with AI assistance

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions