Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
- [Feature] Add the `OPENEDX_INIT_REINDEX_COURSES` setting (default: `true`). Set it to `false` to skip the synchronous `reindex_course --active` step of CMS init, which can make every launch very slow on instances with many courses. (by @ahmed-arb)
- [Feature] Add the `OPENEDX_REINDEX_COURSES_SCHEDULE` setting (default: `""`). When set to a cron expression, `tutor k8s start` creates a `cms-reindex-courses` CronJob that reindexes active courses on that schedule. Also add the `k8s-cronjobs` patch. `tutor k8s stop` now also deletes CronJobs created by Tutor and its plugins; `tutor k8s start` recreates them. Jobs spawned by CronJobs no longer block `tutor k8s do` commands. (by @ahmed-arb)
12 changes: 12 additions & 0 deletions docs/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,18 @@ This defines the version that will be pulled from just the Open edX platform git

By default, there are 2 `uwsgi worker processes <https://uwsgi-docs.readthedocs.io/en/latest/Options.html#processes>`__ to serve requests for the LMS and the CMS. However, each worker requires upwards of 500 Mb of RAM. You should reduce this value to 1 if your computer/server does not have enough memory.

- ``OPENEDX_INIT_REINDEX_COURSES`` (default: ``true``)

When enabled, ``tutor local/k8s launch`` and ``do init`` rebuild the courseware search index of all active courses with ``./manage.py cms reindex_course --active``. This step runs synchronously and can take a long time on instances with many courses. Set it to ``false`` to skip it; note that on-publish indexing only covers recently edited content, so you should then reindex courses on a schedule (see ``OPENEDX_REINDEX_COURSES_SCHEDULE`` below) or manually after importing courses or restoring Meilisearch data::

tutor local run cms ./manage.py cms reindex_course --active

- ``OPENEDX_REINDEX_COURSES_SCHEDULE`` (default: ``""``)

A `cron schedule expression <https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#schedule-syntax>`__, such as ``"0 1 * * *"`` for every day at 01:00 in the timezone of the Kubernetes control plane (usually UTC). When set, ``tutor k8s start`` creates a ``cms-reindex-courses`` Kubernetes CronJob that runs ``./manage.py cms reindex_course --active`` on that schedule. Combine it with ``OPENEDX_INIT_REINDEX_COURSES: false`` to move course reindexing out of ``tutor k8s launch``. When you unset it later, run ``tutor k8s stop`` (or ``kubectl delete cronjob cms-reindex-courses``) to remove the existing CronJob. The CronJob has its own pod spec: customizations that you apply to the ``cms-job`` Job, such as a private image registry or extra environment variables, must also be applied to the ``cms-reindex-courses`` CronJob, e.g. with the :patch:`k8s-override` patch. With ``tutor local``, which has no scheduler, add an equivalent entry to the host crontab instead. Use absolute paths, since cron runs with a minimal ``PATH``::

0 1 * * * /path/to/tutor local run cms ./manage.py cms reindex_course --active

- ``OPENEDX_CELERY_REDIS_DB`` (default: ``0``)
- ``OPENEDX_CACHE_REDIS_DB`` (default: ``1``)

Expand Down
9 changes: 9 additions & 0 deletions docs/reference/patches/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,15 @@ Files: ``apps/openedx/config/cms.env.yml``, ``apps/openedx/config/lms.env.yml``

File: ``dev/docker-compose.jobs.yml``

.. patch:: k8s-cronjobs

``k8s-cronjobs``
================

File: ``k8s/cronjobs.yml``

Add Kubernetes ``CronJob`` resources here. Like other Tutor resources, they are created by ``tutor k8s start`` and deleted by ``tutor k8s stop``. Jobs spawned by a CronJob do not block ``tutor k8s do`` commands.

.. patch:: k8s-deployments

``k8s-deployments``
Expand Down
65 changes: 65 additions & 0 deletions tests/commands/test_k8s.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,75 @@
import unittest
from types import SimpleNamespace
from unittest.mock import patch

from tests.helpers import temporary_root
from tutor import config as tutor_config
from tutor import env, serialize
from tutor.commands import k8s
from tutor.types import Config

from .base import TestCommandMixin


def _job(name: str, active: int, owner_kind: str = "") -> SimpleNamespace:
owners = [SimpleNamespace(kind=owner_kind)] if owner_kind else None
return SimpleNamespace(
metadata=SimpleNamespace(name=name, owner_references=owners),
status=SimpleNamespace(active=active),
)


def _config() -> Config:
config: Config = {}
tutor_config.update_with_base(config)
tutor_config.update_with_defaults(config)
tutor_config.render_full(config)
return config


class K8sTests(unittest.TestCase, TestCommandMixin):
def test_k8s_help(self) -> None:
result = self.invoke(["k8s", "--help"])
self.assertIsNone(result.exception)
self.assertEqual(0, result.exit_code)

def test_active_job_names_ignores_cronjob_jobs(self) -> None:
jobs = [
_job("lms-job-20261113", active=1),
_job("cms-reindex-courses-29387", active=1, owner_kind="CronJob"),
_job("cms-job-20261112", active=0),
]
with temporary_root() as root:
runner = k8s.K8sTaskRunner(root, _config())
with patch.object(k8s.K8sClients, "instance") as clients:
clients.return_value.batch_api.list_namespaced_job.return_value = (
SimpleNamespace(items=jobs)
)
self.assertEqual(["lms-job-20261113"], runner.active_job_names())

def test_reindex_courses_cronjob(self) -> None:
config = _config()
self.assertNotIn(
"k8s/cronjobs.yml", env.render_file(config, "kustomization.yml")
)

config["OPENEDX_REINDEX_COURSES_SCHEDULE"] = '0 1 * * * "quoted"'
self.assertIn("k8s/cronjobs.yml", env.render_file(config, "kustomization.yml"))
rendered = env.render_file(config, "k8s", "cronjobs.yml")
assert isinstance(rendered, str)
manifests = list(serialize.load_all(rendered))
cronjobs = [m for m in manifests if m and m["kind"] == "CronJob"]
self.assertEqual(1, len(cronjobs))
self.assertEqual('0 1 * * * "quoted"', cronjobs[0]["spec"]["schedule"])

def test_init_reindex_courses(self) -> None:
config = _config()
self.assertIn(
"reindex_course --active",
env.render_file(config, "jobs", "init", "cms.sh"),
)
config["OPENEDX_INIT_REINDEX_COURSES"] = False
self.assertNotIn(
"reindex_course --active",
env.render_file(config, "jobs", "init", "cms.sh"),
)
20 changes: 18 additions & 2 deletions tutor/commands/k8s.py
Original file line number Diff line number Diff line change
Expand Up @@ -213,10 +213,20 @@ def active_job_names(self) -> List[str]:
k8s_namespace(self.config),
label_selector="app.kubernetes.io/managed-by=tutor",
).items
if job.status.active
if job.status.active and not is_owned_by_cronjob(job)
]


def is_owned_by_cronjob(job: Any) -> bool:
"""
Jobs spawned by a CronJob inherit Tutor's labels, but they should not block
`do` commands: a scheduled task may legitimately run for hours.
"""
return any(
owner.kind == "CronJob" for owner in (job.metadata.owner_references or [])
)


class K8sContext(BaseTaskContext):
def job_runner(self, config: Config) -> K8sTaskRunner:
return K8sTaskRunner(self.root, config)
Expand Down Expand Up @@ -359,7 +369,13 @@ def delete_resources(

The load balancer is never deleted.
"""
resources = resources or ["deployments", "services", "configmaps", "jobs"]
resources = resources or [
"deployments",
"services",
"configmaps",
"jobs",
"cronjobs",
]
not_lb_selector = "app.kubernetes.io/component!=loadbalancer"
name_selector = [f"app.kubernetes.io/name={name}"] if name else []
utils.kubectl(
Expand Down
2 changes: 2 additions & 0 deletions tutor/templates/config/defaults.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,11 @@ OPENEDX_AWS_SECRET_ACCESS_KEY: ""
OPENEDX_CACHE_REDIS_DB: 1
OPENEDX_CELERY_REDIS_DB: 0
OPENEDX_CMS_UWSGI_WORKERS: 2
OPENEDX_INIT_REINDEX_COURSES: true
OPENEDX_LMS_UWSGI_WORKERS: 2
OPENEDX_MYSQL_DATABASE: "openedx"
OPENEDX_MYSQL_USERNAME: "openedx"
OPENEDX_REINDEX_COURSES_SCHEDULE: ""
# the common version will be automatically set to "master" in the main branch
OPENEDX_COMMON_VERSION: "release/verawood.2"
OPENEDX_EXTRA_PIP_REQUIREMENTS: []
Expand Down
4 changes: 4 additions & 0 deletions tutor/templates/jobs/init/cms.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,12 @@ fi
# this enqueues a Celery task that cms-worker consumes to populate the index
# incrementally in the background.
./manage.py cms reindex_studio
{% if OPENEDX_INIT_REINDEX_COURSES %}
# Create the courseware content index
./manage.py cms reindex_course --active
{% else %}
echo "Skipping courseware content reindex (OPENEDX_INIT_REINDEX_COURSES=false)"
{% endif %}

# Load default policies for the Open edX Authorization framework
# Check if openedx-authz package is installed if not skip loading policies and exit
Expand Down
48 changes: 48 additions & 0 deletions tutor/templates/k8s/cronjobs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
{% if OPENEDX_REINDEX_COURSES_SCHEDULE %}
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: cms-reindex-courses
labels:
app.kubernetes.io/component: cronjob
app.kubernetes.io/name: cms-reindex-courses
spec:
schedule: {{ OPENEDX_REINDEX_COURSES_SCHEDULE|tojson }}
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: cms
image: {{ DOCKER_IMAGE_OPENEDX }}
args: ["./manage.py", "cms", "reindex_course", "--active"]
env:
- name: SERVICE_VARIANT
value: cms
- name: DJANGO_SETTINGS_MODULE
value: cms.envs.tutor.production
volumeMounts:
- mountPath: /openedx/edx-platform/lms/envs/tutor/
name: settings-lms
- mountPath: /openedx/edx-platform/cms/envs/tutor/
name: settings-cms
- mountPath: /openedx/config
name: config
volumes:
- name: settings-lms
configMap:
name: openedx-settings-lms
- name: settings-cms
configMap:
name: openedx-settings-cms
- name: config
configMap:
name: openedx-config
{% endif %}
{{ patch("k8s-cronjobs") }}
3 changes: 3 additions & 0 deletions tutor/templates/kustomization.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- k8s/namespace.yml
{%- if OPENEDX_REINDEX_COURSES_SCHEDULE or patch("k8s-cronjobs") %}
- k8s/cronjobs.yml
{%- endif %}
- k8s/deployments.yml
- k8s/jobs.yml
- k8s/services.yml
Expand Down
Loading