diff --git a/changelog.d/20261112_160000_ahmed.khalid_skip_reindex_course.md b/changelog.d/20261112_160000_ahmed.khalid_skip_reindex_course.md new file mode 100644 index 0000000000..f1b2c658c5 --- /dev/null +++ b/changelog.d/20261112_160000_ahmed.khalid_skip_reindex_course.md @@ -0,0 +1,2 @@ +- [Feature] Add the `OPENEDX_INIT_REINDEX_COURSES` setting (default: `true`). Set it to `false` to skip the synchronous `reindex_course --active` step of CMS init, which can make every launch very slow on instances with many courses. (by @ahmed-arb) +- [Feature] Add the `OPENEDX_REINDEX_COURSES_SCHEDULE` setting (default: `""`). When set to a cron expression, `tutor k8s start` creates a `cms-reindex-courses` CronJob that reindexes active courses on that schedule. Also add the `k8s-cronjobs` patch. `tutor k8s stop` now also deletes CronJobs created by Tutor and its plugins; `tutor k8s start` recreates them. Jobs spawned by CronJobs no longer block `tutor k8s do` commands. (by @ahmed-arb) diff --git a/docs/configuration.rst b/docs/configuration.rst index b82c038108..a76a40ed3a 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -145,6 +145,18 @@ This defines the version that will be pulled from just the Open edX platform git By default, there are 2 `uwsgi worker processes `__ to serve requests for the LMS and the CMS. However, each worker requires upwards of 500 Mb of RAM. You should reduce this value to 1 if your computer/server does not have enough memory. +- ``OPENEDX_INIT_REINDEX_COURSES`` (default: ``true``) + +When enabled, ``tutor local/k8s launch`` and ``do init`` rebuild the courseware search index of all active courses with ``./manage.py cms reindex_course --active``. This step runs synchronously and can take a long time on instances with many courses. Set it to ``false`` to skip it; note that on-publish indexing only covers recently edited content, so you should then reindex courses on a schedule (see ``OPENEDX_REINDEX_COURSES_SCHEDULE`` below) or manually after importing courses or restoring Meilisearch data:: + + tutor local run cms ./manage.py cms reindex_course --active + +- ``OPENEDX_REINDEX_COURSES_SCHEDULE`` (default: ``""``) + +A `cron schedule expression `__, such as ``"0 1 * * *"`` for every day at 01:00 in the timezone of the Kubernetes control plane (usually UTC). When set, ``tutor k8s start`` creates a ``cms-reindex-courses`` Kubernetes CronJob that runs ``./manage.py cms reindex_course --active`` on that schedule. Combine it with ``OPENEDX_INIT_REINDEX_COURSES: false`` to move course reindexing out of ``tutor k8s launch``. When you unset it later, run ``tutor k8s stop`` (or ``kubectl delete cronjob cms-reindex-courses``) to remove the existing CronJob. The CronJob has its own pod spec: customizations that you apply to the ``cms-job`` Job, such as a private image registry or extra environment variables, must also be applied to the ``cms-reindex-courses`` CronJob, e.g. with the :patch:`k8s-override` patch. With ``tutor local``, which has no scheduler, add an equivalent entry to the host crontab instead. Use absolute paths, since cron runs with a minimal ``PATH``:: + + 0 1 * * * /path/to/tutor local run cms ./manage.py cms reindex_course --active + - ``OPENEDX_CELERY_REDIS_DB`` (default: ``0``) - ``OPENEDX_CACHE_REDIS_DB`` (default: ``1``) diff --git a/docs/reference/patches/index.rst b/docs/reference/patches/index.rst index 06ad5985cf..a4bf628a44 100644 --- a/docs/reference/patches/index.rst +++ b/docs/reference/patches/index.rst @@ -87,6 +87,15 @@ Files: ``apps/openedx/config/cms.env.yml``, ``apps/openedx/config/lms.env.yml`` File: ``dev/docker-compose.jobs.yml`` +.. patch:: k8s-cronjobs + +``k8s-cronjobs`` +================ + +File: ``k8s/cronjobs.yml`` + +Add Kubernetes ``CronJob`` resources here. Like other Tutor resources, they are created by ``tutor k8s start`` and deleted by ``tutor k8s stop``. Jobs spawned by a CronJob do not block ``tutor k8s do`` commands. + .. patch:: k8s-deployments ``k8s-deployments`` diff --git a/tests/commands/test_k8s.py b/tests/commands/test_k8s.py index f8513fd49e..16b79a2262 100644 --- a/tests/commands/test_k8s.py +++ b/tests/commands/test_k8s.py @@ -1,10 +1,75 @@ import unittest +from types import SimpleNamespace +from unittest.mock import patch + +from tests.helpers import temporary_root +from tutor import config as tutor_config +from tutor import env, serialize +from tutor.commands import k8s +from tutor.types import Config from .base import TestCommandMixin +def _job(name: str, active: int, owner_kind: str = "") -> SimpleNamespace: + owners = [SimpleNamespace(kind=owner_kind)] if owner_kind else None + return SimpleNamespace( + metadata=SimpleNamespace(name=name, owner_references=owners), + status=SimpleNamespace(active=active), + ) + + +def _config() -> Config: + config: Config = {} + tutor_config.update_with_base(config) + tutor_config.update_with_defaults(config) + tutor_config.render_full(config) + return config + + class K8sTests(unittest.TestCase, TestCommandMixin): def test_k8s_help(self) -> None: result = self.invoke(["k8s", "--help"]) self.assertIsNone(result.exception) self.assertEqual(0, result.exit_code) + + def test_active_job_names_ignores_cronjob_jobs(self) -> None: + jobs = [ + _job("lms-job-20261113", active=1), + _job("cms-reindex-courses-29387", active=1, owner_kind="CronJob"), + _job("cms-job-20261112", active=0), + ] + with temporary_root() as root: + runner = k8s.K8sTaskRunner(root, _config()) + with patch.object(k8s.K8sClients, "instance") as clients: + clients.return_value.batch_api.list_namespaced_job.return_value = ( + SimpleNamespace(items=jobs) + ) + self.assertEqual(["lms-job-20261113"], runner.active_job_names()) + + def test_reindex_courses_cronjob(self) -> None: + config = _config() + self.assertNotIn( + "k8s/cronjobs.yml", env.render_file(config, "kustomization.yml") + ) + + config["OPENEDX_REINDEX_COURSES_SCHEDULE"] = '0 1 * * * "quoted"' + self.assertIn("k8s/cronjobs.yml", env.render_file(config, "kustomization.yml")) + rendered = env.render_file(config, "k8s", "cronjobs.yml") + assert isinstance(rendered, str) + manifests = list(serialize.load_all(rendered)) + cronjobs = [m for m in manifests if m and m["kind"] == "CronJob"] + self.assertEqual(1, len(cronjobs)) + self.assertEqual('0 1 * * * "quoted"', cronjobs[0]["spec"]["schedule"]) + + def test_init_reindex_courses(self) -> None: + config = _config() + self.assertIn( + "reindex_course --active", + env.render_file(config, "jobs", "init", "cms.sh"), + ) + config["OPENEDX_INIT_REINDEX_COURSES"] = False + self.assertNotIn( + "reindex_course --active", + env.render_file(config, "jobs", "init", "cms.sh"), + ) diff --git a/tutor/commands/k8s.py b/tutor/commands/k8s.py index 4cdc246beb..2e4f388c8e 100644 --- a/tutor/commands/k8s.py +++ b/tutor/commands/k8s.py @@ -213,10 +213,20 @@ def active_job_names(self) -> List[str]: k8s_namespace(self.config), label_selector="app.kubernetes.io/managed-by=tutor", ).items - if job.status.active + if job.status.active and not is_owned_by_cronjob(job) ] +def is_owned_by_cronjob(job: Any) -> bool: + """ + Jobs spawned by a CronJob inherit Tutor's labels, but they should not block + `do` commands: a scheduled task may legitimately run for hours. + """ + return any( + owner.kind == "CronJob" for owner in (job.metadata.owner_references or []) + ) + + class K8sContext(BaseTaskContext): def job_runner(self, config: Config) -> K8sTaskRunner: return K8sTaskRunner(self.root, config) @@ -359,7 +369,13 @@ def delete_resources( The load balancer is never deleted. """ - resources = resources or ["deployments", "services", "configmaps", "jobs"] + resources = resources or [ + "deployments", + "services", + "configmaps", + "jobs", + "cronjobs", + ] not_lb_selector = "app.kubernetes.io/component!=loadbalancer" name_selector = [f"app.kubernetes.io/name={name}"] if name else [] utils.kubectl( diff --git a/tutor/templates/config/defaults.yml b/tutor/templates/config/defaults.yml index 901e64b41c..9a5b1c8800 100644 --- a/tutor/templates/config/defaults.yml +++ b/tutor/templates/config/defaults.yml @@ -57,9 +57,11 @@ OPENEDX_AWS_SECRET_ACCESS_KEY: "" OPENEDX_CACHE_REDIS_DB: 1 OPENEDX_CELERY_REDIS_DB: 0 OPENEDX_CMS_UWSGI_WORKERS: 2 +OPENEDX_INIT_REINDEX_COURSES: true OPENEDX_LMS_UWSGI_WORKERS: 2 OPENEDX_MYSQL_DATABASE: "openedx" OPENEDX_MYSQL_USERNAME: "openedx" +OPENEDX_REINDEX_COURSES_SCHEDULE: "" # the common version will be automatically set to "master" in the main branch OPENEDX_COMMON_VERSION: "release/verawood.2" OPENEDX_EXTRA_PIP_REQUIREMENTS: [] diff --git a/tutor/templates/jobs/init/cms.sh b/tutor/templates/jobs/init/cms.sh index a623f9ccbb..2a4120e48b 100644 --- a/tutor/templates/jobs/init/cms.sh +++ b/tutor/templates/jobs/init/cms.sh @@ -18,8 +18,12 @@ fi # this enqueues a Celery task that cms-worker consumes to populate the index # incrementally in the background. ./manage.py cms reindex_studio +{% if OPENEDX_INIT_REINDEX_COURSES %} # Create the courseware content index ./manage.py cms reindex_course --active +{% else %} +echo "Skipping courseware content reindex (OPENEDX_INIT_REINDEX_COURSES=false)" +{% endif %} # Load default policies for the Open edX Authorization framework # Check if openedx-authz package is installed if not skip loading policies and exit diff --git a/tutor/templates/k8s/cronjobs.yml b/tutor/templates/k8s/cronjobs.yml new file mode 100644 index 0000000000..ca8eb55ce0 --- /dev/null +++ b/tutor/templates/k8s/cronjobs.yml @@ -0,0 +1,48 @@ +{% if OPENEDX_REINDEX_COURSES_SCHEDULE %} +--- +apiVersion: batch/v1 +kind: CronJob +metadata: + name: cms-reindex-courses + labels: + app.kubernetes.io/component: cronjob + app.kubernetes.io/name: cms-reindex-courses +spec: + schedule: {{ OPENEDX_REINDEX_COURSES_SCHEDULE|tojson }} + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 1 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 0 + template: + spec: + restartPolicy: Never + containers: + - name: cms + image: {{ DOCKER_IMAGE_OPENEDX }} + args: ["./manage.py", "cms", "reindex_course", "--active"] + env: + - name: SERVICE_VARIANT + value: cms + - name: DJANGO_SETTINGS_MODULE + value: cms.envs.tutor.production + volumeMounts: + - mountPath: /openedx/edx-platform/lms/envs/tutor/ + name: settings-lms + - mountPath: /openedx/edx-platform/cms/envs/tutor/ + name: settings-cms + - mountPath: /openedx/config + name: config + volumes: + - name: settings-lms + configMap: + name: openedx-settings-lms + - name: settings-cms + configMap: + name: openedx-settings-cms + - name: config + configMap: + name: openedx-config +{% endif %} +{{ patch("k8s-cronjobs") }} diff --git a/tutor/templates/kustomization.yml b/tutor/templates/kustomization.yml index 6a7722fd20..5808e80fc8 100644 --- a/tutor/templates/kustomization.yml +++ b/tutor/templates/kustomization.yml @@ -3,6 +3,9 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - k8s/namespace.yml +{%- if OPENEDX_REINDEX_COURSES_SCHEDULE or patch("k8s-cronjobs") %} +- k8s/cronjobs.yml +{%- endif %} - k8s/deployments.yml - k8s/jobs.yml - k8s/services.yml