Skip to content

Ollama compatibility: weekly check of new releases, wider tested range - #3

Merged
Vayz0001 merged 2 commits into
mainfrom
ollama-range
Nov 9, 2026
Merged

Vayz0001 merged 2 commits into
mainfrom
ollama-range

Conversation

@Vayz0001

@Vayz0001 Vayz0001 commented Nov 9, 2026

Copy link
Copy Markdown
Member

Keeps --use-ollama-engine working as Ollama releases new versions. Today rerankd accepts only Ollama 0.40.13, but Ollama is already at 0.40.15.

What's in this PR

  • Weekly CI check (ollama-compat.yml + integration/ollama_compat.py, standard library only). It picks the newest Ollama release that is at least 7 days old and not a pre-release, and requires GitHub's digest of the upload to match Ollama's own sha256sum.txt. It then installs that release, serves Qwen3-Reranker-0.6B through Ollama's bundled llama-server, and compares rankings with the Hugging Face reference (integration/reference-qwen3.json). A pass doesn't change anything by itself: a maintainer raises testedOllama after a full integration run. It also runs on this PR.
  • Harness can test any Ollama version (OLLAMA_RELEASE=vX=sha256). Release files are now downloaded and verified locally, then uploaded to Modal. GitHub's release downloads kept returning 504s to Modal's builders, but never from here.

Results so far

  • Ollama 0.40.11: every check passes. That covers all endpoints, streaming, rerankd check, the bundled engine (same order as the llama.cpp release, score difference 0.0000), and both models matching the HF originals.
  • Ollama 0.40.12: the full run is in progress. If it passes, a follow-up commit widens the accepted range to 0.40.11–0.40.13.
  • 0.40.14 and 0.40.15 were released on 3 November and are under the 7-day minimum age. The weekly check tests them once they're old enough.

🤖 Generated with Claude Code

Vayz0001 and others added 2 commits November 9, 2026 06:30
… Ollama version

- .github/workflows/ollama-compat.yml + integration/ollama_compat.py (stdlib): weekly, picks the
  newest Ollama release that is at least 7 days old, requires GitHub's asset digest to match
  Ollama's sha256sum.txt, then checks --use-ollama-engine rankings against the HF reference
  (integration/reference-qwen3.json).
- Harness: OLLAMA_RELEASE=vX=sha selects the Ollama version; release files are downloaded and
  verified locally, then uploaded, so runs no longer depend on Modal reaching GitHub.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ollama 0.40.11 and 0.40.12 each passed the full integration run on Modal: every endpoint,
streaming, rerankd check, the bundled engine ranking like the llama.cpp release (score
difference 0.0000), and both rerankers matching the Hugging Face models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Vayz0001
Vayz0001 merged commit 0eb0b83 into main Nov 9, 2026
15 checks passed
@Vayz0001
Vayz0001 deleted the ollama-range branch November 9, 2026 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant