Skip to content
rushlabs-aiPublic

About

Keep a fine-tuned model working after its provider retires the base model: inventory, record, compare, retrain on open weights.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

tunekeep

CI PyPI License: Apache-2.0

Keep a fine-tuned model working after its provider retires the base model.

OpenAI shuts down fine-tuned gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini models on 14 January 2027, and stops new fine-tuning jobs on 6 January 2027. The named replacement, gpt-6-luna, cannot be fine-tuned. Once a model is gone, so is the only exact record of how it behaved.

tunekeep helps you get out in time:

Step Command
1. See every fine-tune you own and its days left tunekeep inventory
2. Record how each model answers, as a golden eval set tunekeep record
3. Check whether gpt-6-luna (or any model) already matches it tunekeep compare
4. Retrain an open model with LoRA from your own data tunekeep retrain
5. One HTML report: quality, speed and tokens side by side tunekeep report

Do the recording first. Everything else can happen later, but recording cannot happen after the shutdown.

Install

uv tool install tunekeep      # or: pipx install tunekeep

Python 3.10+. No dependencies: tunekeep uses only the Python standard library. (The retraining kit it writes has its own pinned requirements, installed only where you run it.)

1. Inventory

export OPENAI_API_KEY=sk-...          # a key for the org/project that owns the fine-tunes
tunekeep inventory
Fine-tuned models: 1 (plus 1 checkpoint)
New fine-tuning jobs stop on 2027-01-06 (60 days).

MODEL                                   SHUTS DOWN   DAYS LEFT  SOURCE
ft:gpt-4o-mini-2024-07-18:acme::AbC123  2027-01-14   68         api

Files: 1 saved, 1 training/validation file the API would not return.
  Keep your own copy of these; tunekeep retrains only from your original data:
  - file-train-0  train.jsonl  (job ftjob-000)

Next: record how each model behaves while it still answers, e.g.
  tunekeep record --model ft:gpt-4o-mini-2024-07-18:acme::AbC123 --prompts prompts.jsonl

tunekeep-out/inventory.json holds the full detail: jobs, hyperparameters, checkpoints and their metrics, and every file. Result files (training metrics) are downloaded. Files you uploaded for training are not: OpenAI does not return them through the API, so keep your own copy.

2. Record

tunekeep record --model ft:gpt-4o-mini-2024-07-18:acme::AbC123 --prompts prompts.jsonl
  • prompts.jsonl can be your training file as-is. The final assistant message on each line is removed before sending and kept as the reference answer. DPO lines (input.messages) work too.
  • Real production prompts make the best golden set. Use a few hundred, covering your real traffic.
  • Answers are recorded at temperature 0 by default. Use --seed for more repeatable runs.
  • Safe to stop and re-run: prompts already recorded are skipped, failed ones are retried.
  • --dry-run counts prompts and estimated input tokens without sending anything.

3. Compare

tunekeep compare --golden tunekeep-out/golden-ft_gpt-4o-mini-2024-07-18_acme_AbC123.jsonl \
  --candidate gpt-6-luna --examples train.jsonl --shots 5

Each recorded prompt goes to the candidate, optionally with a few examples from your own training file, and the answer is scored against the recorded one:

  • Tool calls: matched by function name, arguments compared field by field.
  • JSON answers: the share of leaf fields with the same value. Invalid JSON scores 0.
  • Text: 100% when equal after normalising case and spacing, otherwise word overlap (F1). Overlap is a rough signal for free text, so the report shows the biggest differences side by side.

GPT-6 Luna reasons by default, and in Chat Completions it accepts temperature and function tools only with reasoning_effort: "none". tunekeep sends none by default (--reasoning-effort auto), which is also closest to how a fine-tuned gpt-4o-mini answered. Try --reasoning-effort low to see whether reasoning helps. tunekeep then drops temperature, and refuses before sending anything if your prompts use tools.

Any OpenAI-compatible endpoint works as a candidate: --candidate-base-url http://127.0.0.1:8080/v1 for llama-server, vLLM or Ollama (no key needed on localhost, --candidate-key-env for others).

4. Retrain

tunekeep retrain --data train.jsonl          # default base: Qwen/Qwen3-1.7B, pinned revision
cd tunekeep-out/retrain && modal run modal_app.py

retrain converts your OpenAI training file (chat format, tools, legacy functions, per-message weight) and writes a small kit: your data, a config, a training script and a Modal app. It uploads nothing itself. The kit:

  • trains a LoRA adapter with loss only on assistant messages ("weight": 0 turns are skipped, as on OpenAI)
  • reuses your original job's epoch count when tunekeep-out/inventory.json has it
  • pins every package by hash, the base model by revision, and llama.cpp by commit
  • stops after a time limit (default 2 hours), which caps the cost of a run
  • exports merged weights and a q8_0 GGUF for llama-server or Ollama

Use --check to only validate and summarise the data. See the kit's own README for running it elsewhere and serving the result. Not yet supported: DPO, images and audio in training data.

Does retraining actually work?

A public proof run (examples/banking77): bank customer messages, 77 intents, JSON answers. Qwen3-1.7B went from 50.3% to 90.6% accuracy on 770 unseen messages after tunekeep retrain (3,075 examples, 8.5 minutes on one L40S), served locally from the exported GGUF and scored with tunekeep compare.

tunekeep report for the banking77 run

5. Report

tunekeep report                      # reads tunekeep-out/, writes tunekeep-out/report.html

One self-contained HTML file with deadlines, every candidate's scores, latency and token use, and the lowest-scoring answers side by side. It loads nothing from the network and runs no scripts, and every value from your data is escaped.

What tunekeep will not do

It never trains on recorded outputs. OpenAI's Services Agreement (section 3.3(e)) restricts using Output to develop competing models. Recorded answers are used only to evaluate a replacement. Retraining uses only your original training data. If that data was itself generated by an OpenAI model, it counts as Output too. Check with your own counsel. This is not legal advice.

Privacy

  • tunekeep talks only to the API base URLs you give it. No telemetry, no other network calls.
  • Outputs can contain production prompts, so the output folder is created 0700 and files 0600.
  • API keys are read from the environment and are never written to disk or printed. A candidate endpoint never receives your OpenAI key unless you name that variable with --candidate-key-env.

Other providers

  • --base-url / OPENAI_BASE_URL for OpenAI-compatible gateways.
  • --azure --base-url https://<resource>.openai.azure.com/openai/v1 with AZURE_OPENAI_API_KEY (experimental). Azure retires fine-tuned gpt-4o and gpt-4o-mini deployments on 1 October 2027, and the gpt-4.1 family on 14 October 2027.

Development

python -m unittest discover -s tests -v

The tests run against a fake OpenAI server built from the public API reference: pagination, the shutdown_date field, blocked training-file downloads, rate limits, tool calls and GPT-6 Luna's parameter rules. Both the fake server's responses and every request tunekeep sends are checked against OpenAI's official API spec (openai-openapi, pinned by commit and SHA-256, see tools/openai_spec.py), so the tests notice when either drifts from the real API. The retraining kit is tested end to end on CPU with Qwen3-0.6B. tools/lock-kit.sh regenerates the kit's hash-locked requirements. Reports from real accounts are very welcome. Please open an issue with anything that behaves differently.

Sources

Licence

Apache-2.0. Built by Rush Labs, an applied AI lab.

About

Keep a fine-tuned model working after its provider retires the base model: inventory, record, compare, retrain on open weights.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages