When Can Pre-Trained Policies Be Reused? Inference-Time Policy Composition from Fixed Policy Libraries
CIKM 2026 Short Paper Track
Ihor Vitenko1, Noha Ibrahim2, Sihem Amer-Yahia3
1National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute", 2Grenoble INP, Université Grenoble Alpes, 3CNRS, Université Grenoble Alpes
The project uses uv for environment management.
make syncThat creates .venv/ from pyproject.toml and uv.lock. The repository pins Python 3.12 in .python-version.
For manual runs, use:
uv run python <script>.py ...The experiment entrypoints are grouped by workflow:
tabular/: tabular policy preparation and evaluationdqn/: DQN policy preparation and evaluationppo/: PPO policy preparation and evaluation
The tabular workflow now exposes four explicit SARSA policy libraries:
states_8_0states_8_99states_16_0states_16_99
They correspond to:
- 8x8 with
gamma=0 - 8x8 with
gamma=0.99 - 16x16 with
gamma=0 - 16x16 with
gamma=0.99
List all commands:
make helpTrain the four SARSA libraries:
make states-8-0
make states-8-99
make states-16-0
make states-16-99Build pi2vec assets:
make prep-8-0
make prep-8-99
make prep-16-0
make prep-16-99Run the composition experiments:
make exp-8-0
make exp-8-99
make exp-16-0
make exp-16-99Generate comparison plots:
make plots-8-0
make plots-8-99
make plots-16-0
make plots-16-99Run the hybrid top-k sweeps:
make sweep-8-0
make sweep-8-99
make sweep-16-0
make sweep-16-99Run full pipelines:
make repro-8-0
make repro-8-99
make repro-16-0
make repro-16-99
make repro-allFor any one configuration:
- Run
make sync. - Train the library with the matching
make states-*target. - Build pi2vec assets with the matching
make prep-*target. - Run the experiment with the matching
make exp-*target. - Generate plots with the matching
make plots-*target.
The tabular entrypoints used by the Makefile are:
tabular/full_experiment.pytabular/pi2vec_preparation.pytabular/targeted_direct_eval.pytabular/hybrid_direct_eval.py
The DQN and PPO workflows are also present under dqn/ and ppo/.
The deep-RL commands are available through the Makefile as well.
Train the 8x8 DQN library with the settings documented in dqn_train.md:
make dqn-train-8Train the 16x16 DQN library:
make dqn-train-16Build the DQN pi2vec assets:
make dqn-prep-8make dqn-prep-16Run the DQN composition experiment:
make dqn-exp-8make dqn-exp-16Run the full DQN workflow:
make dqn-repro-8
make dqn-repro-16These commands use:
- training script:
policy_reusability/data_generation/deeprl/train_dqn.py - preparation script:
dqn/pi2vec_preparation.py - experiment script:
dqn/full_experiment.py
Train the 8x8 PPO library with the settings documented in ppo_train.md:
make ppo-train-8Build the PPO pi2vec assets:
make ppo-prep-8Run the PPO composition experiment:
make ppo-exp-8Run the full PPO workflow:
make ppo-repro-8These commands use:
- training script:
policy_reusability/data_generation/deeprl/train_ppo.py - preparation script:
ppo/pi2vec_preparation.py - experiment script:
ppo/full_experiment.py
- If dependency metadata changes, run
make lockand thenmake sync. - If artifacts are missing for one configuration, rerun the matching
prep-*orexp-*target.
If you use this code, please cite the CIKM 2026 paper:
@inproceedings{vitenko2026mosaic,
title = {When Can Pre-Trained Policies Be Reused? Inference-Time Policy Composition from Fixed Policy Libraries},
author = {Vitenko, Ihor and Ibrahim, Noha and Amer-Yahia, Sihem},
booktitle = {Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM)},
year = {2026}
}An extended version is available as a preprint (arXiv:2604.20174):
@article{vitenko2026lever,
title = {LEVER: Inference-Time Policy Reuse under Support Constraints},
author = {Vitenko, Ihor and Ibrahim, Noha and Amer-Yahia, Sihem},
journal = {arXiv preprint arXiv:2604.20174},
year = {2026}
}