Optimizer

Portfolio optimization in one uv workspace: a composable skfolio / scikit-learn optimization library, a scheduled market-data ingestion daemon that keeps PostgreSQL fresh, a shared database layer, and an agentic fund where an LLM chooses, the optimizer computes, and paper trades wait behind human approval gates — four packages, one shared virtual environment.

---

What's in here

optimizer is not a single program — it is a workspace of four cooperating Python packages that share one virtual environment but keep strict import boundaries:

You want to…	Use	Needs a database / Docker?
Build and tune portfolio-optimization pipelines in your own code	the optimizer library (portopt-core)	No — pure Python (numpy/pandas/skfolio)
Keep a PostgreSQL store of market / fundamental / macro data fresh on a schedule	the portopt ingestion daemon	Yes — PostgreSQL (via Docker)
Share models, repositories and migrations across packages	the portopt-db layer	Yes — it is the DB layer
Run an LLM-driven fund manager that decides and lets the optimizer compute	the fund bridge (portopt-fund)	Yes — PostgreSQL + a chat-model endpoint (DeepSeek on Ollama Cloud by default; any of ten providers via LLM_PROVIDER)

The library installs and runs with zero Docker/DB dependency. Docker + PostgreSQL are required only for the ingestion daemon and the fund bridge, which persist data. If all you want is optimization, install portopt-core and skip the rest.

---

Architecture

Four workspace members, one shared venv (uv sync resolves them together):

Directory	Dist name	Import name	Role
packages/portopt-core/ (declared from the repo-root pyproject.toml)	portopt-core	optimizer	Pure-Python optimization library (DB-agnostic, sklearn/skfolio-based)
ingestion/	portopt	app	Headless ingestion daemon (APScheduler in-process, no HTTP API) + the portopt CLI
packages/portopt-db/	portopt-db	portopt_db	Shared SQLAlchemy Base + models + repositories + DatabaseManager + the single Alembic tree
fund/	portopt-fund	fund	deepagents/langgraph bridge: the LLM chooses, the optimizer computes

Import boundaries (enforced by static import-scan tests)

Because there is one shared venv, install-time isolation does not exist — the boundaries below are guarded by source-scan hygiene tests, not by what's installed:

• ingestion/ and portopt-db/ do not import optimizer.
• portopt-db/ carries no sklearn/skfolio stack.
• optimizer (portopt-core) does not import portopt_db — the library stays DB-free.
• fund/ is the only member allowed to import both optimizer and portopt_db; deepagents/langgraph live only in fund/.

The ingestion daemon does ship scikit-learn — not for optimization, but because yfinance's price-repair path (repair=True) imports sklearn.cluster.DBSCAN. Without it ~22% of tickers return empty history and get dropped. It is a data-layer dependency there.

The optimization library

optimizer is a set of composable primitives, not a fixed end-to-end runner. There is no run_full_pipeline; opinionated, DB-connected orchestration lives in fund/. Every module follows the same convention: a frozen @dataclass config + a factory function + str, Enum types. Transformers are sklearn BaseEstimator + TransformerMixin and compose in sklearn.pipeline.Pipeline.

Module flow:

prices → preprocessing → pre_selection → moments → views
       → optimization → validation → tuning → rebalancing

Plus the cross-cutting modules factors/, synthetic/, scoring/, universe/, distance/, cluster/, uncertainty_set/, linear_model/, online/, fx/ — 18 submodules in total.

The optimization module exposes fifteen optimizer builders (each a frozen config + build_* factory): Mean-Risk (build_mean_risk) and its regime-blended and robust variants, HRP, HERC, NCO, Schur-Complementary, Risk Budgeting, Max-Diversification, Distributionally-Robust CVaR, Stacking, Benchmark-Tracker, and three naive baselines (Equal-Weighted, Inverse-Volatility, Random). Full per-module inventories (covariance/mu estimators, factor definitions, investability screens, presets) live in each submodule's source and docstrings under packages/portopt-core/optimizer/.

---

Requirements

• Python ≥ 3.12 (all four packages; the numpy 2.5 / pandas 3.0 pins make 3.12 a hard floor). CI runs 3.12 and 3.13.
• uv — the workspace resolver / runner.
• Docker + the docker compose v2 plugin — only for the ingestion daemon and the fund bridge (they need PostgreSQL 16). Not needed for the library.
• Optional API keys (ingestion): TRADING_212_API_KEY (+ secret), FRED_API_KEY. Absent Trading 212 ⇒ only the t212_annotate step skips cleanly; the universe itself is built from the yfinance Screener either way.

---

Installation

1. The optimization library only (no Docker, no DB)

# From a clone, into the current environment (the repo root declares `portopt-core`):
pip install -e .                            # editable install of dist `portopt-core`

Then import optimizer works with nothing else running. The library has no sibling-package dependencies, so plain pip is safe here. The daemon and the fund bridge are not: they depend on portopt-db / portopt-core, which exist only as workspace paths that uv resolves (pip ignores [tool.uv.sources] and would look those unpublished names up on PyPI), so install those two through uv.

2. The full platform (ingestion daemon)

The daemon ships an install wizard. The one-line bootstrap needs git: it clones this repo, installs uv, installs the portopt CLI straight from it (the CLI and the daemon are never published to PyPI; release.yml builds only the portopt-core library), then launches the wizard:

# macOS / Linux
curl -LsSf https://raw.githubusercontent.com/SilvioBaratto/optimizer/development/install.sh | bash
# Windows (PowerShell)
powershell -c "irm https://raw.githubusercontent.com/SilvioBaratto/optimizer/development/install.ps1 | iex"

Both scripts pull from the development branch by default; set PORTOPT_REF (a branch, tag or commit SHA) or PORTOPT_REPO before running to install a different ref or fork.

portopt runs docker compose and renders its plaintext secrets inside a checkout of this repo (it owns docker-compose.yml), and installing the CLI into a uv tool venv does not provide one. The scripts therefore clone into ~/optimizer (%USERPROFILE%\optimizer on Windows; set PORTOPT_HOME to choose elsewhere, which install.sh also accepts as PORTOPT_DIR), reuse that directory if it already holds a checkout, and export OPTIMIZER_REPO for the wizard, which then remembers the location in ~/.portopt/config.toml. install.sh also launches the wizard from inside the checkout and, on a re-run, updates an existing git checkout to PORTOPT_REF; it refuses to touch a directory that is a checkout of some other repo. Without a checkout, portopt setup stops with "Cannot locate the optimizer checkout" before it persists anything.

Equivalently, by hand — clone first, then run the from-clone front door:

git clone https://github.com/SilvioBaratto/optimizer && cd optimizer
./setup.sh      # setup.cmd / setup.ps1 on Windows; interactive wizard (see below)

To install the CLI yourself, point it at an existing checkout:

uv tool install "git+https://github.com/SilvioBaratto/optimizer.git@development#subdirectory=ingestion"
OPTIMIZER_REPO=/path/to/optimizer portopt setup

portopt setup verifies Docker + the compose plugin, validates your Trading 212 / FRED keys live, encrypts them (Fernet + scrypt) to ~/.portopt/secrets.enc (mode 0600, passphrase never persisted), brings up PostgreSQL (docker compose up -d --wait db), and applies the Alembic migrations. It seeds no data. Re-run any time with portopt setup.

Lifecycle:

portopt start     # decrypt secrets → render as compose secrets → docker compose up -d --wait (ingestion always, fund when an LLM backend is configured)
portopt stop      # stop the fund engine + delete its plaintext LLM keys (daemon + db keep running)
portopt stop --all  # stop the whole stack + delete every rendered plaintext secret
portopt status    # report Docker + service health (non-zero exit if anything is down)

portopt start brings up the fund engine only when an LLM backend is configured (a local Ollama base URL, or the selected provider's key plus an explicit model id; Ollama Cloud is the default provider). Without one it starts the data pipeline alone and prints one line saying so; run portopt setup --reconfigure to add a backend, then portopt start again.

portopt start renders the encrypted secrets into git-ignored ./secrets/<name> files. portopt stop deletes the LLM keys only the fund mounts, but keeps the three the ingestion daemon mounts (Trading 212 key and secret, FRED key): the daemon restarts on its own and needs them. Run portopt stop --all to stop the daemon too and leave no plaintext on disk. Both refuse while a fund run, fund approve or fund-tui is running inside the fund container, because stopping it would kill that round mid-flight; add --force to stop anyway.

First run. Setup seeds no data, so a fresh database has an empty instruments table. The first daily_pipeline (07:00, or run it straight away with docker compose exec scheduler python -m app.cli daily) therefore builds the universe first, a multi-minute yfinance Screener sweep, and then downloads prices and fundamentals for every instrument it found. Plan for many hours: about 8,700 tickers on 14 exchanges took roughly 14 hours on the first reported install. The scheduler logs a notice when it starts that download, later daily runs are incremental and short, and an interrupted bootstrap resumes on the next run because already-fresh data is skipped.

3. Developer setup (all four packages)

git clone https://github.com/SilvioBaratto/optimizer
cd optimizer
uv sync --all-packages --all-extras     # one venv with every member + every extra
docker compose --profile ingestion up -d # PostgreSQL (54320) + Adminer (18081) + scheduler (metrics 9000)
export DATABASE_URL=postgresql://postgres:postgres@localhost:54320/optimizer_db   # alembic reads only this variable, never .env
cd packages/portopt-db && alembic upgrade head   # apply migrations (single owner)

---

Usage

The optimization library

Feed linear returns (never log returns), keep temporal order, and compose pre-selection + an optimizer into one flat sklearn Pipeline:

from skfolio.preprocessing import prices_to_returns
from optimizer.optimization import MeanRiskConfig, build_mean_risk
from optimizer.pre_selection import PreSelectionConfig, build_portfolio_pipeline

# `prices`: a wide DataFrame — DatetimeIndex, one column per ticker.
X = prices_to_returns(prices)            # linear returns; runs OUTSIDE the pipeline

estimator = build_mean_risk(MeanRiskConfig())          # frozen config + factory
pipeline = build_portfolio_pipeline(                   # flattens pre-selection + optimizer
    estimator,
    pre_selection_config=PreSelectionConfig(),
)
pipeline.fit(X)
weights = pipeline[-1].weights_          # fitted skfolio optimizer exposes weights_

build_portfolio_pipeline flattens the steps so nested params are tunable via get_params() (e.g. "optimizer__l2_coef", "drop_correlated__threshold"). sector_mapping, expiration_dates and an outlier_protection_mask are optional keyword arguments (plain values, not queried from any database).

See each submodule's source and docstrings under packages/portopt-core/optimizer/ for every config, factory, preset and shape contract, and the skfolio/yfinance skills under .claude/skills/ for API-level guidance.

The ingestion daemon

Run the daemon (blocks; SIGTERM to stop) — normally via Docker Compose, but it can run in-process:

uv run --package portopt python -m app.worker

It schedules seven jobs with APScheduler in-process (a SQLAlchemyJobStore):

Job	Cadence
daily_pipeline	daily 07:00
midday_news	daily 14:00
universe_build	Saturday 02:00
weekly_refetch	Saturday 03:00
weekly_market_wide	Saturday 04:00
fred_monthly	1st of month 08:00
orphan_reaper	on an interval

universe_build runs before weekly_refetch (a stale universe caps yfinance). daily_pipeline additionally builds the universe itself when instruments has no active rows (a fresh install), so prices do not wait for the first Saturday. The reaper is a pure heartbeat lease: it fails any active (pending/running) job whose last_heartbeat_at is NULL or older than the lease TTL — so run exactly one daemon per database.

Trigger the same work manually (same job-slot / heartbeat path):

docker compose exec scheduler python -m app.cli daily
docker compose exec scheduler python -m app.cli refetch-all
docker compose exec scheduler python -m app.cli yfinance --mode full --period 5y

Full CLI surface (portopt <command> or python -m app.cli <command>): daily, refetch-all, universe, yfinance, macro, fx, fred, news, market-structure, calendars, market-summary, options, add-instrument (validates one Yahoo ticker, stores 5y of prices; --pin keeps it through the weekly universe build), daily-events (--backfill START:END rebuilds a date range), plus the lifecycle commands setup, start, stop, status.

The fund manager (fund)

The fund bridge lets an LLM make portfolio decisions and delegate the numeric work to optimizer, persisting runs to PostgreSQL via portopt_db. Orders are paper tickets, and by default a round pauses at a human-in-the-loop gate before place_orders (each portfolio's mandate sets its gates). The chat model is chosen by LLM_PROVIDER from a ten-provider registry (ollama, openai, openrouter, anthropic, google, groq, nvidia, huggingface, aws, microsoft); set nothing and it stays on the verified default — DeepSeek on Ollama Cloud. Install a provider's SDK with the matching extra (uv sync --package portopt-fund --extra <provider>, or --extra all-llm; openrouter and microsoft use --extra openai, ollama needs none). CLI (fund <command>):

Command	What it does
fund profile <portfolio_id>	Build/refresh an investor profile from a questionnaire
fund mandate set <portfolio_id> / fund mandate show <portfolio_id>	Write or display a portfolio's mandate (capital, base currency, drift threshold, benchmark, eligible tickers via repeatable --eligible / --clear-eligible, the Trading 212 scope via --tradeable-only / --no-tradeable-only)
fund run <portfolio_id> --asof <date>	Run a decision round for a rebalance bar
fund approve <run_id> / fund reject <run_id>	Resolve a paused (human-in-the-loop) run
fund status [portfolio_id]	Show the paused-run queue
fund report <run_id>	Tabular audit of one run
fund nav-mark [--rebuild]	Mark every active mandate's NAV now (the daily step, run by hand)

Two more entry points ship with it:

• fund-worker — the daemon: a weekly rebalance sweep, an interval drift monitor, a daily NAV mark and an orphan-job reaper (a heartbeat lease, like the ingestion reaper). The NAV mark waits for the day's ingestion price sweep to finish (up to FUND_NAV_MARK_WAIT_SECONDS, 4 h by default) so it values holdings at fresh closes.
• fund-tui — a Textual oversight cockpit with eight views (Posture, Deliberation, Decisions, Mandate, Audit, Performance, Ask, Daily brief). Every action it takes (approve, reject, freeze, mandate edits) goes through a confirm and lands in the audit log; Ask answers questions about the fund in plain language. Press ? for the key map.

Restricting a portfolio to a set of instruments

A portfolio can be limited to a hand-picked list (an ETF-only book, say). The list is a hard allowlist: the fund tools only list, optimise, risk-check and order tickers on it, and an empty list (the default) leaves the portfolio unrestricted. To set one up:

# 1. put each ticker in the universe (validates it on Yahoo, stores 5y of prices and the
#    ETF fund data; --pin keeps it through the weekly universe build)
portopt add-instrument VWCE.DE --asset-class equity --pin
portopt add-instrument IBGS.L --asset-class fixed_income --fi-subclass government --pin

# 2. allowlist them on the portfolio's mandate (repeatable; replaces the list)
fund mandate set <portfolio_id> --eligible VWCE.DE --eligible IBGS.L
fund mandate set <portfolio_id> --clear-eligible   # back to unrestricted

The same field is on the TUI Mandate view ("eligible tickers", comma-separated, saved with ctrl+s). A ticker with no priced instrument yet is accepted with a warning, so the mandate can be written before the instrument exists; until it has prices the ticker stays barred. Saving the mandate also pins the tickers (source mandate:<portfolio_id>) so the weekly build cannot delist them.

A run that must be executable at the broker can also be scoped to Trading 212 with fund mandate set <portfolio_id> --tradeable-only (--no-tradeable-only lifts it; the Mandate view has a "Tradeable on Trading 212 only" box). It composes with --eligible (both must hold) and works without an allowlist, then dropping every instrument the broker lacks from the whole universe, while research and benchmarking keep the full multi-region one. Only an instrument the ingestion annotation step found absent from the Trading 212 catalogue is barred (not_t212_tradeable); one it never evaluated stays. Until the step has evaluated anything the flag restricts nothing, which the save warns about and the PM seed repeats rather than silently emptying the universe.

With ESG exclusions in the client's profile, a fund is allowed by look-through: its latest sector snapshot must keep the excluded share within the client's tolerance (look_through_tolerance, strict 0 by default). Commodity funds and government or inflation-linked bond funds pass; any other fund whose snapshot is missing, older than FUND_ESG_SECTOR_WEIGHTS_MAX_AGE_DAYS (180) or incomplete is barred, so most corporate and aggregate bond ETFs are out under exclusions. The PM's instruction names every barred ticker with its reason.

---

Dependencies

Each member declares its own; the single shared venv installs one version of each shared pin. Exact runtime pins:

portopt-core (library): numpy==2.5.4, pandas==3.0.6, scipy==1.18.1, scikit-learn==1.9.1, skfolio==1.9.10, jinja2==3.1.6. (arch is not declared — it reaches bootstrap uncertainty-set classes transitively via skfolio. Code importing it directly must guard with try/except ImportError or declare it.)

portopt-db (DB layer): SQLAlchemy==2.0.52, psycopg2-binary==2.9.12, alembic==1.19.1, pandas==3.0.6, pydantic==2.14.1. Owns the SQLAlchemy pin the whole workspace inherits.

portopt (ingestion): yfinance==1.7.0, scikit-learn==1.9.1 (declared, for DBSCAN price-repair), numpy==2.5.4, pandas==3.0.6, APScheduler==3.11.3, SQLAlchemy==2.0.52, exchange_calendars==4.13.2, httpx==0.28.1, beautifulsoup4==4.15.0, requests==2.34.2, prometheus-client==0.26.0, typer==0.27.1, questionary==2.1.1, rich==15.0.0, cryptography==50.0.2, pydantic==2.14.1, pydantic-settings==2.15.0, python-dotenv==1.2.3, plus portopt-db (workspace; psycopg2-binary/alembic arrive transitively).

portopt-fund (bridge): deepagents==0.7.14 (the only exact pin), langgraph, langgraph-checkpoint-postgres, psycopg[binary], psycopg-pool, langchain-ollama, APScheduler, typer, textual, pydantic, python-dotenv, plus portopt-core + portopt-db (workspace).

---

Development

uv drives everything. Run per-package with uv run --package <name> ….

# Library (portopt-core) — tests, lint, types
uv run --package portopt-core pytest tests/ -v
uv run --package portopt-core pytest -k "test_name"
uv run --package portopt-core ruff check packages/portopt-core/optimizer/ tests/
uv run --package portopt-core mypy packages/portopt-core/optimizer/
uv run pyright                              # all four packages, via [tool.pyright] include

# Ingestion daemon (portopt) — member suites run from the member directory
cd ingestion && uv run pytest tests/

# Shared DB layer (portopt-db)
cd packages/portopt-db && uv run pytest
cd packages/portopt-db && DATABASE_URL=postgresql://postgres:postgres@localhost:54320/optimizer_db alembic upgrade head   # head: f4b6d8a0c2e3

# Fund bridge (portopt-fund)
cd fund && uv run pytest tests/

--package only selects the dependency set; it does not change the working directory. Run from the repo root, pytest always collects the root testpaths = ["tests"] (the library suite), so a member suite must be started from its own directory, exactly as CI does.

The root Makefile (make lint | format | typecheck | test | all) runs the same ruff and mypy gates as the CI lint/typecheck jobs: ruff over the library, ingestion/, fund/ and packages/portopt-db/, and mypy over the library, portopt-db and ingestion. make test runs only the library suite.

CI (.github/workflows/ci.yml, push/PR to main and development, Ubuntu, Python 3.12 & 3.13, uv-driven) runs nine jobs: lint (ruff check → ruff format --check on all four members → shellcheck → pip-audit), typecheck (mypy), pyright (pinned 1.1.414), test (--cov=optimizer ≥ 90%), ingestion-test (--cov=app ≥ 80%), portopt-db-test (--cov=portopt_db ≥ 90%), fund-test (--cov=fund ≥ 80%), postgres-integration (the Alembic upgrade to head on a real PostgreSQL plus the integration-marked tests), and docker-images (compose config + build of both images). release.yml publishes portopt-core to PyPI on v* tags, but only after checking that the tagged commit is on main, that the tag equals v<pyproject version>, and that the whole CI workflow passes on it.

---

Repository layout

optimizer/
├── pyproject.toml                 # root = dist `portopt-core`; declares the uv workspace
├── packages/
│   ├── portopt-core/optimizer/    # the optimization library (import `optimizer`)
│   └── portopt-db/                # shared DB layer + the single Alembic tree
├── ingestion/                     # the `portopt` daemon + CLI (import `app`)
├── fund/                          # the `portopt-fund` bridge (import `fund`)
├── tests/                         # library test suite (mirrors optimizer/ + scheduler/)
├── scheduler/                     # shell wrappers over the CLI
├── install.sh / install.ps1       # one-line bootstrap (clone + uv + the setup wizard)
├── setup.sh / .cmd / .ps1         # the from-clone setup front door
└── docker-compose.yml             # db + adminer + scheduler + fund (every service profile-gated)

---

License

PolyForm Noncommercial 1.0.0.