This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program.

Filtered Vector Search Benchmarks

This repository contains benchmarks for filtered vector search algorithms (HNSW, ACORN) implemented in:

1. C++ / HNSWLib — in-memory index tests comparing ACORN vs HNSW sweeping. See third_party/fvsb_hnswlib/README.md.
2. PostgreSQL — via a custom pgvector extension (see PostgreSQL section below).

Check out more details in our SIGMOD 2026 paper. If you use our work, please cite:

An In-Depth Study of Filter-Agnostic Vector Search on a PostgreSQL Database System
Duo Lu, Helena Caminal, Manos Chatzakis, Yannis Papakonstantinou, Yannis Chronis, Vaibhav Jain, Fatma Özcan.
SIGMOD 2026

Prerequisites

• git, wget, make, gcc, g++ (supporting C++20)
• python3, pip
• pkg-config (for HDF5 detection)

Python dependencies:

pip install numpy scipy h5py torch tqdm

---

Environment Setup

All configuration is centralized in a .env file at the repository root. Create one from the provided template:

cp .env.template .env
# Edit .env with your dataset paths and preferences

The C++ binaries (benchmark_runner, acorn_fvs_bitmap_test) auto-load .env from the working directory on startup — no need to export variables each session. For Python scripts, source the file first:

source .env && python3 -m gen_workloads.convert_fvecs_to_hdf5 --dataset sift1M

Shell-exported variables take precedence over .env values. If neither is set, built-in defaults are used (e.g., FVS_DATA_DIR=data).

See .env.template for the full list of variables.

---

1. HNSWLib (C++) — Quick Start

This is the fastest way to run ACORN vs HNSW filtered search tests.

A. Build the C++ tools

cd build
./setup_hnswlib_env.sh

This downloads dependencies and compiles both the benchmark runner and the ACORN test binary into $HOME/fvsb_hnswlib_build/.

B. Prepare your dataset

If you have an existing fvecs dataset (e.g., sift1M):

# Make sure .env is configured (see Environment Setup above)
source .env
python3 -m gen_workloads.convert_fvecs_to_hdf5 --dataset sift1M

Expected input layout at $FVS_DATA_DIR/sift/:

sift_base.fvecs
sift_query.fvecs
sift_groundtruth.ivecs

This creates:

• $FVS_DATA_DIR/sift1M-128-euclidean.hdf5 — for the C++ test
• $FVS_DATA_DIR/datasets/sift1M/sift1M.{index,test}.fvecs — for bitmap generation

C. Generate selectivity bitmaps

Edit gen_workloads/gen_bitmaps.py and set DATASETS_TO_GEN_BITMAPS to include only your dataset (e.g., ["sift1M"]), then run:

python3 -m gen_workloads.gen_bitmaps

See gen_workloads/README.md for details.

D. Run the ACORN vs HNSW test

cd $HOME/fvsb_hnswlib_build
./acorn_fvs_bitmap_test

The binary auto-loads .env from the working directory (the build script copies it there during setup). No manual export needed.

The test loads the dataset and bitmaps, builds HNSW + ACORN indexes, and reports recall/latency/speedup at each selectivity level.

See third_party/fvsb_hnswlib/README.md for advanced configuration.

---

2. PostgreSQL + PGVector Benchmark

This section covers how to set up the modified PostgreSQL environment, prepare data, index it, and run benchmarks.

A. Environment Setup

We provide a script to automate the build and setup of PostgreSQL with our custom pgvector extension.

1. Navigate to the build directory:

   cd build

2. Run the setup script:

   ./setup_pg_env.sh

   What this does:

   • Downloads and compiles PostgreSQL (v14.10) to $HOME/fvsb_pg_build/postgres.
   • Clones pgvector and replaces HNSW source files with our custom implementation from fvsb_pgvector/.
   • Installs the custom pgvector extension.
   • Initializes a database cluster at $HOME/fvsb_pg_build/data.
   • Starts the PostgreSQL server.
   • Creates a superuser vecbench and database vecbench.

3. Important: Source the environment variables to add psql and pg_ctl to your PATH:

   source $HOME/fvsb_pg_build/env.sh

B. Dataset Preparation

Before running benchmarks, you must download and preprocess the datasets (e.g., converting to HDF5/fvecs, generating ground truth, and creating selectivity bitmaps).

1. Download and Preprocess Datasets: From the root directory, run:

   python3 -m gen_workloads.prep_ds

   Note: You can modify DS_TO_PREP in gen_workloads/prep_ds.py to specify which datasets to download (e.g., openai5M, sift10M).

2. Generate Bitmaps: Generate bitmaps for different selectivity levels:

   python3 -m gen_workloads.gen_bitmaps

C. Indexing the Database

This step creates the table, inserts vectors, builds the HNSW/ACORN index, and generates TID mappings.

Run the indexing script:

python3 benchmarks/db_table_indexing_parallel.py --workload <workload_name>

• <workload_name>: Matches the keys in conf.json (e.g., openai5M, sift10M, cohere10M).

Example:

python3 benchmarks/db_table_indexing_parallel.py --workload openai5M

D. Running the Benchmark

This step executes the search queries with various filtering strategies (ACORN, Inline Filtering, etc.).

Run the benchmark script:

python3 benchmarks/db_benchmark_parallel.py --workload <workload_name> --k <k> --verbose

Example:

python3 benchmarks/db_benchmark_parallel.py --workload openai5M --k 10 --verbose

• Results will be saved in vecbench_results/.
• Logs are written to /tmp/fvsb_pg_logs/.

---

Repository Structure

fvs/
├── README.md                          # This file
├── build/
│   ├── setup_hnswlib_env.sh           # Build script for C++ benchmarks
│   └── setup_pg_env.sh                # Build script for PostgreSQL
├── gen_workloads/                      # Dataset preparation & bitmap generation
│   ├── README.md                      # Detailed workload generation docs
│   ├── convert_fvecs_to_hdf5.py       # Convert fvecs → HDF5
│   ├── prep_ds.py                     # Download & preprocess HDF5 datasets
│   ├── gen_bitmaps.py                 # Generate selectivity bitmaps
│   ├── load_data.h                    # C++ HDF5/bitmap loading utilities
│   └── conf.json                      # Dataset configuration
├── third_party/
│   ├── fvsb_hnswlib/                  # C++ HNSW/ACORN implementations
│   │   ├── README.md                  # Detailed HNSWLib docs
│   │   ├── acorn/acornalg.h           # ACORN algorithm
│   │   ├── hnsw/hnswalg.h             # Modified HNSW
│   │   ├── indices_tests/             # C++ test programs
│   │   └── experiment/                # Benchmark runner
│   └── fvsb_pgvector/                 # Custom pgvector extension
└── benchmarks/                        # PostgreSQL benchmark scripts