Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Mille Feuille

This is the repository to the preprint Task- and dataset-specific information in protein language models. It contains all code necessary to reproduce the results as well as a small example dataset.

Installation

To install the required dependencies, run:

pip install -r requirements.txt

All code has been tested with Python 3.12 on an NVIDIA RTX 3090 with 24GB RAM and CUDA 12.9.

Quickstart

After successfully installing the dependencies, you can start by using the example dataset. We provide a small example dataset examples/fluorescence_sample.csv with a 500-row subsample of the TAPE fluorescence benchmark. This is already preprocessed. For any other dataset, please check the preprocessing section below.

We advice to set a root directory for your embeddings, for example:

$ROOT='/scratch/mille_feuille/'

You can execute the following codes from the main folder of this repository.

First, to compute the embeddings for the example dataset, run:

python -m src.plm --model-name esm_t6 --data-path examples/fluorescence_sample.csv --output-path $ROOT/embeddings/esm_t6/fluorescence_sample/

then, to probe the layers of the computed embeddings, run:

python -m src.downstream.probe_layer --data-path examples/fluorescence_sample.csv --embed-base $ROOT/embeddings/esm_t6/fluorescence_sample/ --max-layer 6 --task regression

Additional arguments needed for special experiments in the manuscript can be found in the respective scripts and are documented there, e.g., by running python -m src.plm --help or python -m src.downstream.probe_layer --help.

The execution takes only a few minutes.

Supported datasets

The manuscript lists 10 datasets and 15 downstream tasks. The preprocessing scripts are deposited in src/datasets.

Folder structure

This codebase follows a structured folder organization for storing embeddings and related files. The main directories are embeddings/ for model embeddings and aa_embeddings/ for amino acid embeddings. Each model has its own subdirectory, and within each model, datasets are organized with layers and associated files as shown below.

$ROOT
   ├── embeddings/
   │   ├── esm_t6/
   │   │   ├── fluorescence_sample/
   │   │   │   ├── layer_0/
   │   |   │   │   ├── P00000.pkl
   │   |   │   │   ├── ... (more embeddings)
   │   |   │   │   ├── ids.csv  # Intrinsic dimension of layer 0
   │   |   │   │   ├── noverlap.csv  # Number of overlapping dimensions between layer 0 and layer 1
   │   |   │   │   ├── pca.pkl  # variances of PCs from a PCA on layer 0 embeddings
   │   |   │   │   ├── predictions_knn.pkl  # k-NN predictions for layer 0 embeddings
   │   |   │   │   └── predictions_lr.pkl  # Linear regression predictions for layer 0 embeddings
   │   │   │   └── ... (more layers)
   │   │   └── ... (more datasets)
   │   └── ... (more models)
   └── aa_embeddings/

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages