This is the repository to the preprint Task- and dataset-specific information in protein language models. It contains all code necessary to reproduce the results as well as a small example dataset.
To install the required dependencies, run:
pip install -r requirements.txtAll code has been tested with Python 3.12 on an NVIDIA RTX 3090 with 24GB RAM and CUDA 12.9.
After successfully installing the dependencies, you can start by using the example dataset. We provide a small example dataset examples/fluorescence_sample.csv with a 500-row subsample of the TAPE fluorescence benchmark. This is already preprocessed. For any other dataset, please check the preprocessing section below.
We advice to set a root directory for your embeddings, for example:
$ROOT='/scratch/mille_feuille/'You can execute the following codes from the main folder of this repository.
First, to compute the embeddings for the example dataset, run:
python -m src.plm --model-name esm_t6 --data-path examples/fluorescence_sample.csv --output-path $ROOT/embeddings/esm_t6/fluorescence_sample/then, to probe the layers of the computed embeddings, run:
python -m src.downstream.probe_layer --data-path examples/fluorescence_sample.csv --embed-base $ROOT/embeddings/esm_t6/fluorescence_sample/ --max-layer 6 --task regressionAdditional arguments needed for special experiments in the manuscript can be found in the respective scripts and are documented there, e.g., by running python -m src.plm --help or python -m src.downstream.probe_layer --help.
The execution takes only a few minutes.
The manuscript lists 10 datasets and 15 downstream tasks. The preprocessing scripts are deposited in src/datasets.
This codebase follows a structured folder organization for storing embeddings and related files. The main directories are embeddings/ for model embeddings and aa_embeddings/ for amino acid embeddings. Each model has its own subdirectory, and within each model, datasets are organized with layers and associated files as shown below.
$ROOT
├── embeddings/
│ ├── esm_t6/
│ │ ├── fluorescence_sample/
│ │ │ ├── layer_0/
│ | │ │ ├── P00000.pkl
│ | │ │ ├── ... (more embeddings)
│ | │ │ ├── ids.csv # Intrinsic dimension of layer 0
│ | │ │ ├── noverlap.csv # Number of overlapping dimensions between layer 0 and layer 1
│ | │ │ ├── pca.pkl # variances of PCs from a PCA on layer 0 embeddings
│ | │ │ ├── predictions_knn.pkl # k-NN predictions for layer 0 embeddings
│ | │ │ └── predictions_lr.pkl # Linear regression predictions for layer 0 embeddings
│ │ │ └── ... (more layers)
│ │ └── ... (more datasets)
│ └── ... (more models)
└── aa_embeddings/