zxcyfr/Human-Protein-Language-Model-Combined

Human Protein Language Model — Combined

Overview

This repository contains the Combined protein language model checkpoint generated for the study “Does domain-specific unsupervised fine-tuning improve protein language model performance?”.

Pfam families that did not contain sufficient sequences for independent Pfam-level or Clan-level fine-tuning were pooled into a composite training dataset. The sequences were clustered at a sequence-identity threshold of 0.5 and used for continued unsupervised fine-tuning of the ESM-2 650M model.

Unlike the Pfam- and Clan-level checkpoints in this project, the Combined model was trained across multiple otherwise ineligible protein families and therefore represents broad cross-family continual pretraining rather than adaptation to a single family or clan.

Model details

• Base model: ESM-2 650M (facebook/esm2_t33_650M_UR50D)
• Training objective: masked language modeling
• Training strategy: pooled cross-family continual pretraining
• Sequence-identity threshold: 0.5
• Number of checkpoints: 1

Intended use

This checkpoint is provided for research on protein representation learning, protein language model benchmarking, continual pretraining, and the effects of cross-family unsupervised adaptation.

The model is intended for research use only and has not been validated for clinical or diagnostic applications.

Code and usage

Source code, benchmarking workflows, and usage instructions are available at:

https://github.com/TianBoxue-lab/DS-UFT-Benchmark

Related models

The accompanying Pfam- and Clan-level model checkpoints are available through the DS-UFT Protein Language Models for Human Protein Families collection on Hugging Face.

License

This repository is distributed under the AFL-3.0 license.