Genetics Language Models

Language models trained on DNA, RNA, and protein sequences instead of natural-language text. The same transformer architecture that predicts the next word can predict the next base pair or amino acid, letting a model learn the statistical structure of genomes the way an LLM learns the structure of language.

Genetics Language Models surveys the field.

Applications

Predicting gene function and co-regulation from genomic context, without needing a labeled dataset for every gene.
Designing cis-regulatory elements (promoters, enhancers) for biomanufacturing and gene-therapy applications.
Modeling protein structure and function jointly with the genomic sequence that encodes it.

Targets

Most models in this space operate on one of three sequence types: raw DNA (nucleotide-level), protein sequences (amino-acid-level), or a joint genomic-and-protein representation that links a gene's sequence to what it produces.

Research

Genomic language model predicts protein co-regulation and function

The authors show in their paper the ability to train genomic language models on top of protein language models (ESM2) "on millions of metagenomic scaffolds to learn the latent functional and regulatory relationships between genes." Their reveal "a promising approach to encode functional semantics and regulatory syntax of genes in their genomic context and uncover complex relationships between genes in a genomic region."

RegLM - a toolkit for training hyenaDNA based autoregressive language models on DNA sequences

Developments The authors show in their paper a model capable of generating Cis-regulatory elements (CREs) like promoters and enhancers that can regulate the expression of Genes. These are useful for biomanufacturing as well as otheir therapeutic applications.