GraphML News (June 29th) - ESM 3, TDC 2, AI 4 Genomics Conference

🧬 Evolutionary Scale (formerly a team in Meta, now a standalone startup) released ESM 3 - the next version of the SOTA protein LM, pretty much GPT-4 of pLMs. Now it’s not only a sequence model, but also a structure and function model. Following best LLM practices, ESM 3 even employs RLHF for aligning with human feedback! Besides, the model features SE(3)-invariant geometric attention based on distances between frames (equivariance not dead!) and VQ-VAE to tokenize structures and functions. The ESM 3 family is available in three sizes: 1.4B is open weights, 8B and 98B are available in the API (it’s time to embrace that). The preprint is quite informative about training data, pre-/post-training details, and RLHF details - kudos for not sweeping it under the rug. The model code is also available, so you only need 10,000 H100’s to train it on your own 🙂

💊 The team of Harvard, MIT, and Stanford researchers led by Marinka Zitnik released Therapeutic Data Commons 2 adding even more datasets and modalities: over 1000 single-cell datasets over 85M cells, the first protein-peptide binding dataset, drug-target interaction data, clinical trials data, and much more covering 10+ modalities. TDC-2 packages several pre-trained embeddings and can be used for evaluating a variety of models - from LLMs to GNNs. TDC-1 received some critic from drug discovery people back in the days, let’s see if TDC-2 closes those gaps.

The AI for Genomics and Health conference will be held in Boston, Oct 17-18th, with a stellar lineup of speakers including Shekoofeh Azizi (DeepMind, the author of Med-PaLM), Mo Lotfollahi (Sanger Institute), Sergey Ovchinnikov (MIT), Marinka Zitnik (Harvard), and James Zou (Stanford).

🔮 A small update: MatterSim by MSR AI4 Science became SOTA on MatBench Discovery beating recent GNoME from DeepMind - competition makes wonders even in such advanced scientific topic as materials discovery, ML potentials, and molecular dynamics.

Weekend reading:

Multimodal Graph Benchmark by Zhu et al feat. Danai Koutra - three datasets combining graphs, texts, and images for node classification and link prediction tasks.

Transformers meet Neural Algorithmic Reasoners by Bounsi et al feat. Petar Veličković - Transformer cross-attenting to the pre-trained Triplet-GMPNN solves algorithmic reasoning tasks (CLRS-text) better than the vanilla Transformer (but still struggles with OOD generalization though)

Clifford-Steerable Convolutional Neural Networks by Maksim Zhdanov et al - ConvNets go spacetime — equivariant to the Lorentz group and useful for electrodynamics. The thread by Maurice Weiler explains the work much more in details. Someday (after another PhD in math and physics) I will be able to understand the math behind this paper