Graph Machine Learning

​​Call for papers for the ICCV Workshop on Scene graphs and graph representation learning

Guest post by Azade Farshad

We invite you to submit your graph related work to the ICCV Workshop on Scene graphs and graph representation learning. The workshop will feature talks by Fei Fei Li (Stanford), Luca Carlone (MIT), Bernard Ghanem (KAUST), Nicolas Padoy (Uni Strasbourg), Emanuele Rodola (Sapienza Rome), Hanwang Zhang (NTU), and Helisa Dhamo (Huawei).

Deadlines and Important Dates:

- Paper submission deadline: July 25th, 2023 (11:59 PM Anywhere on Earth)
- Notification to Authors: August 8th, 2023
- Camera-ready Deadline: August 19th, 2023

Submission instructions are available on the SG2RL website:

https://sg2rl.github.io/

Workshop date: October 2, 2023

Graph ML News (July 15th) - NeurIPS workshops, M2Hub

🪛 NeurIPS’23 announced the list of accepted workshops! Graph learning is well-presented, you might want to have a look at:

- AI for Accelerated Materials Design (AI4Mat-2023)
- AI for Science: from Theory to Practice
- Machine Learning and the Physical Sciences
- Machine Learning in Structural Biology Workshop
- New Frontiers in Graph Learning (GLFrontiers)
- New Frontiers of AI for Drug Discovery and Development
- Symmetry and Geometry in Neural Representations
- Temporal Graph Learning Workshop

We will keep an eye on the submission deadlines, generally you might expect them to be somewhere around the NeurIPS accepted papers announcement.

💠 M2Hub is a fresh collection of datasets and models for materials discovery: 11 datasets spanning organic and inorganic molecules and crystals, 8 models including EGNN, Equiformer, DimeNet, and GemNet, and more experiments in the fresh preprint. Materials discovery is catching up with drug discovery!

📈 UniMol, a 3D framework for molecular representation learning, has updated the performance of UniMol+ showing strong performance on OGB PCQM4M v2 and OpenCatalyst (OC20 IS2RE). Looks like this year’s OGB Large Scale challenge and Open Catalyst challenge are going to have a heated competition, eg, having in mind a recently released EquiformerV2.

🔬Simone Scardapane (Sapienza) prepared a nice slide deck on Designing and Explaining GNNs - the second half of the deck is about current explainability methods, have a look if you work in this area.

Weekend reading:

M2Hub: Unlocking the Potential of Machine Learning for Materials Discovery

An OOD Multi-Task Perspective for Link Prediction with New Relation Types and Nodes

GraphML News (July 7th) - Generative Chemistry, Temporal Graph Benchmark

Lots of news this week!

🔬 Starting with new blog posts, Charlie Harris wrote an article on Diffusion Models in Generative Chemistry for Drug Design covering the basics of denoising diffusion and score-based generative modeling going into molecular usecases with Equivariant Diffusion, DiffSBDD, DiffDock, and raising questions about fair evaluation of generative models vs standard tools.

Looking more from the industrial perspective, Leo Wossnig published a piece Where is generative design in drug discovery today discussing successes and failures of generative approaches and highlighting the main obstacles for ML folks wishing to dive into drug discovery: (1) data scarcity (eg, no data for new targets), (2) slow experimental pipelines to generate new data, (3) end-to-end pipelines and tech stack in general.

🔧 Graphium is the new library for molecular representation learning in the Datamol ecosystem. Graphium is packed with latest algorithms (like Random Walk Structural Encodings) and ML models (like recent GPS++, the winner of OGB LSC’22), and scales to large compute, you can even spin up training on Graphcore IPUs.

📏 Temporal Graph Benchmark (TGB) is finally here! It was long awaited in the graph learning community that OGB needs a temporal branch, and TGB delivers dynamic link prediction and node property prediction datasets, standard loaders, evaluators, and, of course, the leaderboards (good old Temporal Graph Network is still a very strong baseline). Similarly to OGB, there are small, medium, and large graphs (the largest include about 1M nodes, 50M edges, 30M timesteps). More details can be found in the preprint by Shenyang Huang, Farimah Poursafaei, and the OGB gang.

AStarNet, the scalable GNN for KG reasoning, can now be integrated right with ChatGPT to enhance its factual correctness. Given a textual query, AStarNet also runs graph inference on the backbone graph (Wikidata subset) and produces top reasoning paths supporting the answer.

New foundation models: 1️⃣ NSQL, a Copilot-like LM for SQL queries by Numbers Station, is openly available in 350M / 2B / 6B versions outperforming all existing open-source SQL models; 2️⃣ xTrimo, 100B closed-source protein LM

GraphML News (July 1st)

⚛️ 3rd Open Catalyst challenge has been announced! This year the task is to predict the global minimum binding energy (adsorption energy) given an adsorbate and catalyst surface. The main dataset includes the new OC20-Dense split with roughly 15K initial structures and 3M frames. Baselines are GemNet-OC and Equivariant Spheric Channel Network (eSCN). Results will be announced at NeurIPS’23.

🎙️ The biennial Sampling Theory and Applications Conference (SAMPTA) 2023 will soon take place at Yale in July 10-14. This year will feature invited talks by Soledad Villar, Gitta Kutyniok, Michael Bronstein, Dan Spielman, and other prominent researchers. Registrations are still open!

🧬 Folks in comp bio might want to refresh the background on state space models (SSMs) - HyenaDNA, a collab between Stanford, Harvard, and Mila, is a DNA model with a whopping context length of up to 1M tokens of single nucleotides. HyenaDNA scales to lengths unattainable even by linear Transformers and shows SOTA of 23 genomic tasks. Pre-trained checkpoints are already on HuggingFace. The authors hint upon further generative applications, so we’ll keep an eye on that.

A few words about hypergraphs: the anonymous author put on a tutorial on the basics of hypergraphs and building hypergraph GNNs. And a new work Hypergraph factorisation for multi-tissue gene expression imputation shows how to use message passing hypergraph NNs for processing gene expression in comp bio applications.

GraphML News (June 24th)

A plenty of news, finally.

There is going to be a GraphML community meetup at ICML’23 similar to that at ICML’22 and NeurIPS’22, feel free to drop by if you are at the conference. More details are to follow in the LoG2 slack.

Michael Bronstein, Francesco Di Giovanni and Ben Gutteridge wrote a new blogpost on Dynamically Rewired Delayed Message Passing - an approach to address over-squashing and improve long-range capabilities of GNNs. vDREW is based on the idea of sparse skip connections where a node can have a direct access to its k-hop neighbors but delayed in time. Good to see our long-range LGRB dataset gaining more traction in the community 🙂

New PhD thesis: Cristian Bodnar, the author of message passing architectures using simplicial complexes, cellular complexes, and neural sheaves, published his work on Topological Deep Learning: Graphs, Complexes, Sheaves. This is an excellent introduction to topology and topological DL, highly recommended.

LLMs can instruct robots to create chemical compounds: Andrew White presented a nice demo of ChemCrow - an agent that goes from natural language instructions to a sequence of real robotic actions to synthesize something. The authors synthesized 3 organocatalysts and even an insect repellent.

Weekend reading:

Equiformer V2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations feat. Tess Smidt - the next and faster version of the famous and quite popular Equiformer (ICLR’23).

Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materials feat. Shengchao Liu, Anima Anandkumar, and Jian Tang - introduces Geom3D, a massive suite of datasets and 2D/3D geometric models that work on molecules, proteins, and even crystals. The repo offers 10 datasets, 35 models, and 14 pre-training methods.

QH9: A Quantum Hamiltonian Prediction Benchmark for QM9 Molecules feat. Shuiwang Ji - QM9 with Hamiltonian matrices for 2.4K MD trajectories and 130K molecular geometries.

Hyperbolic Representation Learning: Revisiting and Advancing feat. Rex Ying - a nice overview of hyperbolic GNNs in general and a new model, Hyperbolic Informed Embedding, in particular.

On the Connection Between MPNN and Graph Transformer

Chen Cai, Truong Son Hy, Rose Yu, Yusu Wang from UCSD

Invited post by Chen Cai

Our work aims to understand the relationship between local GNN (MPNN) and global GNN (Graph Transformer). Here local GNN refers to MPNN that mix node features locally and global GNN refers to the graph transformer, which incorporates graph topology into positional embeddings (along with existing node/edge features) and send a set of feature vectors into a vanallia transformer.

1️⃣ Local MPNN and global Graph Transformer are two major paradigms of graph learning. MPNN comes early and encompasses several popular architectures such as GCN, GraphSage, and GAT. However, they are known to suffer from issues like limited expressive power, over-smoothing, and over-squashing. Graph Transformer, on the other hand, received a lot of attention recently and shows competitive performance on standard benchmarks like OGB and tasks that require long-ranging reasoning. Our goal is to gain a fine-grained understanding of the relationship between such two paradigms.

2️⃣ Previously, 1 showed that with specific positional embedding, GT can approximate 2-IGN Invariant Graph Networks, which is at least as expressive as MPNN. This work establishes an important link from global GNN to local GNN.

3️⃣ In our work, we tried to establish the link from the inverse direction: can MPNN approximate GT as well? We looked into a very simple way to incorporate global modeling into the local mixing of MPNN: virtual node (VN). MPNN + VN adds a virtual node and connects VN to all the graph nodes and then does (heterogenous) message passing on the modified graph. It is a heuristic proposed in the early days of GNN research and shows consistent improvement over MPNN. However, there is little theoretical understanding of MPNN + VN.

4️⃣ We have a set of approximation results of MPNN + VN on GT. 1) for constant depth and width MPNN + VN (O(1) depth and width), it can approximate self-attention layers of two “linear transformers”, Performer & Linear Transformer. 2) for wide MPNN + VN (O(1) depth O(n^d) width), we show it can approximate full GT via a link to equivariant Deepsets. 3) For deep MPNN + VN (O(n) depth O(1) width), it can approximate one self-attention layer.

5️⃣ On the experimental side, we pushed the limit of simple MPNN + VN and showed that 1) it works surprisingly well on LRGB (long-range graph benchmark) dataset where previously GT dominates. 2) leveraging the GraphGPS framework, our MPNN + VN improves over the previous implementation 3) MPNN + VN outperforms Linear Transformer and MPNN on the climate modeling task. See our paper and code for more details!

Pure Transformers are Powerful Graph Learners

GraphML News (June 17th) -- Distributional Graphormer

It seems researchers took a break after NeurIPS deadlines (or braced themselves with the 6-paper reviewing batches) and there hasn’t been that much news lately.

Microsoft Research announced Distributional Graphormer, a massive generative model based on Graphormer suitable for many AI 4 Science tasks such as protein ligand binding, conformation transition pathway prediction, and even conditional crystal lattice generation (eg, generate a structure with a given band gap). Quoting the authors, “DiG attempts to predict the complicated equilibrium distribution of a given system by gradually transforming a simple distribution (e.g., a standard Gaussian) through the simulation of a predicted diffusion process that leads towards the equilibrium distribution.” The accompanying 80-pager preprint is a nice weekend reading 😉

Apart from that, the Learning on Graphs meetup took place in Paris with exciting keynotes, and Michael Bronstein received a prestigious UKRI Turing AI Fellowship (only two were given this year) to work on Graph ML algorithms inspired by physical systems.

More new papers:

Topological Singularity Detection At Multiple Scales (ICML’23) by Julius von Rohrscheidt and Bastian Rieck. Check out a nice Twitter thread by Bastian for a visual explanation.

Generating Novel, Designable, and Diverse Protein Structures by Equivariantly Diffusing Oriented Residue Clouds (ICML’23) by Yeqing Lin, Mohammed AlQuraishi. Genie is 10x smaller than RFDiffusion but gets quite close in terms of generative performance! Have a look at the thread by Mohammed with fancy generated gifs.

Rigid Body Flows for Sampling Molecular Crystal Structures feat. Pim de Haan and Frank Noé

Enabling tabular deep learning when d≫n with an auxiliary knowledge graph feat. Hongyu Ren and Jure Leskovec

GraphML News (June 10th)

Emanuele Rossi and Michael Bronstein published a new blog post on Directed GNNs. The idea is rather simple (different learnable aggregations for in- and out- neighbors) and shows very good results on heterophilic graphs. In fact, DirGNNs very much resemble relational GNNs (R-GCN, CompGCN, NBFNet, and so on) who have learnable aggregations per unique relation type (and its inverse), and we recently published a theory paper on the expressiveness of such GNNs. Great to see prominent GNN folks joining the relational party 😉

The Learning on Graphs Conference (LoG) is not just one of the coolest venue for Graph ML research - the organizers do care about the community and factor in your feedback. Bastian Rieck and Corinna Coupette summarized the results from the anonymous poll among authors and reviewers in Evaluating the "Learning on Graphs" Conference Experience highlighting what worked (eg, monetary awards for reviewers and lesser paper load) and not quite (some papers were rejected even when authors submitted rebuttals but reviewers did not engage). Let’s help LoG to grow to the best Graph ML research venues!

FAIR, CMU, and The Open Catalyst Project are about to announce the next large NeurIPS challenge - most likely it would be about adsorption energy estimation. Brace yourselves and prepare your best equivariant geometrics models.

Da Zheng and Florian Saupe from AWS published a post introducing GraphStorm (we noticed the original paper a few weeks ago), a low-code framework for large-scale graph learning targeted for enterprise applications. The post goes through several examples on graph building and running inference.

Some weekend reading:

Ewald-based Long-Range Message Passing for Molecular Graphs (ICML’23) and its LOG2 reading group presentation

Validation of de novo designed water-soluble and transmembrane proteins by in silico folding and melting - comparison of AlphaFold2 vs ESMFold

How does over-squashing affect the power of GNNs? feat. Francesco Di Giovanni, Michael Bronstein, and Petar Veličković

A Fractional Graph Laplacian Approach to Oversmoothing feat. Gitta Kutyniok

Graph ML News (June 3rd)

Molecular ML conference (MoML) took place in Montreal on Monday and hosted invited talks from Marinka Zitnik, Mohammed AlQuraishi, Gábor Csányi, and other top researchers in the field. The recordings are now online on YouTube.

Accepted papers at ICML 2023 are now visible on the conference website (available after registration) - several editors of this channel are going to ICML this year and it is likely we will prepare an overview of the most interesting graph papers!

Weekend reading:

Geometric Latent Diffusion Models for 3D Molecule Generation (ICML’23) feat. Minkai Xu, Stefano Ermon, and Jure Leskovec

Protein Design with Guided Discrete Diffusion feat. Kyunghyun Cho and Andrew Wilson

Graph Inductive Biases in Transformers without Message Passing (ICML’23) feat. Derek Lim

Unsupervised Embedding Quality Evaluation feat. Anton Tsitsulin

Smooth, exact rotational symmetrization for deep learning on point clouds

Graph ML News (May 27th): New Antibiotic found with Geometric DL, Differential Privacy, NeurIPS Submissions

A new antibiotic abaucin is discovered by the power of Geometric Deep Learning! Abaucin targets a stubborn Acinetobacter baumannii pathogen resistant to many drugs. The new Nature Chem Bio paper (feat. Regina Barzilay and Tommi Jaakkola from MIT) sheds more light on the screening process and used methods.

Stanford launches the online version of the flagship CS224W course of Graph ML. The 10-credit course is priced at $1,750 and starts on June 5th.

The TAG in ML workshop on topology announced a new challenge: implementing more topology-enabled neural nets with the TopoModelX framework where top contributors will become co-authors of a JMLR submission. That’s a great option for those who’d like to start working with topological neural architectures!

Vincent Cohen-Addad and Alessandro Epasto of Google Research published a post on differentiably-private clustering: introducing an approach for DP hierarchical clustering with formal guarantees and lower bounds, and an approach for large-scale DP clustering.

The Weekend Reading section this week is brought to you by NeurIPS submissions, quite a number of cool papers:

Link Prediction for Flow-Driven Spatial Networks - the work introduces the Graph Attentive Vectors (GAV) framework for link prediction (based on the labeling trick commonly used in LP) and smashes the OGB-Vessel leaderboard with a 10-points rocauc margin to the previous SOTA.

Edge Directionality Improves Learning on Heterophilic Graphs feat. Emanuele Rossi, Francesco Di Giovanni, Fabrizio Frasca, Michael Bronstein, and Stephan Günnemann

PRODIGY: Enabling In-context Learning Over Graphs feat. Qian Huang, Hongyu Ren, Percy Liang, and Jure Leskovec - a cool attempt to bring prompting to the permutation-invariant nature of graphs.

Uncertainty Quantification over Graph with Conformalized Graph Neural Networks feat. Kexin Huang and Jure Leskovec — one of the first works on Conformal Prediction with GNNs.

Learning Large Graph Property Prediction via Graph Segment Training feat. Jure Leskovec and Bryan Perozzi

ChatDrug - a neat attempt at combining ChatGPT with retrieval plugins and molecular models to edit molecules, peptides, and proteins right with natural language. Extension of MoleculeSTM that we featured in the recent State of Affairs post.

MISATO - Machine learning dataset for structure-based drug discovery - a new dataset of 20K protein-ligand complexes with molecular dynamics traces and electronic properties.

Multi-State RNA Design with Geometric Multi-Graph Neural Networks feat. Chaitanya Joshi and Pietro Lio

Graph ML News (May 20th)

The NeurIPS deadline has passed so we could finally disconnect from the cluster, breathe in some fresh air and get ready for the supplementary deadline and/or paper bidding depending on your status.

The Workshop on Mining and Learning with Graph (MLG) at KDD’23 accepts submissions until May 30th. This year KDD will feature both MLG and Graph Learning Benchmarks (GLB), so two more reasons to visit Long Beach and chat with the fellow graph folks 😉

CS224W, one of the best graph courses from Stanford, started publishing project reports of the Winter 2023 cohort: some new articles include solving TSP with GNNs, approaching code similarity, and building music recommendation system with GNNs. More reports will be published within the next few weeks.

Weekend reading:

DRew: Dynamically Rewired Message Passing with Delay feat. Michael Bronstein and Francesco Di Giovanni (ICML’23)

Random Edge Coding: One-Shot Bits-Back Coding of Large Labeled Graphs (ICML’23)

Can Language Models Solve Graph Problems in Natural Language?

On the Connection Between MPNN and Graph Transformer

Academic opportunities in Geometric DL

A new batch of open positions (apparently when the funding got finally secured for the next financial year 🙂 ) - keep in mind that some of them accept applications only for the next few weeks.

Internship at the AI4Science PDE Team at Microsoft Research with Johannes Brandstetter, Cristian Bodnar, and Max Welling

Postdoc in ML and Systems Biology with Karsten Borgwardt at MPI Munich

PhD Student in Generative Models for Inverse Molecular Design with Simon Olsson at Chalmers University of Technology

Faculty and PhD positions on Foundation Models for Science with Miles Cranmer, Shirley Ho, Kyunghyun Cho at the Simons Foundation and Flatiron Institute

Graph ML News (May 13th) - $100M Wall Street Edition

💸 Perhaps the biggest news of the month: Recursion, a major player in drug discovery, acquires two startups: Valence Discovery (Mila, Montreal) for $47.5M and Cyclica (Toronto) for $40M. Not pretending to wear a Wall Street market analyst hat, I’d speculate those are the biggest M&A deals of the past years in the Graph ML industry. Graph ML and Geometric Deep Learning are at the core of modern drug discovery powering pretty much all stages of the pipeline reducing the time to market from standard 10 years by a factor of 2-3x.

I happen to know many smart folks from Valence Discovery including Prudencio Tossou, Therence Bois, and Dominique Beaini with whom we co-authored a few papers for NeurIPS’22. Valence also supports the most popular public reading groups on Graph ML: Learning on Graphs and Geometry (LOG2) and Molecular Modeling & Drug Discovery (M2D2) covering hot new papers with original authors. Big congratulations to the team and hope we’ll see more cool stuff in the future!

With the Wall Street hat on, I’d hypothesize the next big wave of investment rounds and huge M&As would be in the material discovery and AI4Science fields where Geometric DL is at the core either.

Venues:

ECML PKDD’23 in Torino published the list of accepted workshops - have a look at the Mining and Learning with Graphs (MLG) workshop featuring keynotes from Bastian Rieck and Giannis Nikolentzos. Bastian gives amazing talks on topology, highly recommend to attend if you are at ECML this year. Paper submission deadline is June 12th, consider submitting as well.

Weekend reading:

Alex Barghi wrote a blogpost introducing the new cuGraph backend of PyG covering new accelerated primitives, feature store, and neighbor sampling using node classification on the MAG graph as example.

Zhaocheng Zhu posted a viral tweet with the Colab Notebook comparing PyTorch and JAX performance of common GNN operators. Key takeaways are: JAX with JIT is faster than PyTorch on homogeneous graphs, and much faster and memory-efficient on larger heterogeneous graphs when PyTorch throws OOM; new torch.compile() often makes the code 2x faster than vanilla torch, so make sure to update your envs to torch 2.0 🚀

New papers for the weekend reading:

Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files by Daniel Flam-Shepherd and Alan Aspuru-Guzik - “In this work, we show how language models, without any architecture modifications, trained using next-token prediction - can generate novel and valid structures in three dimensions from various substantially different distributions of chemical structures.” Sparks of chemical intelligence 👀

Advancing structural biology through breakthroughs in AI by Laksh Aithani and folks from Charm Therapeutics - a nice introductory survey how (Geometric) DL transforms structural biology.

Graph ML News (May 6th)

ICLR’23 has finished this week, to those who travelled to Kigali - have a safe trip back 🙂

Meanwhile, you might have missed the ICLR Blogposts Track - a collection of insightful articles for which it is often more handy to express the content as a blog post rather than a full paper. Particularly interesting are On Universality of Neural Networks on Sets vs Graphs (by Fabian B. Fuchs and Petar Veličković), on Neural PDE Solvers (by Yolanne Lee), and Thinking Like Transformer (by Alexander Rush, Gail Weiss). I would generally recommend submitting there (my post was accepted at ICLR’22 Blog Post Track) - it was a pleasant experience and you also do some community serving writing about your research.

A few upcoming events:

LoG Paris Meetup on June 14th in Paris at CentraleSupélec, Université Paris-Saclay with the keynote from Michael Bronstein.

Michael is going to be one of the keynote speakers at ECML PKDD 2023 in September in Torino - the list of accepted workshops should appear soon, so far we know about the Workshop on Learning and Mining with Blockchains. If you fancy Lisboa in September, you might want to submit to the Special Track on AI on Networks for Social Good, part of the ACM Conference on Information Technology for Social Good. Thanks to Manuel Dileo for the pointers 👏

For the weekend reading, have a look at:

Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes by Simran Arora and Christopher Ré’s lab

When Do Graph Neural Networks Help with Node Classification: Investigating the Homophily Principle on Node Distinguishability by Sitao Luan feat. Jure Leskovec and Doina Precup

An Exploration of Conditioning Methods in Graph Neural Networks by Yeskendir Koishekenov and Erik J. Bekkers

Graph ML News (April 29th) - Upcoming ICLR and Accepted ICML papers

ICLR in Kigali starts next week! There is going to be a flurry of materials and reviews prepared by small and big labs, for instance, A Guide to ICLR 2023 — 10 Topics and 50 papers you shouldn't miss - so we’ll try to keep you updated. Meanwhile, the Machine Learning for Drug Discovery (MLDD) and ML4Materials workshops announced accepted papers - those are nice venues to see where the community moves and what would be next major conference submissions.

More resources on topology: 🍩 Database of Original & Non-Theoretical Uses of Topology (DONUT) - a collection of TDA applications beyond machine learning. TopoEmbedX - a python library for working with topological data, pretty much networkx for higher-order structures. Following that, a fresh talk on the Curvature for Graph Learning by Bastian Rieck!

Finally, ICML acceptances have arrived - some particularly interesting preprints that made it to the conference include:

- Graph Neural Networks can Recover the Hidden Features Solely from the Graph Structure
- STRIDERNET: A Graph Reinforcement Learning Approach to Optimize Atomic Structures on Rough Energy Landscapes
- MoleculeSDE - A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining (a project website so far)
- GREAD: Graph Neural Reaction-Diffusion Networks
- On the Expressive Power of Geometric GNNs
- Improved Graph HyperNetwork (GHN-3)

​​GraphML News (April 23rd) - Topological Deep Learning, Scalable Molecular Simulations, Network Games

Architectures of Topological Deep Learning: A Survey on Topological Neural Networks by Mathilde Papillon, Sophia Sanborn, Mustafa Hajij, and Nina Miolane - a wonderful survey on Topological Deep Learning explaining basic concepts from sets and graphs to simplicial and cellular complexes using message passing framework. The survey also covers prominent deep learning architectures employing topological features and tasks that benefit from them. Must read 👍

Scaling the leading accuracy of deep equivariant models to biomolecular simulations of realistic size by Albert Musaelian, Anders Johansson, Simon Batzner, Boris Kozinsky - the work introduces Allegro v2, an improved version of the SOTA equivariant model Allegro, probed on the humongous problem scale: nanoseconds of the full HIV capsid (44M atoms) and scaling up to 100M atom structures on 5120 A100 GPUs 👀.

New blogs:

Michael Bronstein and Emanuele Rossi wrote an article on Learning Network Games - an intersection of the game theory and Graph ML. The main task is to infer the network structure between the agents in a game based on the observations of actions and outcomes.

Not directly about graphs, but Shashank Prasanna wrote an intro to torch.compile() introduced in PyTorch 2.0 and what’s happening under the hood when you execute it on your model.

GraphML News, April 16th edition - Generalist Medical AI, more diffusion papers

No particularly outstanding Graph ML event or announcement (that we hadn’t covered before) happened this week, so here is a collection of fresh papers you might want to have a look at:

Foundation models for generalist medical artificial intelligence - perhaps a landmark paper on using foundation models and many its exciting applications like generative models (eg, text-to-molecule or text-to-protein) in real world medicine.

DiffDock-PP: Rigid Protein-Protein Docking with Diffusion Models - extension of the famous DiffDock that translates and rotates unbound protein structures into their bound conformations.

Graph Generation with Destination-Driven Diffusion Mixture - the next version of the score-matching GDSS generative model (ICML 2022). Here, the model learns to “keep in mind” the final destination of the diffusion process at each time step - this trick greatly improves the performance in 2D and 3D tasks.

DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization - turns out discrete diffusion on graphs is able to generate very strong priors for combinatorial optimization tasks like Traveling Salesman or Maximum Independent Set when paired with a postprocessing solver.

GraphGUIDE: interpretable and controllable conditional graph generation with discrete Bernoulli diffusion - another take on discrete diffusion on graphs where authors define Bernoulli noising process as adding/removing/flipping edges instead of marginal transition probabilities mined from data (like in DiGress). Strength of that approach is that any intermediate state with added noise is still a legit graph retaining its sparsity instead of adding direct noise to node features or adjacency matrix.

Graph ML News, April 8th edition - MoML’23, GLB’23, and more

Molecular Machine Learning Conference (MoML) 2023 is going to take place at Mila in Montreal on May 29th. MoML is the premier venue for ML applications in drug discovery, quantum chemistry, molecular dynamics, and protein design. Confirmed speakers are Yoshua Bengio (Mila), Djork-Arné Clevert (Pfizer), Marinka Zitnik (Harvard), Gregory Bowman (UPenn), Mohammed AlQuraishi (Columbia), and Dominique Beaini (Mila, Valence Discovery). Posters submission deadline is April 24th, The ‘22 event was held at MIT and was a huge success!

In this context, University of Amsterdam (UvA) announced 4 open postdoc positions in the new program on AI 4 Molecules & Materials.

The Workshop on Graph Learning Benchmarks (GLB’23) will be held in conjunction with KDD 2023 in Long Beach (California) on Aug 7th. Submit your works on new graph datasets, benchmarks, and software until May 26th. The workshop is non-archival.

PyG expands the range of supported hardware to Graphcore IPUs with examples on training temporal GNNs, molecular property prediction GNNs, and inductive KG reasoning GNNs on IPUs. Following up on that, you might want to attend the GNN meetup organized by Graphcore and Kumo in London on April 13th next week.

For the weekend reading, check out EigenFold: Generative Protein Structure Prediction with Diffusion Models by Bowen Jing, Ezra Erives, Peter Pao-Huang, Gabriele Corso, Bonnie Berger, and Tommi Jaakkola. The take on protein tasks by the authors of DiffDock 😉

Graph ML News, April 1st edition

Apart from Neural Graph Databases and Twitter Algorithm (and SIGBOVIK), a few more things happened this week.

The Learning on Graphs Conference (LoG) 2023 has been announced! One of the most premiere graph learning venues is going to take place online on Nov 27-30th accompanied by local meetups, you can actually volunteer and organize it at your place!

Baker Lab open-sourced RF Diffusion, a SOTA protein generation model, as part of ColabFold. We covered RF Diffusion a few months ago and its capabilities are quite astounding. Since the time of announcement, the authors further improved the quality and managed to test hundreds of generated proteins in the wet lab to test their properties.

ICML 2023 announced accepted workshops - the graph learning audience might want to attend:

- Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators
- Topology, Algebra, and Geometry in Machine Learning (TAG-ML)
- Knowledge and Logical Reasoning in the Era of Data-driven Learning
- Sampling and Optimization in Discrete Space
- The Synergy of Scientific and Machine Learning Modelling (SynS & ML)
- Workshop on Computational Biology
- Structured Probabilistic Inference and Generative Modeling

Rishi Puri and Matthias Fey published a post on accelerating Heterogeneous Graph Transformers in pyg-lib resulting in about 3x speed boost. Meanwhile, AWS Labs released GraphStorm, a Graph ML framework for enterprise use-cases based on DGL.

For the weekend reading, check out Machine Learning for Partial Differential Equations by Steven L. Brunton and J. Nathan Kutz - perhaps the best intro into ML with PDEs. Yes, it is from the author of awesome YouTube lectures on dynamical systems, physics-inspired ML, and control theory.

​​🐦 Special: Graph algorithms behind The Twitter Algorithm

Twitter has recently published some details on their tweet recommendation algorithm (denoted as The Algorithm). Let’s dive into it from the graph learning perspective - it does have some interesting features spanning clustering, KG embeddings, ANN, and PageRank.

Data-wise, the GraphJet framework operates on the Twitter interaction graph (in-memory) supporting dynamic edge updates and lookup queries. Several algorithms prepare features:

- Graph clustering based on sparse binary factorization (SBF) to mine communities, and then the SimClusters approximate nearest neighbor search library to query for the most similar clusters. There are approximately 145k communities on Twitter and they are updated every few weeks.

- Twitter Heterogeneous Information Network (TwHIN) embedding - this is largely based on the classic TransE for knowledge graph embedding. The KG is a multi-relational graph among Users, Tweets, Ads, and Advertisers. TwHIN learns shallow embeddings for all nodes. For inductive capabilities — building embeddings for newly arrived tweets or users — the model simply aggregates embeddings of neighboring nodes (my 2 cents - NodePiece would fit pretty well into this setup).

- RealGraph models the user interactions graph and outputs the likelihood of two users’ interaction. There is a relatively straightforward logistic regression model for edge scoring on top of the RealGraph.

- TweepCred - a PageRank score for users, this is your “influencer” score.

In your feed, 50% of tweets come from your network (RealGraph features), 50% from out-of-network (SimCluster, TwHIN, and Social Graph traversals). 1500 candidates are sent to the ranking models: a lightweight logreg and heavier 48M-param neural net based on MaskNet. Ranked candidates are subject to filtering and postprocessing.

Overall, the recommender pipeline runs about 5 billion times a day, so the latency requirements do play a major role in selecting shallow’ish graph models. Check the repos for more details.

We’ll leave other peculiarities like “the Elon feature” for other researchers 🙂