GraphGPS: Navigating Graph Transformers
Invited post by Ladislav Rampášek

In 2021, graph transformers (GT) won recent molecular property prediction challenges thanks to alleviating many issues pertaining to vanilla message passing GNNs. Here, we try to organize numerous freshly developed GT models into a single GraphGPS framework to enable general, powerful, and scalable graph transformers with linear complexity for all types of Graph ML tasks.

With GraphGPS, we managed to scale Graph Transformers to much larger graphs and get SOTA in several competitive benchmarks, e.g. 0.07 MAE on ZINC. Positional and structural embeddings are necessary for graph Transformers, encoding “where” a node is and “how” its neighborhood looks like, respectively. Bonus: they even make MPNNs provably more powerful! We organize them into local, global, and relative types.

Key observation: It is better to combine an MPNN and Transformer layer together into one: helps with over-smoothing, and allows for plug & play linear global attention, e.g., Performer. In fact, linear attention enables graph transformers to scale to dramatically larger graphs compared to typical molecules - we confirm it easily works on graphs with 5K nodes without any special batching!

Putting these 3 ingredients together: positional/structural encodings, choice of MPNN and Transformer layer combined into one layer, gives the blueprint for our GraphGPS: General, Powerful, Scalable graph Transformer. Plain numbers:

🚀 400% faster than previous graph transformers;
📈 Scaling to batches of graphs up to 10,000 nodes each thanks to linear attention models;
🛠 The GraphGPS library allows co combine any MPNN with any Transformer and any positional/structural encoding.

Find more details in:
- Medium blog post with a deep-dive into GraphGPS: https://mgalkin.medium.com/graphgps-navigating-graph-transformers-c2cc223a051c
- arxiv preprint: https://arxiv.org/abs/2205.12454
- Github repo: https://github.com/rampasek/GraphGPS