Gradient Dude

⚙️ Model architecture:
z_t is a time-variant learnable 1024-dimensional latent code. The rest is almost the same as in NERF.

🔪 Limitations:
- Training requires time-synchronized input videos from multiple static cameras with known intrinsic and extrinsic parameters.
- Training for a single 10-seconds video is still quite slow for any real-life application: It takes about a week with 8 x V100 GPUs (~1300 GPU hours).
- Blur in the moving regions in highly dynamic scenes with large and fast motions.
- Apparent artifacts when rendering from viewpoints beyond the bounds of the training views (baseline NERF model has the same problems).

Neural 3D Video Synthesis
Facebook Reality Labs

These guys created a novel time-conditioned Neural Radiance Fields. The results are impressive. When it gets faster, it will enable mind-blowing applications!

It is a sort of extension of NeRF model for videos. The model learns to generate video frames conditioned on position, view direction and time-variant latent code.
Temporal latent codes are optimized jointly with other network parameters.
NeRF model is notoriously slow and requires a long training time. Training separate NERF models for every frame requires ~15K GPU hours, while the proposed model - only 1.3K GPU hours.

📝 Paper: https://arxiv.org/abs/2103.02597
🌐 Project page: https://neural-3d-video.github.io

Hey Rebyata!

I brought you some good news for software engineers w/o an academic background in AI! 🦾
Yesterday Facebook AI launched a new program to draw more diverse talent into AI - Rotational AI Science & Engineering (RAISE).

RAISE is a 24-month full-time research engineering position in Facebook AI. You will work at 4 different AI teams (6 months at each) during that time and towards the program en, you will select a permanent team in Facebook AI.

Applications deadline: May 10th, 2021 @ 5pm PT.
https://ai.facebook.com/blog/facebook-ais-raise-program-an-innovative-new-career-pathway-into-ai

GANsformer or StyleGAN2 on steroids
Facebook AI Research

GANsformer is a novel and efficient type of transformer, explored for the task of image generation. The network employs a bipartite structure that enables long-range interactions across the image, while maintaining computation of linearly efficiency, that can readily scale to high-resolution synthesis. The model iteratively propagates information from a set of latent variables to the evolving visual features and vice versa, to support the refinement of each in light of the other and encourage the emergence of compositional representations of objects and scenes. In contrast to the classic transformer architecture, it utilizes multiplicative integration that allows flexible region-based modulation, and can thus be seen as a generalization of the successful StyleGAN network.

Authors promise to release the pretrained models soon.

📓 arxiv.org/abs/2103.01209
⚒github.com/dorarad/gansformer

😅

This is just awesome 😀

VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation

I continue discussing deep learning approaches for self-driving cars.
Future motion prediction is a task of paramount importance for autonomous driving. For a self-driving car to safely operate it is crucial to be able to anticipate the actions of other agents on the road.

In this video, I explain VectorNet - one of the methods for future motion prediction based on the vectorized representation of the scene instead of RGB images.

▶️YouTube Video
📝Paper

Stable View Synthesis (by Vladlen Koltun from Intel)

Given a set of source images depicting a scene from arbitrary viewpoints, it synthesizes new views of the scene.

The method operates on a geometric scaffold computed via structure-from-motion and multi-view stereo. Each point on this 3D scaffold is associated with view rays and corresponding feature vectors that encode the appearance of this point in the input images. The core of SVS is view-dependent on-surface feature aggregation, in which directional feature vectors at each 3D point are processed to produce a new feature vector for a ray that maps this point into the new target view. The target view is then rendered by a convolutional network from a tensor of features synthesized in this way for all pixels. The method is trained end-to-end.

The results are magnificent!

Source code
Paper

Guys from RunwayML created an awesome user-friendly demo for our approach "Adaptive Style Transfer".

You can play around with it and easily stylize your own photos. One important thing: the larger an input image, the more crispy becomes a stylization.

Run Models for 8 different artists
Run Picasso model
Run Van Gogh model

Method source code on Github: https://github.com/CompVis/adaptive-style-transfer

A talk on Theoretical Foundations of Graph Neural Networks by Petar Veličković from DeepMind.

In this talk Petar derives GNNs from first principles, motivates their use in the sciences, and explain how they emerged along several research lines.
Should be very interesting for those who wanted to learn about GNNs but could not find a good starting point.

Video: https://youtu.be/uF53xsT7mjc
Slides: https://petar-v.com/talks/GNN-Wednesday.pdf

How does Bayesian optimization help to find the proper hyperparameters for a machine learning model?

Bayesian optimization works by constructing a posterior distribution of the objective function (Gaussian process) and use it to select the most promising hyperparameters to evaluate.
As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in the parameter space are worth exploring, and which are not.

Good blogposts to learn about Bayesian optimization: [at towardsdatascience] [at research.fb.com]

Google open-sourced its AutoML framework for model architecture search at scale.
It helps to find the right model architecture for any classification problems (i.e., CNN with different types of layers).
Now you can write fit(); predict() and call it a day! Of course, in case you have enough GPUs 🙊😅

You can define your own model building blocks to use for search as well.
The framework uses Bayesian optimization to find proper hyperparameters and can build an ensemble of the models.
Works both for table and image data.

https://github.com/google/model_search

Hi guys! New video on my YouTube channel!

I this video I give intuition behind self-supervised representation learning (also easy to understand for beginners).

You will learn how to learn useful representation from just a bunch of unlabeled images.
I will explain CliqueCNN method which builds compact cliques for classification as a pretext task and give an overview of other recent self-supervised learning approaches.

https://youtu.be/DEm6pDyYbt4

#beginners_guide
Learn About Transformers: A Recipe

A blogpost summarizing key study material to learn about the Transformer models (theory + code).
Tasty!

Graph Representation Learning Book 🦾

A brief but comprehensive introduction to graph representation learning, including methods for embedding graph data, graph neural networks, and deep generative models of graphs.

https://cs.mcgill.ca/~wlh/grl_book/