Smart ideas do not come in the only head. FastNeRF has the same idea as in NeX, but a bit different implementation.
The main idea is to factorize the voxel color representation into two independent components: one that depends only on positions
p=(x,y,z) of the voxel and one that depends only on the ray directions v.Essentially you predict K different (R,G,B) values for ever voxel and K weighting scalars
H_i(v) for each of them:color(x,y,z) = RGB_1 * H_1 + RGB_2 * H_2 + ... + RGB_K * H_K.Then 2 neural networks
f(x,y,z) = [RGB_1, ... RGB_K] and h(v) = [H1, .. H_K] are learn to predict color components an their weights. After that the values of these functions are cached for every voxel in the volume which enables swift online rendering.⚔️ NeX(➖) vs FastNeRF(➕):
➖ NeX achieves ~60Fps and PSNR 27 on the Real Forward-Facing dataset using Nvidia 2080Ti . While FastNeRF - 50-80 fps and PSNR 26 using Nvidia RTX 3090 GPU, fps rate varies from scene to scene. 200fps is only achieved on synthetic datasets or lower resolution.
➖FastNeRF requires a bit more memory to cache their scene representation because NeX uses sparse representation along the depth direction (only 192 slices) and share
RGB_i values between every 12 consecutive depth planes.➖ The same idea - factorize color representation. Predict
K RGB values for every voxel instead of a single one. ➕ FastNeRF attempted to justify such factorization by the Rendering equation (see the image with an intuitive explanation below).
➕ Very similar architecture. NeX has one extra learnable Tensor which represents average RGB colors independent of ray direction. All other components are learned by neural networks. FastNeRF learns everything with neural nets.
➖ NeX has more extensive experiments and also experiments on fixed basis functions (i.e., compute
H_i(v) using the Fourier’s series, spherical harmonics, etc). Interestingly, using the Fourier's series instead of neural network g(v)yields only a slightly worse PSNR score and even better LPIPS score.➖ NeX introduces and evaluated on more challenging Shiny objects dataset. Would be interesting to see the results on FastNeRF on the same dataset as well.
❗️👉 Overall, I would say the approaches are very similar with some minor implementation differences. One would need to combine both implementations to get the best result.
🌐 FastNeRF paper
Unfortunately there are video results available and no code of FastNeRF yet.
