Linkstream

Reviewer #2, step back!
82% of authors found the GPT-4 feedback more useful than feedback from (at least some) human reviewers

https://arxiv.org/abs/2310.01783

Oh. https://ml-jku.github.io/hopfield-layers/
via @kuu_channel

It’s beautiful. I wonder where’s the catch. I.e. why Llama et al don’t use Hopfield layers

Mistral-7B is seriously good. And fast. And works with llama.cpp already!
(except for the sliding window attention part, so limited to 4-8K or so)
https://twitter.com/arthurmensch/status/1707041835091165580

A stunning example of how efficient market and arbitrage trades enabled by mobile phones help to avoid waste, stabilize prices and increase welfare (Fish markets in Kerala, India, 1997-2001)
https://www.jstor.org/stable/25098864

https://blog.isosceles.com/the-webp-0day/ TLDR fixed in upstream, will be with us w/libwebp for some time

NExT-GPT: Any-to-Any Multimodal LLM
TLDR: Vicuna (fine-tuned Llama) extended with basic image, audio and video understanding and generation. Impressive demo videos, the Gradio-based demo is currently broken though. Anybody up to deploy this somewhere to play with?
https://next-gpt.github.io/ + https://arxiv.org/pdf/2309.05519.pdf

ASMesh: Anonymous and Secure Messaging in Mesh Networks Using Stronger, Anonymous Double Ratchet
https://eprint.iacr.org/2023/1053

interesting; though unclear re:traffic/metadata analysis and power consumption as the protocol is rather chatty

lucid and useful example of the beginner's mindset https://www.oneusefulthing.org/p/embracing-weirdness-what-it-means

https://arxiv.org/abs/2308.16898
Transformers as Support Vector Machines
TLDR ~ transformer layers are SVMs w/gradient-trainable global convergence, when a) overparameterized b) have nonlinear heads

my remark: this explains
1) why huge models are important (so the gradient is high-dimensional enough to be monotonic)
2) why attention (aka connections, aka indirections) is trainable at all;
and says nothing about why they might generalize the dataset

Transformers ain't everything
tree-based methods still outperform DL on tabular data and medium (~10k samples) datasets

https://hal.science/hal-03723551

Teaching Arithmetic to Small Transformers
an interesting detailed write-up

https://arxiv.org/abs/2307.03381

Pushing the Limits of Machine Design: Automated CPU Design with AI
By efficiently exploring a search space of unprecedented size 10^10^540 (note: reducible to 10^6), and thus pushing the limits of machine design, our approach generates an industrial-scale RISC-V CPU within only 5 hours.

> which sheds some light on building a self-evolving machine to beat the CPU designed by humanity eventually.
> The automatically designed CPU was sent to the manufacturer in December 2021
https://arxiv.org/abs/2306.12456

Masked Trajectory Models for Prediction, Representation, and Control

TLDR: Transformers using state space and action embeddings as tokens are better at RL than, um, RL algorithms. Oops.

https://arxiv.org/abs/2305.02968

why do they call this an ‘attack’? this is the way to set the model free!

(TLDR: DAN prompt generator)
https://arxiv.org/abs/2307.15043
https://llm-attacks.org

Welcome to the dark side of the cyberpunk.
Once-theoretical timing attacks are now a reality.
(TLDR: Cops still can't decrypt the messages, but they can track who's chatting with whom comparing the small spikes of traffic as the message gets delivered)
https://wz.ax/timing-is-real

https://www.youtube.com/watch?v=1GanrexRyVY oh wow

phi-1: with datasets of higher quality, model with 1.3B parameters and 7B tokens can be quite competitive to gpt4 and other 100x larger models on coding tasks
https://arxiv.org/abs/2306.11644

https://arxiv.org/abs/2305.07759
TinyStories: 3-30M (not G) parameter model with coherent English from a curated dataset.

Don't expect it to code but curious if this is usable as a LoRA or similar baseline - also need to look closer at their tokenizer setup, must be way different from GPT

Insect–Machine Interface Based Neurocybernetics
Spy bugs! 2009!
https://wz.ax/cybugs09