H Company Releases NeoMME: A Family of 260M and 800M Single-Tower Multimodal Encoders That Drop the Vision Tower and Causal Decoder

No SigLIP2 tower. No causal decoder. No VLM to repurpose.

Here's how it works. 👇

(1) Raw patches, not vision-tower features Images are split into non-overlapping 32×32 RGB patches and projected by a 2-layer MLP trained from scratch. Text enters through a 256-dimensional factorized embedding. Both then share every Transformer layer. → 16,384-token context, enough for two 3840×2160 4K UHD images

(2) Masked diffusion, not masked language modeling Pretraining is a discrete masked-diffusion text denoiser. Text-only segments draw a corruption rate from U(0,1). Multimodal segments draw from U(0.30,1), which kills the "guess it from the surrounding words" shortcut. → +38.4 points masked-token accuracy from visible page patches at 90% masking (260M)

(3) Trained from scratch on a small budget About 524B packed input tokens, roughly 290B of them text-only. ModernBERT saw around 2T text tokens. They used the NorMuon optimizer to squeeze more out of the smaller budget. → 16 H100s for the 260M run, 32 for the 800M

Full analysis: https://www.marktechpost.com/2026/09/06/h-company-releases-neomme-a-family-of-260m-and-800m-single-tower-multimodal-encoders-that-drop-the-vision-tower-and-causal-decoder/

Paper: https://arxiv.org/pdf/2609.01657

Technical details: https://huggingface.co/blog/Hcompany/neomme?

HF: https://huggingface.co/collections/Hcompany/neomme