Data eXplore : Data Science, ML, Big Data, LLMs and AI Security

Pre-processing and decoding in LLM inference

Have you ever wondered why first token always appears with a delay, while rest of stream proceeds almost instantly?

It's not network latency or model warmup, it's a structural property of how LLMs actually execute.

Inference consists of two phases that use the same model and execution path, but the workload in each phase is fundamentally different, and the bottlenecks are opposite.

𝗣𝗿𝗲𝗳𝗶𝗹𝗹 — this is the request processing phase. The model processes all input tokens in a single parallel pass, computing Q, K, and V for all tokens at once.

The attention mechanism is implemented as a large matrix operation, for which GPUs are optimized, so the computational units are heavily loaded and the chip operates at the limit of its arithmetic throughput.

Pre-processing is memory-bound, and the metric that reflects this is the time to the first token.

𝗗𝗲𝗰𝗼𝗱𝗲 starts after the first token appears. To generate the next one, the model calculates Q, K, and V only for the new token, because everything previous is already cached.

Then comes the "one token - one pass" cycle: the new query is multiplied by the already stored keys instead of the full matrix, and the computational volume becomes small.

However, the GPU still has to read all weights and the entire cache from memory to perform even this small operation, so the memory bandwidth becomes the bottleneck, and the computational units are idle.

Decoding is memory-bound, and the metric here is the delay between tokens.

This separation explains a number of effects that seem non-obvious from the outside.

The GPU load is high during pre-processing and drops sharply during decoding, because in the second phase, the memory becomes the limiting factor rather than the computations.

Adding computational power often doesn't help with slow generation, because for memory-bound workloads, the solution is faster memory or a smaller cache, not more FLOPs.

A long context slows down generation disproportionately, because the key and value caches grow with each token, and each step of decoding must read them all.

This cache is a key optimization, without which decoding would be impossible, because the attention would have to be recalculated for the entire growing sequence at each step.

With the cache, it's built once during pre-processing and then expanded by one element for each new token, reusing already computed values.

However, the cache is stored in GPU memory and grows linearly with the sequence length. For a 13B model, this is about 1 MB per token, so a 4K context occupies about 4 GB of video memory just for the cache.

Therefore, a long context feels slow not because of the "lack of model power," but because of the memory pressure.

Currently, the industry is optimizing this limitation through quantized caches, sliding windows, grouped attention, and PagedAttention, while the DeepSeek V4 series goes further and redesigns the attention mechanism itself to make the cache smaller from the start.

When attention starts being redesigned for memory constraints, it means that the limitation has shifted towards memory.

Practical takeaway: if the model seems slow, it's important to distinguish — is it slow starting or slow streaming? A slow start corresponds to pre-processing and computational bottlenecks, while slow streaming corresponds to decoding and memory limitations.

Read further material that breaks down LLM inference from scratch: tokenization, embeddings, attention, the separation of pre-processing and decoding, key/value caches and quantization.

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

In an interview about ML engineering at Apple, question is asked:

There are two models with an accuracy of 88%.

- Model A has a confidence of 89%
- Model B has a confidence of 99%

Which would you choose?

The ANSWER "any of them, they have the same accuracy" ends the interview.

➡️ what's missing?

Modern neural networks often mislead.

They give an overconfidence in predictions.

For example, in one experiment on the CIFAR-100 dataset, LeNet and ResNet were compared.

LeNet:

- accuracy ≈ 0.55
- average confidence ≈ 0.54

ResNet:

- accuracy ≈ 0.7
- average confidence ≈ 0.9

Despite the higher accuracy, ResNet is overconfident in its predictions. The model believes it's right with a 90% probability, but the actual accuracy is about 70%.

Calibration solves this problem.

A model is considered calibrated if the probabilities of predictions correspond to the real outcomes.

For example: if the model gives a probability of 70%, then in about 70% of cases, the event should actually occur.

This is important because such models are used in decision-making.

A poorly calibrated but confident model can give critically misleading results.

Example: a state hospital plans expensive medical tests.

A realistic assessment of probabilities helps optimally allocate the budget and make decisions.

If the model is not calibrated, it will give overly confident predictions.

Reliability diagrams are used to visually check calibration.

They show the dependence of the actual accuracy on the predicted confidence (softmax values).

An ideally calibrated model gives a line y = x.

They also use a scalar metric - expected calibration error (ECE).

One of its approximations is to divide the predictions into intervals and average the difference between accuracy and confidence across these bins.

The main methods of model calibration:

For binary classification:

- histogram binning
- isotonic regression
- Platt scaling

For multi-class classification:

- binning
- matrix and vector scaling

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

CocoIndex v1: Release of an incremental engine for agents.

CocoIndex released first stable version of its engine for building data for long-acting agents - those that serve RAG, knowledge graphs, memory, and context in production systems.

CocoIndex is a specialized open-source incremental ETL (Extract, Transform, Load) engine for building AI systems. It is used to automate data processing and instant indexing into vector databases or knowledge graphs.

➡️ What are the Change?

The tool is popular for its ability to update information in real time: as soon as the source data changes, CocoIndex recalculates only the necessary parts of the index, eliminating the need for a full and costly re-indexing of the entire database.

➜ Main change of v1 is the complete abandonment of the DSL

The entire pipeline is now described by ordinary asynchronous Python functions that call each other. The engine continues to track changes and materialize target states, but does so via the native Python API, rather than a separate type system.

The authors were inspired by the thesis of Jeff Dean and Bill Dally from GTC 2026: agents work about 50 times faster than a human, but rely on tools designed for human pace.

Nightly rebuilds of indexes in this logic become a problem - a need for an engine that synchronizes derived data with the source incrementally, reprocessing only changed chunks and overwriting only changed rows.

➜ In addition to the abandonment of the DSL, the release brought three more changes

⁠☞ Firstly, the engine uses Python's own type system: PIL.Image, pyarrow.Table, torch.Tensor, and any class from an imported library can be passed directly to functions without wrappers and bidirectional conversion.

⁠☞ Secondly, Postgres is no longer needed - the engine's state is stored in a single local file. Postgres remains a full-fledged target, it just ceased to be a mandatory dependency.

⁠☞ Thirdly, sources and targets are created at runtime: you can mount a separate target for each tenant, build a topology from the rows of a configuration table, or connect a Kafka topic via a feature flag.

➜ Core is still in Rust

All the hot logic for detecting and applying changes lives there. At the Python level, a decorator connects a function to change tracking, and a separate flag caches its result by the hash of the arguments and code: a change in a helper invalidates only those callers that actually depend on it.

➜ Contract of managed targets has been preserved

The developer declares what a table, graph, or directory should look like, and CocoIndex itself performs create/alter/drop for containers and insert/update/delete for content, including deleting orphaned objects when the schema changes. If you stop declaring an entity, it disappears from the target.

The contract works identically for Postgres, LanceDB, Neo4j, Kafka, S3, and regular files on disk.

Examples of pipelines, from embedding code in LanceDB and processing PDFs to building a knowledge graph from conversations, are in the repository on GitHub, alongwith Documentation and You Tube.

#ML #ETL #RAG #Agents #СocoIndex

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

Madmen implemented MicroGPT by Andrej Karpathy entirely in FPGA logic.

Without a graphics processor. Without PyTorch. Without inference loops on the central processor.

Just a transformer, embedded in hardware, generating 50,000+ tokens per second.

The model is small, but the point is not that. The point is that inference does not have to exist only in a software environment.

The goal was not to create the largest possible model.

The goal was to present the entire path of transformer inference in a form readable for hardware: memory, counters, state machines, accumulators, lookup tables, and multi-cycle arithmetic blocks.

The base scheme uses fixed Q4.12 arithmetic and weights stored in ROM.
Most of the model boils down to one repetitive operation: matrix-vector multiplication. Therefore, a reusable 16-channel stream block for matrix-vector calculations was implemented, and then it was temporarily multiplexed to Q/K/V, MLP, and the output layer of the language model.

The most interesting was the attention mechanism.

In Python, it's a single neat equation.

In RTL, it turns into a schedule: generation of Q/K/V, passage through scalar products, tracking the maximum, approximate calculation of the exponential, accumulation, division, mixing V, then reverse projection.

Source

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

Stanford showed that Claude, GPT, and Gemini only utilize a fraction of the available creative potential.

Just one prompt can unlock a stronger version of the LLM.

When you ask a question, the model calculates multiple response options.

Among them are strong, strange, and groundbreaking ones.

But it almost never outputs them. Due to training through human feedback, the "mode collapse" effect occurs.

The default model defaults to safe, typical, and predictable responses. It knows a stronger option, but prioritizes the safe one.

Researchers described a way to bypass this filter. The method is called Verbalized Sampling.

If you ask for one response, the model selects the most likely one. If you ask to generate 5 options and specify the probability for each, the behavior changes.

The model starts exploring the "tails of the distribution". Instead of 99% predictable responses, less likely but stronger options appear.

In tests, this technique increased diversity and creativity by up to 2.1 times on top models.

Without losing accuracy and safety. 🤖

Article

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

What if we could guarantee that the output of an LLM always matches the expected format?

Classification tasks with LLMs often become messy. Instead of a clear label, you might get "Option A", "Answer: A", or a full explanation.

Transforming this into a normal format requires additional parsing, retraining, and validation, which makes the system fragile.

With Guidance, the select() function constrains the model to return exactly one option from a specified list.

Key advantages:
• ensures that the output corresponds to one of the predefined options
• eliminates the need for parsing code and regular expressions
• works with any list of acceptable values

Article comparing 5 Python tools for structured LLM outputs.

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

8 techniques for getting better answers from LLM

Most people interact with LLM in the same way: ask a question, send it, and work with the result.

This is zero-shot prompting, the basic level. If the answers are not satisfactory, they first improve the prompt, rather than changing the model.

8 prompt engineering techniques:

1. Few-shot prompting: show several input-output examples. The model captures the pattern and applies it to new data.

2. Chain-of-thought (CoT): request a step-by-step reasoning. Breaks complex tasks into verifiable steps.

3. Hierarchy of prompts: system, developer, and user levels with different priorities. The upper levels override the lower ones.

4. Role prompting: assign a role, for example, "you are a security researcher". The model shifts the distribution to the corresponding training data.

5. Negative prompting: explicitly state what not to do. For example, "do not use a marketing style".

6. JSON prompting: specify a JSON schema directly in the prompt. The model returns a structured response according to this schema.

7. Attentive reasoning queries (ARQ): instead of free CoT — structured domain questions. In tests: 90.2% compliance with instructions versus 81.5% for direct prompting.

8. Verbalized sampling: ask the model to generate several variants with probability estimates. It returns diversity suppressed by RLHF.

The techniques combine well: few-shot + CoT, JSON + negative prompting. ARQ is essentially structured CoT for agent scenarios.

Additionally, quality increases with context, tools, and retrieval.

But these 8 techniques are entirely in the prompt — without changing the model, infrastructure, or setup. Only the structure of the request changes.

Here's another article on this topic

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

More an agent "remembers", Less it "knows".

It sounds counterintuitive, but this is a direct consequence of how agents' memory is structured today. Memory inherits the properties of the storage on which it is built.

* A vector database provides associative memory - recognition of familiar patterns.
* A graph provides relational memory - understanding of connections between entities.

➡️ How most agents use the first and ignore the second?

Example:

A study assistant stores three facts in a vector database:

* Mark is in 10th grade.
* The 10th grade has final exams in March.
* The library closes 2 weeks before the exams.

Mark asks: "Will the library be open next week?"

The vector database will return the first and third fact, because the query includes Mark and the library. The middle fact is skipped, as it does not include either Mark or the library.

It is too far in the embedding space to be retrieved in the extracted context. As a result, the agent either responds with incomplete information or completes the answer with a plausible assumption.

This is a typical case. Any query with several reasoning steps goes beyond similarity search.

Increasing the context and retrieving more data is a partial solution. But the accuracy drops by more than 30% if the relevant fact is in the middle of a long context - a classic "lost in the middle" problem.

A large context window does not equal better memory. It's just more space to miss the important thing.

The solution is to stop viewing memory as a single storage and move to three layers:

* Relational layer: stores the source of the fact, the time of recording, and access. The data origin layer.
* Vector layer: stores the semantics and similarity. The retrieval layer.
* Graph layer: stores the connections and dependencies. The reasoning layer.

Each layer fulfills its task:

* Vector database without a graph - similarity without connections.
* Graph without a vector layer - connections without semantic search.
* Relational storage - accounting for the source without the ability to reason.

In the open-source project Cognee, this approach is implemented in practice.

It uses the ECL (Extract, Cognify, Load) pipeline, which in one pass writes data to all three storages and synchronizes them when new data arrives. Vectors and graph edges are built immediately at the indexing stage.

Additionally:

1. Entity resolution: you can specify a domain dictionary, and the system merges duplicates.
For example, "car manufacturer", "automotive manufacturer", and "automotive concern" are reduced to one canonical entity.

2. Local mode by default: installation via pip, everything works locally. For production, you can switch to Postgres and Neo4j without changing the API.


Project co-founder described this approach from scratch and built a full-fledged agent based on Cognee.

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

Meta solved main bottleneck in RAG.

30X decoding speedup… No accuracy loss…

A problem that's hardly ever mentioned:

When you feed 80 extracted fragments into LLM, only 5–10 are actually useful.

Rest is dead weight. But attention is counted for ALL of them.

Math is harsh: Traditional RAG with 16K context:
→ 100+ seconds to first token
→ 10× drop in throughput
→ huge memory consumption

➡️ How REFRAG solves problem?

What REFRAG does?
Compresses context chunks into single embeddings.

Instead of processing 16,384 tokens → processes 1,024 chunk embeddings.

Results:
✓ 30.85× faster time to first token
✓ zero loss of perplexity
✓ 16× context expansion (4K → 64K tokens)
✓ 3.75× better than previous SOTA

Why it works?
RAG contexts have sparse attention patterns. Most extracted fragments don't interact with each other. REFRAG exploits this through:

1. Precomputed embeddings - cached at extraction stage and reused during inference
2. Reinforcement learning-based compression - a policy decides what to compress
3. Works at any position - unlike previous approaches

Practical impact:
• 8 fragments with the same latency as one
• higher accuracy with weaker retrieval models
• supports unlimited dialogue history
• no model architecture changes needed

This changes economics of RAG: more context with lower latency.

Article

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

LLM break all the assumptions on which classical inference in ML was built.

Traditional model (convolutional neural network, transformer classifier, XGBoost) outputs a result in a single forward pass.

Nothing is stored between requests. The graphics processor performs the same type of work each time.

➡️ How LLM work completely differently?

The output is generated one token at a time, autoregressively, which turns a single request into hundreds of consecutive forward passes.

The prefill stage relies on computations, and decoding relies on memory bandwidth, and running them together on a single graphics processor degrades both stages.

The key-value cache grows with the length of the dialogue and is shared between requests, so routing is no longer about the least loaded servers, but about which replica already has the needed prefix cached.

Models with a mix of experts add parallelism of experts on top of this.

None of this is in classical machine learning model serving.

That's why a separate stack of optimizations specifically for inference of large language models has emerged: compression, attention mechanism, key-value cache management, batching, decoding, parallelism, and routing.

The demo shows a photo map of 72 techniques for optimizing large language models in production, grouped into nine blocks.

I also attach an article that explains how inference of large language models differs from classical inference and why each of these blocks is needed.

Question: what other techniques for optimizing large language models would you add here?

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

For 10 years, we've been increasing width and depth of models - but we've hardly changed the way layers interact with each other.

Since ResNet with x + F(x) in 2015, deep residual connections have remained the only channel for inter-layer communication.

On paper, models look deep. But many layers "learn to be silent": as information accumulates, the signal becomes blurred and lost.

➡️ The bottleneck isn't computations within layers, but communication between them.

All previous approaches - DenseNet, DenseFormer, Hyper-Connections, MUDDFormer - answer the same question: "how to better merge the outputs of layers?"

Better coefficients. More channels. Adaptive weights.

But all of this is accumulation. And this is a categorical mistake.

Inter-layer communication should be extraction, not accumulation.

Request = "what do I need".
Key = "what do I have".
Both sides participate.

Layer 152 shouldn't "strain" to hear Layer 3 in the general chorus. It should simply ask him: "what did you say?"

Problem: naive implementation of depth attention took 44,924 ms for forward and backward passes. Too slow.

Introducing Flash Depth Attention (FDA) - a hardware-efficient core that accelerates depth attention by more than 40,000 times, making full-scale extraction of depth suitable for large-scale training.

Classic transformer pipeline: residual connections → sequential attention → residual connections → fully connected layer.

Pipeline with Flash Depth Attention (FDA): depth attention → sequential attention → depth attention → fully connected layer.

Next - Mixture-of-Depths Attention (MoDA): combining depth and sequential extraction into a single softmax.

Each head simultaneously accesses the KV of the current layer (sequentially) and the KV of all previous layers (depth-wise).

One operation, two dimensions of extraction.

Results: the model actively uses inter-layer extraction, the "attention sink" effect disappears, MoDA improves the baseline model OLMo2 on all metrics.

The first half of architecture development was about scaling components.
The second - about scaling communication.

Welcome to second half: Article, Blog (recommended), Code

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

You're at interview for an ML Engineer

Interviewer asks: We need a language model that works well in code, mathematics, and creative writing.
How to achieve multi-domain performance?

You: I'll increase the number of attention heads.

interview ends…

➡️ What you missed?

Attention heads capture patterns, not domain expertise.

More heads = richer representations in a single pass.
More experts = dedicated subnets for different types of knowledge.

The correct answer: Mixture of Experts (MoE).

Let's break down how MoE differs from standard transformers:

Transformers and MoE differ in the decoder block:

- Transformers use a fully connected feed-forward network.
- MoE uses experts — also fully connected feed-forward networks, but smaller in size compared to transformers.

During inference, a subset of experts is selected. This speeds up inference in MoE.

Since the network contains multiple decoder layers:

- the text passes through different experts at different layers
- the selected experts also differ for different tokens

But how does the model decide which experts are better suited?

This is done by the router. Let's break it down further.

Task 1) Note this pattern at the beginning of training:

- the model selects "Expert 2"
- the expert slightly improves
- it might be selected again
- the expert trains further
- it's selected again
- it continues to train
- and so on

Many experts remain undertrained.

We solve this in two steps:

- Add noise to the output of the fully connected layer of the router so that other experts can receive higher logits.
- Set all logits except the top-K to -infinity. After softmax, their values become zero.

This way, other experts also get the opportunity to train.

Task 2) Some experts might receive more tokens than others — this leads to some experts remaining undertrained.

This is prevented by limiting the number of tokens that a single expert can process.

If an expert reaches the limit, the input token is redirected to the next most suitable expert.

MoE contains more parameters to load. However, only a portion of them are activated, as only a limited number of experts are selected.

This leads to faster inference. Mixtral 8x7B from MistralAI is a well-known language model built on MoE.

Visuals on first comment that compares transformers and MoE again

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

Acommon cross-domain memory layer for coding agents (someone proposed)

This idea is called Memory Transfer Learning (MTL).

A large memory pool is assembled from different types of development tasks, after which the agent reuses this memory across domains.

→ Such memory becomes a common resource and a universal library of experience for many agents and models.

An increase (+3.7% on average) is achieved due to meta-knowledge:

* how to validate a solution
* how to structure debugging
* what checks to run
* how to detect failure patterns

At the same time, the level of abstraction is important: memory that is too tied to a specific task degrades quality.

Memory for debugging, code generation, and testing is combined into a single common pool. The more memory, the better the transfer works.

MTL enables the agent to reuse general reasoning and checks, not just exact solution paths.

GitHub, Article

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

Claude Mythos has once again stirred up discussions about whether we have achieved AGI or not.

At the same time, there still isn't a single definition of AGI. Back in 2023, Google DeepMind tried to sort out the chaos and came up with a whole taxonomy with levels ranging from zero to superhuman.

It didn't help much at all.

conclusion: everyone is arguing about whether we have achieved AGI, for which we still can't agree on a definition. Classic.

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

Does your team need an ML process?

Do these questions arise for you as a manager or specialist of an ML team?

• How to help newcomers get up to speed with the processes adopted by the team faster?
• How to consolidate all instructions, standards, rules, recommendations, useful and dangerous practices so that at least someone reads them?
• Is it possible to ensure uniformity in different teams' projects without suffocating them with rules?
• How to share the results of work with another team without unnecessary questions?

If so, this post is for you. And that also means that your team needs an ML process.

An ML process is a meta-instruction that becomes an "entry point" for finding answers to questions about project development. It has the following properties:

• Contains all necessary links to resources for development.
• Explains what to do and what not to do.
• Has a structure that repeats the development process with a comprehensive and up-to-date description of all stages.
• Becomes the basis for creating future project documentation
• Suitable for 95% of processes.
• Simplifies life, rather than imposing restrictions.

An ML process should not and cannot be written by one person, otherwise no one will use it. It's worth gathering a working group from different teams and creating a solution that will be useful and understandable to everyone.

What you will get if you work on an ML process?

• You will see "blind spots" in development and write new instructions that were lacking.
• You will gather in one place a navigator for all resources, tools, and instructions adopted in your team.
• You will facilitate the onboarding of newcomers in your team, the transfer of projects to colleagues, and the understanding of results by the manager.
• You will reduce the time spent on project development.

In the next post, we will provide a template for an ML process, which we use to collect project documentation.

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

Interview Question about LLMs:

You have 80000 agent trajectories from production.
You need to select those that should be sent for review to improve the agent.
It's not possible to use LLMs to evaluate the trajectories.

➡️ How would you solve?

The simplest option is random sampling. We take 100 random trajectories and send them for manual review. But in production, agents usually handle typical requests well, so a significant portion of the annotation budget is wasted on noise.

Another approach is to filter long dialogs, assuming that 10+ user messages indicate a more complex scenario. But long dialogs are heavily skewed towards obvious failures. In the end, you find obvious failures and miss subtle problems in scenarios where the agent formally succeeded.

A recent work from DigitalOcean proposes a different approach: calculating light behavioral signals directly from the trajectory data using deterministic rules on top of dialogs and execution logs.

The signals are divided into three groups:

1. Interaction signals - from the user-agent dialogue.
If the user reformulates the same request or corrects the agent - this is a desynchronization. If the agent repeats without progress - it's stagnation. A user's confirmation that everything worked - satisfaction. All of this is determined through a normalized comparison of phrases and checking the similarity of neighboring replies.

2. Execution signals - from tool calls and runtime events.
A tool call that returns an empty result or does not advance the task is considered a failure. Repeated calls with identical or "drifting" inputs indicate a loop. These signals are easily extracted from structured execution logs.

3. Environment signals - frequency limits, context overflow, API errors.
Useful for diagnostics, but not suitable for learning, as they reflect system limitations rather than the agent's decisions.

Each trajectory is assigned a score based on the signals that triggered, after which the trajectories with the highest score are sent for review.

On the τ-bench, three approaches were compared on 100 trajectories:

- Random sampling - 54% informative
- Heuristic by length - 74%
- Signal approach - 82%

That is, approximately 4 out of 5 selected trajectories are really useful for improving the agent.

Even among dialogs where the agent correctly performed the task, the signal approach found useful patterns in 66.7% of cases compared to 41.3% for random sampling.

Hidden PROBLEMS: policy violations, inefficient use of tools, unnecessary steps. The task is formally completed, but there is potential for optimization.

Entire pipeline works without overhead of LLMs and can continuously operate in production, labeling each trajectory at input.

If you need a practical implementation, this approach is already integrated into Plano - an open-source proxy for AI that combines routing, orchestration, protective restrictions and observability.

GitHub, Article

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

Full global attention vs alternating attention, visual explanation:

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

MIT proven that it's possible to remove 90% of a AI without losing accuracy.

Researchers discovered that within every large model, there is a "winning ticket" - a small subnetwork that performs the main work.

They proved that if you find it and reset it to its initial state, it will work just like the entire large model.

But there was a nuance that immediately killed its practical application...

to find this "ticket", you first need to train the full model. no one wanted to train the model twice for a single deployment. it looked cool in academia, but it was useless in production.

The original 2018 article was truly impressive.

But today, after 8 years... we've finally achieved a breakthrough at the hardware level: structured sparsity

Modern GPUs (NVIDIA Ampere and newer) no longer just "emulate" pruning.

They have native support for block sparsity (2:4 patterns), built directly into the hardware.

This isn't theory - it's silicon-level acceleration.

The math looks very convincing: a network with 90% sparsity = 50% less memory bandwidth + 2× computing bandwidth. Real acceleration without losing accuracy.

Three factors made this ready for production in 2026:

- training with sparsity in mind (the model is initially trained sparse)
- native support in PyTorch 2.0 and Apple Neural Engine
- understanding that AI models are inherently 90% redundant

Evolution complicates systems. We've finally learned to "thin out" them.

The era of bloated and inefficient models is officially over. Tools have finally caught up with theory, and those who stop paying for 90% of the weights they don't need will win.

Future of AI is more compact, faster, and more efficient models.

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

Shannon Lite: An AI penetration tester that breaks your code before production

An autonomous white-box AI penetration tester for web applications and APIs. It doesn't just search for vulnerabilities based on patterns. it reads your source code, finds attack points, and immediately attempts to exploit them.

Key idea: If there's no exploit, there's "no" vulnerability either. Only issues that have actually been reproduced make it into the report. No false positives.

How it works:
🔵Analyzes the application's logic, finds input → sink
🔵Attacks the already running application via browser and CLI
🔵Uses XSS, SSRF, injections, bypasses authentication

Autonomous run:
One run — and it automatically handles logins (including 2FA), bypasses the interface, hits the API, searches for attack surfaces, and simultaneously breaks everything it finds.

Tools: nmap, subfinder, whatweb — not as separate tools, but as part of the pipeline.

Quick start:
npx @keygraph/shannon setup
npx @keygraph/shannon start -u https://your-app.com -r /path/to/repo
Result: a comprehensive report with PoCs and ready-made exploits.

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore

VimRAG is a multimodal RAG agent that maintains context as a memory graph.

Tongyi Lab (Alibaba) released VimRAG, a framework for agent-based RAG that works with text, images, and videos.
The project improves on last year's VRAG-RL and addresses the challenge of multimodal RAG: visual data are token-heavy but semantically sparse, and the classic ReAct history clutters the context with noise and triggers repeated useless queries to the search engine.

➡️ How it works?

Instead of a log of observations, VimRAG models reasoning as a dynamic directed acyclic graph. Each vertex stores a sub-query, an agent's action, a text summary, and a bank of visual tokens.

Edges capture logical dependencies between steps. Such a graph allows the agent to distinguish a dead-end branch from a new hypothesis and avoid getting stuck in repetitive search loops.

➜ Graph-Modulated Visual Memory Encoding operates on top of the graph.

Visual token budgets are allocated based on outgoing degree in the graph, exponential temporal decay (simulating forgetting), and recursive feedback from descendants.

Key fragments receive high resolution, while secondary frames are compressed or discarded. For videos, VLM's ability to tie content to a timeline (extracting key frames by timestamps) is utilized.

➜ Third component is Graph-GPO.

GGPO constructs a critical path from the root to the answer and imposes a gradient mask, excluding dead-end nodes from positive examples and valuable retrievals from negative ones.
On training curves, this leads to faster convergence than the base GSPO without pruning.

➜ Tests

☞ VimRAG outperforms Vanilla RAG, ReAct, VideoRAG, UniversalRAG, and MemAgent on 9 benchmarks (HotpotQA and SQuAD to SlideVQA, MMLongBench, LVBench, and XVBench).

☞ On Qwen3-VL-8B-Instruct, the average score rises from 43.6 to 50.1, and on the 4B version, from 40.6 to 45.2.

However, the average path length is lower than in ReAct and Mem1: structured memory consumes fewer actions per response.

➜ In repository:

☞ the VRAG-RL training framework, a demo of VRAG on the test Qwen2.5-VL-7B-VRAG via vLLM (requires A100 80GB);

☞ a demo on the API Qwen3.5-Plus via DashScope (with visualization of the reasoning DAG and extended rizomics).

The search engine is built on FAISS and supports GVE-3B/7B and Qwen3-VL-Embedding-2B/8B embeddings. Images, PDFs (via conversion), and segmented videos can be indexed.
The VimRAG training code will be released after Alibaba's internal review.

Arxiv, Model, GitHub

#AI #ML #RAG #VRAG #TongyiLab

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore