But indexing ≠ retrieval.
The data you index does not have to be the same data you feed into the LLM during generation.
🟢 What are the 4 smart ways to index data?
1️⃣ Chunk Indexing
→ The most common approach.
→ The document is split into chunks, then each chunk is converted into an embedding and stored in a vector database.
→ At query time, the nearest chunks are retrieved by cosine similarity (or another metric).
Simple and effective, but chunks that are too large or "noisy" can reduce accuracy.
2️⃣ Sub-chunk Indexing
→ Take the original chunks and further split them into smaller sub-chunks.
→ Index these smaller fragments.
→ At retrieval, still return the larger chunk for context.
This approach is useful if the document contains several different concepts in one section — increasing the chance of an accurate match with the query.
3️⃣ Query Indexing
→ Instead of indexing raw text, hypothetical questions are generated that the LLM thinks the chunk can answer.
→ These questions are embedded and stored.
→ At real user query time, search is performed over these "synthetic" questions.
→ A similar idea is used in HyDE, but there the hypothetical answer is matched with real chunks.
Great option for question–answer (QA) systems, as it reduces the semantic gap between the user query and the indexed data.
4️⃣ Summary Indexing
→ An LLM is used to generate a brief semantic representation (summary) for each chunk.
→ The index contains the summary, not the original text.
→ At retrieval, the original chunk is returned for context.
Especially effective for dense or structured data (e.g., CSV or tables), where raw text embeddings do not yield meaningful results.
🤖 Data Science, ML & Big Data with @DataXplore