Vector Database, LSH, Similarity Vectors, RAG, Prompt caching

I know there’s a lot to catch up on, but trust me, in this journey I’ll try my best to explain everything in the easiest possible way the engineer’s way. So let’s get started. For all of you, this should be fairly easy to understand, so sit back, grab a coffee (or tea), and indulge in this reading.

Let’s start with the first reality check: from text to vectors.

As we all know, in LLMs, raw text is not something the model understands directly. So what do we do? We convert text into embeddings, more precisely, vector embeddings.

These embeddings are usually n × 1 dimensional vectors think 384, 768, or 1024 dimensions. This is special data and not something you would casually store in a traditional relational database and expect things to work efficiently.

That’s exactly why vector databases exist.

Databases like Pinecone, FAISS, Milvus, and Weaviate are built specifically for this purpose. They don’t just store embeddings; they are optimized to operate on vectors. This means they can efficiently search for similar vectors, perform nearest neighbour searches, and rank results based on semantic closeness.

For example, you can ask how similar “King” and “Man” are, or how unrelated “Insurance” and “Winter Boots” might be.

Now the obvious question is: how do we actually decide whether two vectors are similar or not?

The answer is simple we use similarity metrics.

The most common ones you’ll encounter are Euclidean distance, cosine similarity, and dot product similarity.

Euclidean distance is the most intuitive. You can think of it as the straight-line distance between two points in space. If the distance is small, the vectors are similar, if it’s large, they are not. This method considers both the direction of the vectors and their magnitude, meaning where the vector points and how long it is both matter.

Cosine similarity works a bit differently. It focuses on the angle between two vectors rather than the distance between them. If two vectors point in the same direction, their cosine similarity is close to one. If they are perpendicular, the similarity is zero. If they point in opposite directions, it approaches negative one. The key idea here is that magnitude does not matter only direction does. This is particularly useful in NLP because sentence length and word frequency can vary, while semantic meaning often aligns directionally. That’s why cosine similarity is so widely used for text embeddings.

Dot product similarity sits somewhere in between. It considers both direction and magnitude. If two vectors are aligned and also have large magnitudes, the dot product becomes large. This can be useful in cases where the magnitude of a vector itself carries meaning, such as importance, confidence, or frequency.

At this point, you might wonder which similarity metric you should use.

There is no universally right or wrong choice. However, one rule is important: you should use the same similarity metric that your embedding model was trained with. If your model was trained using cosine similarity, then you should also use cosine similarity during inference. Mixing metrics without understanding the implications can lead to unexpected behavior. Consistency matters more than clever optimizations here.

Now, coming to Euclidean distance, I personally find it quite interesting.

The reason is simple: many large-scale systems rely on it. Companies like Google, Netflix, Amazon, Spotify, and Uber use similarity search as a core part of their systems. However, they don’t use raw Euclidean distance directly. Instead, they use optimized approaches built on top of it.

One such approach is Locality Sensitive Hashing, or LSH.

LSH is essentially an engineering optimization. Instead of comparing every vector with every other vector, which is computationally expensive, vectors are hashed into buckets in such a way that similar vectors are more likely to end up in the same bucket. Comparisons are then limited to vectors within those buckets.

Interestingly, while hashing is usually meant to reduce collisions, in this case we intentionally increase collisions for similar items. This makes similarity search much faster at scale.

Conceptually, text is broken into shingles, often n-grams. These shingles are hashed, and techniques like MinHash are used to generate compact signatures. Similar texts produce similar signatures, which land in the same buckets. This enables faster similarity search, scalable nearest neighbour lookups, and significant performance improvements. These ideas are also very relevant when building RAG systems.

Now let’s talk about a practical optimization in RAG systems.

Imagine uploading the same PDF and asking multiple questions against it. Each time, the system re-chunks the document, re-embeds it, and processes everything again. This is unnecessary work and leads to higher latency and cost.

A better approach is to cache the document embeddings. Instead of reprocessing the PDF every time, you store its embeddings and reuse them. The model then only needs to process the user query. This reduces computation, improves response time, and lowers overall cost.

You might ask why not cache the queries instead.

The reason is simple: queries change frequently, while documents like PDFs usually remain the same or change far less often. That’s why caching document embeddings makes much more sense.

That’s it for this blog. Stay tuned for the next blog on steering LLMs. That’s where things get more interesting. Until then, enjoy the journey and keep approaching problems with an engineering mindset.