Why a regular database can't just do this
A normal database index (a B-tree, for instance) is built around ordering values you can sort — great for "find all rows where price < 100," useless for "find the 10,000-dimension vector closest to this other 10,000-dimension vector," which isn't a sorting problem at all.
Vector databases use specialized index structures (HNSW is the most common one in production) built specifically for approximate nearest-neighbor search — trading a small amount of accuracy for search speeds that stay fast even as the collection grows into the millions of vectors.
How it fits into a RAG pipeline
When a user asks a question, the system converts that question into a vector using the same embedding model used to index the content, sends that vector to the vector database, and gets back the stored passages whose vectors are closest — the candidates most likely to actually be relevant.
Those retrieved passages get handed to the LLM as context alongside the original question, so the model answers using real, current, specific text instead of whatever it happened to memorize during training — this is the retrieval half of retrieval-augmented generation.
The practical options, from simplest to most scalable
For a small, prototype-scale project, a lightweight library-based option (like Chroma or an in-memory FAISS index) is often enough and requires no separate service to run.
For production scale, a dedicated managed or self-hosted vector database (Pinecone, Weaviate, Qdrant, or Postgres with the pgvector extension if you'd rather not add a new database engine at all) handles millions of vectors, metadata filtering, and horizontal scaling that a simple library wasn't built for.