How to build a local semantic search engine for markdown files using Ollama and ChromaDB

Beyond Grep: Why Keyword Search Is Failing Your Notes

We have all been there. You are digging through your personal knowledge base, a massive directory of markdown files, trying to find that one specific thought you had six months ago about "database scaling solutions". You run a standard keyword search or grep command, but it returns nothing because, back then, you happened to write about "optimising backend storage clusters".

Keyword search is rigid. It requires you to remember the exact vocabulary your past self used. Semantic search, however, understands conceptual intent. It knows that "database scaling" and "backend storage cluster" are siblings in the same conceptual family tree.

Building a semantic search tool used to mean renting expensive vector databases and shipping your sensitive private notes off to third-party APIs. Not anymore. Today, you can build a lightning-fast, highly accurate, and completely offline semantic search engine on your own machine. We will use Ollama for generating vector embeddings locally, ChromaDB as our ultra-lightweight local vector database, and a few lines of clean Python to tie it all together.

The Stack under the Hood

Before we write any code, let us clarify how these pieces interact. If you want a quick primer on how vector databases actually work, check out our AI glossary.

Here is how our local pipeline works: 1. Document Loading: We read your local markdown files and split them into manageable semantic chunks. 2. Vector Embedding: We pass these chunks to Ollama using an open-weight embedding model (like nomic-embed-text) which converts text into numerical coordinates (vectors) representing their meaning. 3. Vector Database: We store these coordinates in ChromaDB, an open-source, serverless vector database that runs entirely in-memory or in a local directory. 4. Querying: When you search, we embed your search query and tell ChromaDB to find the nearest vectors mathematically.

Let us get your environment ready.

Step 1: Install Ollama and Pull the Embedding Model

First, make sure you have Ollama installed and running. Head to the official Ollama website, download the client for your operating system, and start the service.

Once Ollama is running in your background, open your terminal and run the following command to download the embeddings model:

`bash ollama pull nomic-embed-text `

We are using nomic-embed-text because it is exceptionally lightweight, fast on consumer hardware, and has an 8192-token context window, making it perfect for processing long paragraphs of text.

Step 2: Set Up Your Python Environment

Create a new directory for your project and set up a virtual environment. Install the necessary dependencies using pip:

`bash mkdir local_search && cd local_search python3 -m venv venv source venv/bin/activate pip install chromadb ollama `

(Note: If you run into issues installing ChromaDB on certain architectures, verify your Python build tools are up to date, or check [Google Gemini Support](https://googlegemini-support.com) if you decide to port this workflow to enterprise environments.)

Step 3: Write the Indexing Script

Now, we need a Python script that scans your markdown directory, reads the files, chunks them up so we do not hit token limits, embeds them via Ollama, and saves them to ChromaDB.

Create a file named index_notes.py and add the following code:

`python import os import chromadb import ollama

Configure paths NOTES_DIR = "./my_notes" # Change this to your markdown folder DB_PATH = "./chroma_db"

Initialize ChromaDB local client chroma_client = chromadb.PersistentClient(path=DB_PATH) collection = chroma_client.get_or_create_collection(name="local_notes")

def chunk_text(text, max_chars=1000): """Splits text into chunks, trying not to cut sentences in half.""" sentences = text.split('. ') chunks = [] current_chunk = "" for sentence in sentences: if len(current_chunk) + len(sentence) < max_chars: current_chunk += sentence + ". " else: chunks.append(current_chunk.strip()) current_chunk = sentence + ". " if current_chunk: chunks.append(current_chunk.strip()) return chunks

def index_markdown_files(): if not os.path.exists(NOTES_DIR): print(f"Directory '{NOTES_DIR}' not found. Please create it and add markdown files.") return

for root, _, files in os.walk(NOTES_DIR): for file in files: if file.endswith(".md"): file_path = os.path.join(root, file) with open(file_path, "r", encoding="utf-8") as f: content = f.read() chunks = chunk_text(content) for i, chunk in enumerate(chunks): # Generate embedding locally using Ollama response = ollama.embeddings( model="nomic-embed-text", prompt=chunk ) embedding = response["embedding"] # Store in ChromaDB doc_id = f"{file}_{i}" collection.upsert( ids=[doc_id], embeddings=[embedding], documents=[chunk], metadatas=[{"source": file_path, "chunk_index": i}] ) print(f"Indexed: {file} ({len(chunks)} chunks)")

if __name__ == "__main__": index_markdown_files() `

Create a dummy folder called my_notes and drop a few markdown files into it. Then run the indexing script:

`bash python index_notes.py `

Step 4: Write the Search Interface

With your files successfully indexed and converted into high-dimensional vector representations, we can now write a search script that lets you query your local collection conceptually.

Create a file named search_notes.py:

`python import chromadb import ollama import sys

DB_PATH = "./chroma_db" chroma_client = chromadb.PersistentClient(path=DB_PATH) collection = chroma_client.get_collection(name="local_notes")

def search(query_text, num_results=3): # 1. Embed the search query using the exact same model query_response = ollama.embeddings( model="nomic-embed-text", prompt=query_text ) query_embedding = query_response["embedding"] # 2. Query ChromaDB for the closest vector matches results = collection.query( query_embeddings=[query_embedding], n_results=num_results ) # 3. Print the matches nicely print(f"\nSearch Results for: '{query_text}'\n" + "="*40) for i in range(len(results["ids"][0])): doc = results["documents"][0][i] metadata = results["metadatas"][0][i] distance = results["distances"][0][i] print(f"\n[Result #{i+1}] (Match Confidence Score: {round(1 - distance, 4)})") print(f"Source File: {metadata['source']}") print(f"Snippet:\n{doc}\n") print("-"*40)

if __name__ == "__main__": if len(sys.argv) < 2: print("Usage: python search_notes.py 'your search query here'") else: query = " ".join(sys.argv[1:]) search(query) `

Now test it. Run a search query using concepts rather than exact words:

`bash python search_notes.py "improving database load speeds" `

Even if none of your documents use the precise phrase "improving database load speeds", your semantic search tool will surface the notes discussing caching, index creation, or query optimisations.

Customising Your Engine

This setup runs completely offline, ensuring your proprietary research, diary entries, or company wikis never leave your local device.

If you find your computer struggling with local model compilation or if you want to scale this beyond individual project boundaries into a collaborative environment, you might consider migrating your ingestion pipeline to commercial APIs. You can read up on how to configure cloud-based embedding alternatives in our deep-dive comparison on /platforms/openai.

With under 100 lines of code, you have built a private, lightning-fast semantic search tool. No API keys, no monthly subscriptions, and zero data leakage. Now go drop your entire markdown folder in there and see what your notes have been trying to tell you.