Indexer

Indexer

The indexer is a Python script (indexer.py) that runs as a Docker container on cyrion. It walks the Obsidian vault, chunks markdown files by heading, generates embeddings via Ollama, populates tsvector for full-text search, extracts wikilinks, and upserts everything into pgvector.

Source: /home/nox/docker/obsidian-rag/indexer.py (~280 lines) Image: Built locally from Dockerfile (Python 3.12-slim) Dependencies: psycopg[binary], httpx, semantic-text-splitter

Pipeline Steps

1. Walk the Vault

SKIP_DIRS = {".obsidian", ".git", ".trash", "Attachments", ".quartz-cache"}

Walks recursively, only processes .md files.

2. Change Detection (SHA256)

Computes SHA256 of each file and compares against stored file_hash. Unchanged files are skipped entirely — no parsing, no embedding, no DB writes.

3. Frontmatter Extraction

YAML frontmatter is parsed and stored as JSONB metadata alongside each chunk.

4. Semantic Chunking

Uses semantic-text-splitter (MarkdownSplitter) to split at heading/structure boundaries while respecting a 6,000-character max. Each chunk inherits the most recent heading as its label. Multi-chunk files get a (part N) suffix.

Why 6,000 characters? nomic-embed-text has an 8,192-token context window. At ~4 chars/token, 6,000 chars is ~1,500 tokens — well within limits.

5. Wikilink Extraction

Extracts [[wikilinks]] from the full file text:

WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")

Filtering:

• Code blocks are stripped first (prevents false positives from code examples)
• Targets with characters ${}=*() are skipped (code artifacts)
• Known non-note extensions filtered (.png, .jpg, .pdf, etc.)
• Heading links ([[Note#Section]]) are resolved to the base note name

Links are stored in the note_links table — one row per unique source→target pair per file. Old links for a file are deleted before inserting new ones.

6. Embedding via Ollama

Each chunk is embedded as "{heading}\n\n{content}". Requests are individual per chunk with retry logic (3 attempts, exponential backoff). On HTTP 400 (token limit exceeded), chunks are progressively truncated at 70% until they fit.

7. Upsert into pgvector

Delete-then-insert per file, wrapped in a transaction:

cur.execute("DELETE FROM documents WHERE file_path = %s", (rel_path,))
cur.execute(
    """INSERT INTO documents
       (file_path, file_hash, chunk_index, heading, content, embedding,
        content_tsv, metadata, updated_at)
       VALUES (%s, %s, %s, %s, %s, %s::vector,
               to_tsvector('english', coalesce(%s, '') || ' ' || %s),
               %s::jsonb, NOW())""",
    (rel_path, fhash, idx, heading, content, str(embedding),
     heading, content, Json(metadata))
)

The content_tsv tsvector is computed server-side during INSERT, keeping it in sync with heading + content for full-text search.

8. Stale Cleanup

After processing all files:

• Deletes documents rows for files no longer in the vault
• Deletes note_links rows for removed files

9. Wikilink Resolution

After all files are indexed, resolves target_name → target_path by matching against indexed file stems (case-insensitive). This populates the target_path column in note_links for graph traversal.

Scheduling

Runs every 30 minutes via a systemd timer on cyrion.

Timer (/etc/systemd/system/obsidian-rag-indexer.timer):

[Unit]
Description=Run Obsidian RAG Indexer every 30 minutes

[Timer]
OnCalendar=*:00/30
Persistent=true

[Install]
WantedBy=timers.target

Service (/etc/systemd/system/obsidian-rag-indexer.service):

[Unit]
Description=Obsidian RAG Indexer
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
User=nox
WorkingDirectory=/home/nox/docker
ExecStart=/usr/bin/docker compose --profile indexer run --rm obsidian-rag-indexer

Manual Trigger

ssh nox@cyrion.c0smere.net "cd /home/nox/docker && docker compose --profile indexer run --rm obsidian-rag-indexer"

Force Full Re-index

# Invalidate all hashes to force re-processing every file
ssh root@talos.c0smere.net "pct exec 302 -- sudo -u postgres psql -d obsidian_rag -c \"UPDATE documents SET file_hash = 'force-reindex';\""
# Then run the indexer

Docker Configuration

docker-compose.yml (in master compose on cyrion):

obsidian-rag-indexer:
  build: ./obsidian-rag
  container_name: obsidian-rag-indexer
  restart: "no"
  volumes:
    - /home/nox/docker/obsidian/vaults/weeslahw_coppermind:/vault:ro
  environment:
    - DATABASE_URL=postgresql://rag_indexer:${RAG_DB_PASSWORD}@phlegethon.c0smere.net:5432/obsidian_rag
    - OLLAMA_URL=http://ollama:11434
  depends_on:
    - ollama
  profiles:
    - indexer

Environment Variables

Known Issues

Quote Files Fail Embedding

A few files in Digital_Garden/Quotes/ return HTTP 400 from Ollama due to encoding issues in Kobo e-reader highlight content. The indexer handles these via progressive truncation (70% of original size per retry, up to 3 attempts). Most now succeed after truncation; any that still fail are skipped without crashing the run.

Typical Run Output

Starting Obsidian RAG indexer (Semantic + Hybrid Mode)
Vault: /vault
Found 238 markdown files
...
Resolved 46 / 115 wikilink targets to file paths
Done! Files: 0 skipped, 0 added, 236 updated, 0 deleted, 0 errors. Total chunks: 742, Total links: 136

A clean incremental run with no changes completes in seconds. A full re-index takes ~30 minutes (embedding 742 chunks sequentially via Ollama on cyrion’s CPU).