Kobo Highlight Pipeline

End-to-end pipeline that extracts reading highlights from a Kobo e-reader, stores them in a database, exports them to Obsidian markdown, and publishes them on a statically generated website. Five systems connected by HTTP and filesystem mounts — zero manual intervention once the Kobo syncs.

Architecture Overview

┌──────────────────────┐
│   Kobo Clara 2E      │
│   (e-reader)         │
│                      │
│ KoboReader.sqlite    │
│   └─ Bookmark table  │
│                      │
│ kb_exfiltrator (Zig) │
│   - extracts text    │
│   - cleans artifacts │
│   - SHA256 hashes    │
│   - builds JSON      │
└──────────┬───────────┘
           │ HTTP POST (JSON array)
           │ http://192.168.88.69:1880/api/highlights
           ▼
┌──────────────────────────────────────────┐
│  Node-RED (cyrion:1880)                  │
│                                          │
│  ┌─ Kobo Highlights API ───────────────┐ │
│  │  POST /api/highlights               │ │
│  │  Validates → Upserts to MariaDB     │ │
│  │  (loop per highlight, dedup by hash)│ │
│  └─────────────────────────────────────┘ │
│                     │                    │
│              MariaDB (stygies)           │
│              kobo_highlights table       │
│                     │                    │
│  ┌─ Kobo Markdown Export ──────────────┐ │
│  │  Daily 3am (+ manual trigger)       │ │
│  │  SELECT all → group by book         │ │
│  │  Write .md files to /obsidian/Quotes│ │
│  │  Incremental (skips existing hashes)│ │
│  └─────────────────────────────────────┘ │
└──────────────────────┬───────────────────┘
                       │ filesystem (Docker volume mount)
                       ▼
┌──────────────────────────────────────────┐
│  Obsidian Vault                          │
│  Digital_Garden/Quotes/                  │
│    {book_hash}.md per book               │
│    Frontmatter + blockquotes + hashes    │
└──────────────────────┬───────────────────┘
                       │ rsync (change-detected)
                       ▼
┌──────────────────────────────────────────┐
│  Quartz Static Site Generator            │
│  update_quartz_docker.sh                 │
│    - detects vault changes               │
│    - rsyncs content into build context   │
│    - multi-stage Docker build            │
│    - Node.js builder → npx quartz build  │
│    - NGINX Alpine serves static HTML     │
└──────────────────────┬───────────────────┘
                       │ port 18000 → NPM on Lightsail
                       ▼
┌──────────────────────────────────────────┐
│  garden.c0smere.net                      │
│  Public website with quotes section      │
└──────────────────────────────────────────┘

Stage 1: Kobo — kb_exfiltrator

Source: /home/wes/projects/kb_exfiltrator Language: Zig (statically linked with musl libc + embedded SQLite) Binary: kb_exfiltrator-kobo-arm (1.27 MB, ARM EABI5)

What It Does

Reads the Kobo’s internal SQLite database (/mnt/onboard/.kobo/KoboReader.sqlite), extracts all highlights from the Bookmark table, and either prints JSON to stdout or POSTs it to a server endpoint.

Highlight Extraction

The SQL query pulls from the Bookmark table:

Field	Source
`text`	`Bookmark.Text` (the highlighted passage)
`date`	`Bookmark.DateCreated` (ISO timestamp)
`book`	Parsed from `Bookmark.ContentID` path (filename without extension)
`text_hash`	SHA256 of the cleaned highlight text
`book_hash`	SHA256 of the raw `ContentID` (stable book identifier across syncs)

Text Cleaning

The cleanHighlightText() function strips OCR/pagination artifacts — page reference numbers that Kobo sometimes prepends to highlights (e.g., "10After the battle" → "After the battle").

Deduplication

Two SHA256 hashes per highlight:

text_hash — identifies a unique quote. If the same passage is highlighted twice, it’s the same hash. Used as the primary key for upserts.
book_hash — identifies a unique book. Because ContentID is a full filesystem path that can vary between Kobo syncs, the hash provides a stable identifier. Used to group quotes into per-book markdown files.

Deployment on Kobo

The binary lives at /mnt/onboard/.adds/kb_exfiltrator/ alongside two shell scripts:

sync_highlights.sh — Basic wrapper:

kb_exfiltrator "$DB" "$ENDPOINT"
# DB = /mnt/onboard/.kobo/KoboReader.sqlite
# ENDPOINT = http://192.168.88.69:1880/api/highlights

sync_highlights_with_fbink.sh — Enhanced version with e-ink display feedback via FBInk:

Shows “Syncing highlights…” while running
On success: “Synced N highlights!”
On failure: “Sync failed! Check log”
Auto-clears after 2-3 seconds

NickelMenu integration — Adds a “Sync Highlights” button to the Kobo’s main menu and reader menu:

menu_item :main    :Sync Highlights :cmd_spawn :quiet :/mnt/onboard/.adds/kb_exfiltrator/sync_highlights.sh
menu_item :reader  :Sync Highlights :cmd_spawn :quiet :/mnt/onboard/.adds/kb_exfiltrator/sync_highlights.sh

Output Format

[
  {
    "text": "He is a slave, believing himself king.",
    "date": "2026-01-19T12:30:00Z",
    "book": "40K/Heresy/Betrayer (Dembski-Bowden, Aaron) (Z-Library).epub",
    "text_hash": "9f03b997794b00cdf9133e05db122aa5...",
    "book_hash": "f39d5574673a46556fdcccfc3fd2f943..."
  }
]

Stage 2: Node-RED — Ingest & Store

Container: node-red on cyrion (port 1880) Database: MariaDB on stygies (LXC 301, 192.168.88.7:3306)

Kobo Highlights API Flow

Receives the JSON array from kb_exfiltrator and upserts each highlight into the kobo_highlights table.

Endpoint: POST /api/highlights

Processing:

Validates the input is a non-empty array with required fields (text_hash, book_hash, text, book, date)
Loops through highlights one at a time
For each: executes an upsert query against MariaDB
Tracks insert count (new highlights) vs. update count (existing, refreshed)
Returns a summary response

Upsert query:

INSERT INTO kobo_highlights (text_hash, book_hash, text, book, date)
VALUES (?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE
  book = VALUES(book),
  date = VALUES(date),
  updated_at = CURRENT_TIMESTAMP

The ON DUPLICATE KEY clause means re-syncing the same highlights is safe — existing rows get their timestamp refreshed but content isn’t duplicated.

Response (201):

{
  "message": "Highlights processed successfully",
  "total": 209,
  "inserted": 3,
  "updated": 206
}

Database Schema

CREATE TABLE kobo_highlights (
  text_hash  VARCHAR(64) PRIMARY KEY,
  book_hash  VARCHAR(64) NOT NULL,
  text       LONGTEXT NOT NULL,
  book       VARCHAR(512) NOT NULL,
  date       DATETIME,
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);

The text_hash primary key means the same quote can never appear twice, regardless of how many times the Kobo syncs.

Stage 3: Node-RED — Markdown Export

Kobo Markdown Export Flow

Reads all highlights from MariaDB, groups them by book, and writes per-book markdown files to the Obsidian vault.

Triggers:

Automatic: daily at 3:00 AM (cron inject node)
Manual: trigger button in Node-RED UI

Process:

SELECT text_hash, book_hash, text, book, date FROM kobo_highlights ORDER BY book_hash, date
Groups results by book_hash
For each book:
- Filename: {book_hash}.md (SHA256 hash, not the book title — avoids special character issues)
- If new file: Creates it with YAML frontmatter and a heading
- If existing file: Reads it, extracts all  comments to find already-exported quotes
- Appends only new quotes (those whose text_hash isn’t already in the file)
- Writes the updated file

Output directory: /obsidian/Quotes/ inside the Node-RED container, which is a Docker bind mount to:

/home/nox/docker/obsidian/vaults/weeslahw_coppermind/Digital_Garden/Quotes/

Generated Markdown Format

---
title: "40K/Heresy/Betrayer (Dembski-Bowden, Aaron) (Z-Library).epub"
author: ""
book_hash: f39d5574673a46556fdcccfc3fd2f943...
isbn: ""
---

# 40K/Heresy/Betrayer (Dembski-Bowden, Aaron) (Z-Library).epub

> He is a slave, believing himself king.
> — *2026-01-19*
<!-- hash:9f03b997794b00cdf9133e05db122aa5... -->

> Gods lied, just like men.
> — *2026-02-03*
<!-- hash:86abbf778bdff0335578dee76eadec84... -->

Each quote is a blockquote with an italicized date attribution. The HTML comment with the text hash is invisible in rendered markdown but enables incremental export — the export function reads these hashes to skip already-written quotes.

Export Statistics

The flow returns a debug payload like:

{
  "message": "Export complete",
  "booksCreated": 1,
  "booksUpdated": 5,
  "booksSkipped": 20,
  "quotesAdded": 3,
  "totalBooks": 26
}

Stage 4: Quartz — Static Site Generation

Source: /home/nox/docker/digital_garden/ on cyrion Git repo: https://git.c0smere.net/wes/digital_garden Framework: Quartz v4 (TypeScript, Node.js) Public URL: garden.c0smere.net

How Quartz Turns Markdown into a Website

Quartz is a static site generator designed specifically for Obsidian vaults. It understands wiki-links ([[Page]]), frontmatter, callouts, and other Obsidian syntax. It generates a fully navigable HTML site with search, graph view, backlinks, and table of contents.

Build & Deploy Pipeline

The update_quartz_docker.sh script handles the entire build-deploy cycle:

1. Check for changes
   └─ Compare vault file timestamps against .last_build_content_timestamp
   └─ If no changes → exit early (no rebuild)

2. Prepare content
   └─ rsync -a --delete from Obsidian vault Digital_Garden/ into ./content/

3. Docker multi-stage build
   ┌─ Stage 1: node:lts-slim
   │  └─ Clone/update Quartz v4 source into ./quartz/
   │  └─ npm ci (install dependencies)
   │  └─ Copy quartz.config.ts, quartz.layout.ts, quartz_overrides/
   │  └─ Copy content/ (the rsynced vault content)
   │  └─ npx quartz build → generates static HTML in /public
   │
   └─ Stage 2: nginx:alpine
      └─ Copy /public from builder stage
      └─ Custom nginx config (try_files for clean URLs)
      └─ Serve on port 80

4. Docker Compose deploy
   └─ docker compose down --remove-orphans --volumes
   └─ docker compose up -d --build --remove-orphans
   └─ Container runs on port 18000

5. Update timestamp
   └─ Touch .last_build_content_timestamp

Quartz Configuration

quartz.config.ts highlights:

Base URL: garden.c0smere.net
Ignores: private/, templates/, .obsidian/
Plugins: frontmatter, syntax highlighting (GitHub themes), Obsidian + GFM markdown, LaTeX/KaTeX, RSS feed, sitemap, OG images
Analytics: Plausible

quartz.layout.ts:

Left sidebar: search, dark mode, reader mode, file explorer
Right sidebar: graph view, table of contents, backlinks
Footer: custom Cosmere quotes + links to Git, Blog, Resume, LinkedIn

Custom Components

A custom Footer.tsx in quartz_overrides/ adds two Cosmere quotes to the site footer:

“Failure is unique among sin, in that we can turn it into a virtue when it drives us to succeed.”
“The living diminish, but the machine endures…”

Docker Compose

services:
  quartz-site:
    build: .
    image: my-quartz-site-prod
    container_name: quartz-prod-container
    ports:
      - "18000:80"
    restart: unless-stopped

Standalone compose file — not part of the master compose on cyrion.

Stage 5: Public Access

The generated site at cyrion:18000 is served publicly via:

garden.c0smere.net → Lightsail VPS (NPM) → Twingate (LXC 101) → cyrion:18000

The quotes section at garden.c0smere.net/Quotes/ contains one page per book, each with all highlights from that book rendered as blockquotes.

End-to-End Timeline

Event	Latency	What Happens
User taps “Sync Highlights” on Kobo	Instant	kb_exfiltrator runs, extracts highlights, POSTs to Node-RED
Node-RED receives POST	~1-2 seconds	Upserts all highlights into MariaDB
Daily 3am export	Scheduled	Node-RED reads MariaDB, writes new quotes to vault markdown files
Quartz rebuild	On next script run	Detects changed files, rebuilds static site, deploys new container
Public website updated	After rebuild	New quotes visible at garden.c0smere.net

Typical sync-to-published time: Same day (3am export + next Quartz build cycle). Can be immediate if both the export and build are triggered manually.

Key Design Decisions

Why SHA256 hashes everywhere? Book titles from Kobo ContentIDs are full filesystem paths that can change between syncs. Hashing gives stable identifiers. Text hashes prevent duplicate quotes even if the same passage is highlighted multiple times.

Why MariaDB as an intermediate store? Could have gone straight from kb_exfiltrator to markdown files. The database layer adds:

Durability (quotes survive if markdown files are accidentally deleted)
Query flexibility (can build other views/exports later)
Clean separation between ingestion and export
Upsert semantics for safe re-syncing

Why Zig for the Kobo binary? Kobo e-readers run ARM Linux with a minimal userspace — no Python, no Node, no package manager. Zig cross-compiles to a single statically-linked ARM binary with embedded SQLite. Zero runtime dependencies. The binary is 1.27 MB.

Why hash-named markdown files? Book titles from Kobo paths contain slashes, parentheses, special characters, and sometimes Unicode. Using the book_hash as the filename avoids all escaping issues. The actual title goes in the YAML frontmatter where it’s safely quoted.

Monitoring & Troubleshooting

Check if highlights synced to DB:

ssh nox@cyrion.c0smere.net "docker exec node-red curl -s http://localhost:1880/api/highlights-count"

Or query MariaDB directly on stygies.

Check markdown export:

ls -lt /home/nox/docker/obsidian/vaults/weeslahw_coppermind/Digital_Garden/Quotes/ | head

Trigger manual export: Click the inject node for “Kobo Markdown Export” in the Node-RED UI at http://cyrion.c0smere.net:1880.

Rebuild Quartz manually:

ssh nox@cyrion.c0smere.net "cd /home/nox/docker/digital_garden && ./update_quartz_docker.sh force"

Known issue: 3 quote files consistently fail embedding in the Obsidian RAG indexer (HTTP 400 from Ollama). Likely encoding issues in highlight content — non-critical, the quotes still appear on the website.

Node-RED — API flow details
Node-RED — Export flow details
Overview — Quartz Docker build walkthrough
Obsidian RAG Pipeline — How quote files get indexed for semantic search

Kobo Highlight Pipeline

Kobo Highlight Pipeline

Architecture Overview

Stage 1: Kobo — kb_exfiltrator

What It Does

Highlight Extraction

Text Cleaning

Deduplication

Deployment on Kobo

Output Format

Stage 2: Node-RED — Ingest & Store

Kobo Highlights API Flow

Database Schema

Stage 3: Node-RED — Markdown Export

Kobo Markdown Export Flow

Generated Markdown Format

Export Statistics

Stage 4: Quartz — Static Site Generation

How Quartz Turns Markdown into a Website

Build & Deploy Pipeline

Quartz Configuration

Custom Components

Docker Compose

Stage 5: Public Access

End-to-End Timeline

Key Design Decisions

Monitoring & Troubleshooting

Related Documentation