Search

markdown# Ajolla Search

Postgres full-text search over your app's content: a `/search` results page, a ⌘K command palette, an indexer that follows other modules' events, and an admin screen for synonyms, pinned results and query analytics. Admins are users with the `search.manage` permission.

## What was installed

**Routes**
- `/search`: results with highlighted snippets, source and tag filters, pagination, spelling corrections ("Showing results for …") and tips when nothing matches. Works without JavaScript. Signed-out visitors see public documents only.
- `/api/ajolla/search`: JSON search (`q`, `source`, `tag`, `page`, `limit` ≤ 20, `prefix=1`, `log=0`). Runs as the visitor (RLS applies), 60 requests per minute per IP per server instance.
- `/admin/search`: document counts per source, **Reindex** / **Clear** per source, top and zero-result queries (7/30/90 days), synonyms, and every pin/boost/hide rule.
- `/admin/search/tune?q=…`: try a query as an admin (all documents, not logged) and pin, boost or hide individual results for it.
- `/admin/search/documents`: browse and remove indexed documents.

**Components** (`components/ajolla/search/`)
- `<SearchCommandPalette />` (`command-palette.tsx`): a "Search… ⌘K" button plus a modal combobox. ⌘K / Ctrl+K toggles it; ↑ ↓ Home End move; Enter opens; Esc closes. Debounced (160 ms), cancels stale requests, remembers recent searches in `localStorage`. Props: `placeholder`, `searchPage`, `endpoint`.
- `<Highlighted segments={…} />`: renders the core's highlight segments as `<mark>` (text only, never HTML).

**Code** (`lib/ajolla/search/`)
- `shared.ts`: pure helpers (query parsing, tsquery building, ranking, highlighting, snippets, spelling correction, rate limiter). No imports; safe in client components.
- `core.ts`: indexing, search, synonyms, promotions, analytics and embedding storage. Framework-free (also runs on Ajolla Hosted).
- `sources.ts`: adapters that turn Feedback posts and CMS entries into documents, `reindexSource()`, and `SOURCE_ADAPTERS` (add yours to get a Reindex button).
- `server.ts`: `indexDocument()`, `indexDocuments()`, `removeDocument()`, `removeDocuments()` and `searchAsVisitor()`.
- `embeddings.ts`: optional AI Gateway embeddings.
- `lib/ajolla/subscribers/search.ts`: the event indexer. `lib/ajolla/admin/search.ts`: admin page registration.

**Tables** (`ajolla_search_`)
- `documents`: unique `(source, external_id)`; `title`, `body` (first 32,000 characters), `summary`, `keywords` (tags + your keywords), `url`, `tags`, `visibility` (`public` / `signed_in` / `owner` + `user_id`), `boost` (−10…10), `metadata` (≤ 4 KB JSON, returned with results), `updated_at`, `indexed_at`. `fts` is a generated tsvector (title and keywords weight A, body B) with a GIN index; titles have a `pg_trgm` GIN index.
- `sources`: per-source indexing timestamps.
- `terms`: vocabulary from **public** titles and tags, for "did you mean".
- `synonyms`: groups of equivalent words or phrases.
- `promotions`: per-query `pin` (with position), `boost` (×2) or `hide`.
- `query_stats`: searches and zero-result counts per normalized query per day. No user ids; queries that look like emails, phone/card numbers or tokens are never logged.
- `embeddings` (migration 0002, needs `pgvector`): one `vector(256)` per document, used only when embeddings are on.

**RLS**
- Documents: anyone reads `public`; signed-in, non-anonymous users read `signed_in`; only the owner reads `owner`; `search.manage` reads and writes everything.
- Synonyms, promotions and vocabulary: readable by everyone (needed at query time), written by admins.
- Sources and query stats: admins only. Your server records analytics with the secret key.

**Events**
- Consumes `feedback.post_created`, `feedback.post_status_changed` (public, non-merged posts), `cms.entry_published`, `cms.entry_unpublished`, `cms.entry_deleted` (live published entries). Each handler re-reads the record, so redelivery is harmless. Feedback has no delete event: deleted posts drop out on the next **Reindex**.
- Emits `search.source_reindexed` and `search.source_cleared`. Individual searches are **not** events (too chatty) — they're counted in `query_stats`.

**Env** (all optional)
- `SEARCH_EMBEDDINGS=true` + `AI_GATEWAY_API_KEY`: semantic re-ranking (see below). `SEARCH_EMBEDDING_MODEL` overrides `openai/text-embedding-3-small`.
- `SEARCH_QUERY_LOG_SAMPLE_RATE` (0–1, default 1): record only a fraction of searches (weighted, so totals stay comparable).
- `SEARCH_QUERY_RETENTION_DAYS` (default 90).

## Setup
1. Apply the migrations: `supabase db push`. (They enable `pg_trgm`, and `vector` for the optional embeddings table.)
2. Put `<SearchCommandPalette />` in your header or root layout, and link to `/search`.
3. Index your own content from server code:
   ```ts
   import { indexDocument, removeDocument } from "@/lib/ajolla/search/server";
   await indexDocument({ source: "articles", externalId: a.id, title: a.title, body: a.body, url: `/articles/${a.slug}`, tags: ["guides"] });
   await removeDocument({ source: "articles", externalId: a.id });
   ```
   Use `visibility: "signed_in"` for members-only pages, or `visibility: "owner", userId` for one user's private records. These helpers use the secret key, so call them only from trusted server code.
4. Existing Feedback/CMS content: press **Reindex** on `/admin/search` once. New content is indexed from events.

## How search works
1. The query is parsed (`"exact phrase"`, `-exclude`, `a or b`), stopwords are dropped and synonyms expanded into a `to_tsquery` expression; the palette also matches the last word as a prefix.
2. Up to 200 matching documents are fetched through PostgREST (`.textSearch("fts", …, { config: "english" })`, newest first) and ranked in `scoreDocument()`: title matches weigh most, then tags/keywords, then the summary; exact/phrase title matches, coverage of every term, recency and the document's `boost` adjust the score. Promotions apply on top.
3. If nothing matches, titles are searched by substring (served by the trigram index), then misspelled words are corrected against the vocabulary (edit distance 1–2) and the corrected query is shown.
4. Bodies are fetched only for the current page, to build snippets around the densest cluster of matches.

Ranking happens on at most 200 candidates per query, which keeps every query PostgREST-only (no database functions). On very large indexes, very broad queries rank the 200 most recently updated matches; narrow the query or filter by source to reach older ones.

## Semantic re-ranking (optional, costs tokens)
With `SEARCH_EMBEDDINGS=true` and `AI_GATEWAY_API_KEY`, every indexed document gets a 256-dimension embedding (recomputed only when its content hash changes), and each search embeds the query (cached per server instance) and blends cosine similarity into the top 50 full-text results (`blendScores`, weight 0.4). It re-ranks; it doesn't find documents full-text search missed. Pinned results never move. If the gateway fails, search falls back to full-text ranking.

## Customising
- **Ranking:** `scoreDocument()` and `rankResults()` in `shared.ts`; `CANDIDATE_LIMIT`, `SNIPPET_LENGTH`.
- **Language:** the `fts` column and `.textSearch` both use the `english` configuration; change both (new migration) for another language.
- **CMS URLs:** `cmsEntryUrl()` in `sources.ts` (default `/<type>/<slug>`, with a locale prefix for localized entries).
- **New sources:** write a `SourceAdapter` in `sources.ts`, add it to `SOURCE_ADAPTERS`, and subscribe to its events in `subscribers/search.ts`.
- **Rate limit:** the limiter in `app/api/ajolla/search/route.ts`. It's per instance — add a Vercel Firewall rate-limit rule for a global limit.
- **Styles:** `components/ajolla/search/search.module.css`, plus the kit's `--aj-*` variables.

## Ajolla Hosted
The same core is served at `/api/m/search/v1`: secret-key endpoints to upsert, batch (≤ 100 documents, each body ≤ 32 KB), delete and list documents, manage synonyms and promotions and read analytics; a publishable-key `GET /search` over public documents (≤ 20 results, rate limited per IP). Hosted search is full-text only — no embeddings.