AI Assistant

markdown# Ajolla AI Assistant

An AI chat assistant inside your app. Admins configure one or more assistants (model, system prompt, tools, starter prompts, audience, limits) and give each a small knowledge base; users chat at `/assistant` with streaming replies and saved conversations. Every model call goes through the **Vercel AI Gateway** on **your** credentials, and hard limits bound what it can cost.

## What was installed

**Routes**
- `/assistant` — the chat. `?a=<slug|id>` picks an assistant, `?c=<conversationId>` opens a saved conversation. Sidebar: assistants, recent conversations (rename / delete). Chat: streaming replies, Stop, Retry, starter prompts, Enter to send / Shift+Enter for a new line, error and limit messages. Signed-out visitors see only *public* assistants (nothing is stored for them); with none, they're sent to sign-in.
- `POST /api/ajolla/ai-assistant/chat` — the streaming endpoint the chat calls. Body `{ assistantId, conversationId?, message?, regenerate?, history? }`; response is NDJSON (`start`, `delta`…, `done` | `error`). Errors before streaming are `{ error: { code, message } }` with 401/403/404/422/429/503.
- `/admin/assistant` — assistants with this month's token use against budget (permission `assistant.manage`).
- `/admin/assistant/new`, `/admin/assistant/[assistantId]` — settings, knowledge base (paste text or upload `.md`/`.txt`), this month's usage, delete.
- `/admin/assistant/usage` — monthly totals (6 months), heaviest users this month, the last 50 requests. Admins see token counts, never what users wrote.

**Code**
- `lib/ajolla/ai-assistant/core.ts` — all business logic, framework-free (assistants, documents, search, conversations, `prepareChat` / `finishChat`, tools, usage reports).
- `chunking.ts` (paragraph/heading-aware chunks with overlap), `vectors.ts` (dimension fitting, int8 quantisation, cosine ranking), `limits.ts` (limit resolution, UTC periods, history truncation), `markdown.ts` (safe markdown subset → data), `constants.ts` — all pure and unit-tested.
- `ai.ts` — AI SDK glue (`ai` v7): gateway provider, embedder, the built-in tools, `streamReply` / `generateReply`. Takes keys as arguments (no `process.env`).
- `server.ts` — env configuration, the secret-key usage scope, IP hashing.
- `components/ajolla/ai-assistant/*` — chat panel, markdown renderer, usage meter, CSS module. `lib/ajolla/admin/ai-assistant.ts` registers the admin page.

**Tables** (`ajolla_ai_assistant_`)
- `assistants` (unique `slug`), `documents`, `chunks` (`embedding extensions.vector(512)` + compact `embedding_q`), `conversations`, `messages`, `usage` (one row per model call: kind, model, input/output tokens, status), `monthly_usage` (per assistant per UTC month, maintained by a trigger on `usage`).
- Migration enables `create extension if not exists vector with schema extensions`.

**RLS**
- Assistants: enabled ones are readable by their audience (`public` → anonymous too), everything by `assistant.manage`; only admins write. The system prompt is readable by that audience — **never put secrets in it** (prompt extraction makes it public anyway).
- Documents: admins only. Chunks: readable by whoever may use the assistant (the search tool runs as them) — **anything in a public assistant's knowledge base is public**.
- Conversations and messages: owner only (admins can't read them).
- Usage: users read their own rows; admins read all. Chat usage is written by the server with the secret key (`SUPABASE_SECRET_KEY`), so users can't reset their own caps.

**Events** — `ai_assistant.conversation_started` and `ai_assistant.message_sent`, payload `{ assistantId, conversationId, userId, messageId? }`. Ids only, never message content.

## Cost controls

| Control | Default | Where |
|---|---|---|
| Messages per user per assistant per UTC day | 50 | `AI_ASSISTANT_DAILY_MESSAGES`, or per assistant (0 pauses chat) |
| Messages per signed-out visitor (IP) per public assistant per day | 10 | `AI_ASSISTANT_ANON_DAILY_MESSAGES` |
| Tokens per assistant per UTC month (chat + embeddings) | 2,000,000 | `AI_ASSISTANT_MONTHLY_TOKENS`, or per assistant |
| Characters per user message | 4,000 (max 32,000) | `AI_ASSISTANT_MAX_INPUT_CHARS` |
| Reply length | 1,024 tokens (64–4,096) | per assistant |
| History sent per request | last 10 turns (1–50), ≤ 48,000 chars | per assistant |
| Model ↔ tool round trips per reply | 3 | `HARD_LIMITS.maxSteps` in `limits.ts` |
| Knowledge base | 100,000 chars per document, 200 KB uploads, 500 passages per assistant | `core.ts` constants |

How they're enforced:
- The **daily cap** reserves a slot before calling the model (insert a pending usage row, then count), so concurrent requests can't both slip under it. A failed call that spent no tokens gives the slot back.
- The **monthly budget** is checked before every chat and every embedding run against `monthly_usage`. It's a soft stop: requests already in flight finish, so the overshoot is bounded by (concurrent requests × max tokens per request).
- **Usage** is recorded per message from the provider's numbers. If the user presses Stop (no usage reported), tokens are estimated (≈4 chars/token) so budgets still move.

## Knowledge-base search (trade-off)

Module code doesn't call database functions (they can't be made tenant-aware for Ajolla Hosted) and PostgREST can't order by vector distance without one. So `search_knowledge` fetches the assistant's compact int8 embeddings (≈700 bytes each, at most 500 → ≈350 KB) and ranks them in the app by cosine similarity, then loads the top 4 passages. That's fast and cheap at this size. The full-precision `embedding` column is pgvector, so if you outgrow 500 passages you can add an HNSW index and a `match_chunks` SQL function and call it from `searchKnowledge` (self-hosted), then raise `MAX_CHUNKS_PER_ASSISTANT`.

Embeddings are 512-dimensional (`openai/text-embedding-3-small` returns 512 natively when asked; other models are truncated + re-normalised or zero-padded — use a Matryoshka model such as OpenAI `text-embedding-3-*` if you change it). Each chunk records its embedding model; after changing `AI_ASSISTANT_EMBEDDING_MODEL`, use **Re-index** on each document.

## Env

- `AI_GATEWAY_API_KEY` — your Vercel AI Gateway key. On Vercel you can skip it: the deployment's OIDC token is used. Without either, chat returns 503 and the admin shows a warning.
- `AI_ASSISTANT_DEFAULT_MODEL` (default `anthropic/claude-haiku-5.5`), `AI_ASSISTANT_EMBEDDING_MODEL` (default `openai/text-embedding-3-small`).
- `AI_ASSISTANT_DAILY_MESSAGES`, `AI_ASSISTANT_ANON_DAILY_MESSAGES`, `AI_ASSISTANT_MONTHLY_TOKENS`, `AI_ASSISTANT_MAX_INPUT_CHARS` — see the table.
- `AI_ASSISTANT_IP_SALT` — secret mixed into IP hashes for signed-out caps (falls back to one derived from `SUPABASE_SECRET_KEY`). Raw IPs are never stored.
- Uses the kernel's `SUPABASE_SECRET_KEY` for usage accounting.

## Setup
1. Apply migrations: `supabase db push`.
2. Set `AI_GATEWAY_API_KEY` (or deploy on Vercel with AI Gateway enabled). Consider a spend limit on the Gateway itself as a final backstop.
3. As an admin, open `/admin/assistant` → **New assistant**. Pick a model, write a system prompt, choose tools and starter prompts.
4. Add knowledge-base documents on the assistant's page (paste or upload `.md`/`.txt`).
5. Link users to `/assistant` (or `/assistant?a=<slug>`).

## Customising
- Behaviour common to every assistant (date, markdown, "search before answering"): `buildSystemPrompt()` in `core.ts`.
- Add a tool: add a handler in `createToolHandlers()` (`core.ts`), wrap it in `buildTools()` (`ai.ts`), add its id to `TOOL_IDS` (`constants.ts`) and to the `tools` check in a new migration.
- Chunk size/overlap: `DEFAULT_CHUNK_OPTIONS` in `chunking.ts`; results per search: `SEARCH_TOP_K`, `SEARCH_MIN_SCORE`.
- Styling: `components/ajolla/ai-assistant/ai-assistant.module.css` and the kit's `--aj-*` variables. Markdown rendering: `markdown.ts` + `markdown.tsx` (no HTML is ever injected).
- Signed-out chat is capped per IP but still costs you tokens — leave assistants on "Signed-in users" unless you need it, and consider a WAF rate limit on `POST /api/ajolla/ai-assistant/chat`.

## Ajolla Hosted

The hosted API (`/api/m/ai-assistant/v1`, secret keys only) manages assistants and documents, lists conversations and messages, and sends a message (whole JSON reply, no streaming). It is **bring-your-own-key**: model calls use the app's dashboard secret `aiGatewayApiKey` and fail with **402 `byok_required`** without it. Limits come from app settings `aiAssistantDailyMessages`, `aiAssistantMonthlyTokens`, `aiAssistantMaxInputChars`, `aiAssistantEmbeddingModel` (or per assistant). There's no anonymous chat in hosted mode.