Breeze

Breeze has three layers: a Next.js frontend, a FastAPI backend, and a local model served by Ollama. Data flows one way.

Browser -> Next.js API routes -> FastAPI backend -> Ollama
                |
                v
           MongoDB (Mongoose)

Next.js never talks to Ollama

Every LLM call is owned by FastAPI. The Next.js routes are an authenticated passthrough -- they add X-API-Key and forward the tracing headers, nothing more.

The split

Layer	Owns
Next.js	Auth, chat UI, CRUD for conversations and messages, passthrough proxy.
FastAPI	All LLM calls: streaming, summarisation, generative UI routing, web search.
Ollama	Running the models.
MongoDB	Users, conversations and messages, via Mongoose.

Authentication is NextAuth with Credentials and JWT sessions. API routes add the X-API-Key header and forward the Langfuse tracing params (X-User-Id, X-Session-Id). CRUD for conversations and messages runs directly against MongoDB on the Next.js side.

Message state

Messages live exclusively in the TanStack Query cache, under the key ['conversations', id, 'messages']. Zustand holds only isAuthenticated.

Do not move messages into Zustand

Two stores for one list means two sources of truth during a stream. The cache is already the transport: useChatStream writes chunks into it with setQueryData as they arrive.

Prop

Type

The order of a turn

Create the conversation

Only on a first message. POST /api/conversations returns an id, the cache is pre-populated, and the router navigates to /chat/[id].

Write optimistically, save the user message

The user message goes into the cache immediately, and to the database as a fire-and-forget POST /api/conversations/:id/messages.

Stream the reply

/api/chat proxies FastAPI /completion. Each NDJSON line is appended into the cached assistant message, which carries isStreaming: true.

Persist and refresh

After the done event the finished assistant message is saved, and ['conversations'] is invalidated so the sidebar picks up the new title.

Backend

FastAPI is a separate service exposing /completion, /summarize, /health and /. The OpenAI SDK is pointed at Ollama's OpenAI-compatible endpoint (OLLAMA_BASE_URL, default http://localhost:11434/v1) -- the SDK is OpenAI's, the endpoint is not.

Two phases: acquire, then answer

Every turn runs as acquire → answer. The acquire pass decides whether the web is needed and runs the tools; the answer pass writes the reply, receiving any results as an evidence block in the user turn rather than as tool-role messages.

Keeping them apart is what lets web search and generative UI happen in the same answer. A tool-role message ties the answer to a tool-calling model, and the genui model at the time -- gemma3:12b -- cannot call tools at all, which is why search used to veto widgets. The current genui model does support tools, so that no longer binds locally, but the split still earns its keep: a hosted UI_MODEL_BASE_URL need not do tool calls, and the answer model stays free to be whichever one suits the turn.

The acquire model must be non-thinking. A thinking model spends its whole ACQUIRE_MAX_TOKENS budget reasoning and never emits the tool call -- measured against live Ollama, qwen3:8b scored 6/8 on the acquire suite where qwen2.5:7b scored 8/8, and took 4-20s a turn instead of under a second. Since search is on by default, that cost lands on every turn.

Model selection

The answer model is picked after acquire, so it follows what actually happened. Priority is images > thinking > evidence > default.

Condition	Model
Default	`phi4-mini:3.8b`
Images present	`qwen3-vl:8b-instruct`
Thinking mode	`qwen3:8b`
A search actually ran	`qwen2.5:7b`
Summarisation	`phi4-mini:3.8b`
Generative UI	`qwen3-vl:8b-instruct`

Evidence, not the toggle

Search is on by default, so keying the model off the flag would put every "hi" on the tool-capable model. It keys off has_evidence instead -- a turn that needed no search is still answered by the small default model.

Combining modes

The four switches are independent, so any combination can arrive. chat._resolve_answer reconciles them once, in one place:

Images pin the model and the client. A remote generative-UI endpoint may not accept image parts, and dropping your attachment is worse than dropping the widget model. The widget grammar still rides along, so "chart what is in this screenshot" works.
Generative UI outranks thinking for the token budget. Both want the window; a truncated widget spec renders as an error, while truncated reasoning is just shorter.
Evidence decides the model only when nothing above has claimed it.

Streaming

The backend emits NDJSON -- one StreamEvent per line.

lib/types/stream.ts

{ type: 'text' | 'reasoning' | 'done' | 'error', content: string }

The client parses the stream with an AsyncGenerator, appending text and reasoning as they arrive. Full event semantics are in the API reference.

Generative UI

An assistant reply can embed a widget by emitting a fenced ```breeze-ui block whose body is a single JSON object. The fence rides inside the normal text content, so it needs no new StreamEvent, no API change and no database field -- it persists and re-renders on reload for free.

The model emits data, never JSX. See the Generative UI guide.

Observability

Every LLM call can be traced with Langfuse. The OpenAI client is wrapped by Langfuse's auto-instrumentation, and the X-User-Id / X-Session-Id headers attribute each trace to a user and a session. See the Langfuse guide.

Architecture