Local RAG and Deep Agents on Apple Silicon: a LangChain Learning Journey

Everything in this post runs on a MacBook Air (M4) with zero cloud API calls: a RAG pipeline over a 492-page NIST document, a cybersecurity deep research agent, and finally a multi-agent setup where a master model orchestrates three different analyst models and weighs their answers. The goal was never production — it was to learn the basics of each pattern with something simple but functional, and to document what actually breaks along the way.

The stack: LangChain 1.x, deepagents, Weaviate (self-hosted), Ollama and oLMX (an OpenAI-compatible MLX server) as local model backends.

Part 1: A Minimal RAG Pipeline

The architecture is the textbook one, split into two scripts:

• ingest.py — loads a PDF, splits it into chunks, embeds them, stores them in Weaviate
• query.py — interactive CLI: retrieves the top-k relevant chunks and streams the answer from a local LLM

The chunking config is deliberately boring — chunk_size=1000, chunk_overlap=200 — and it worked fine for a structured document like NIST SP 800-53.

Lesson 1: batch your embedding calls

The first ingest run crashed with a cryptic error from Ollama:

ollama._types.ResponseError: Post "http://127.0.0.1:50037/tokenize": EOF (status code: 400)

The cause: LangChain’s Weaviate integration passes all 2,111 chunks in a single embed request, and Ollama’s tokenizer endpoint gives up. The fix is trivial once you know it — batch the calls yourself:

for i in range(0, len(chunks), BATCH_SIZE):  # BATCH_SIZE = 50
    batch = chunks[i : i + BATCH_SIZE]
    vectorstore.add_documents(batch)

Forty-three batches later, the whole document was in Weaviate.

Does RAG actually help? Measure it

Instead of trusting the vibes, a third script (compare.py) asks the same question twice — once to the bare model, once through the RAG chain — and prints the answers side by side in two columns. For document-specific questions (“What does AC-2 require for account management?”) the difference is immediately visible: the bare model generalizes, the RAG answer cites the actual controls.

Part 2: A Cybersecurity Deep Research Agent

The second experiment uses LangChain’s deepagents package: an agent that plans with a todo list, searches a fixed allowlist of trusted sources (NIST, CISA, MITRE ATT&CK, SANS ISC, BleepingComputer, The Hacker News), reads the most promising pages, and answers with a summary plus references.

Two custom tools, shared by every variant in a common research_tools.py module:

• search_sites(query, site="") — DuckDuckGo search constrained with site: filters to the allowlist
• fetch_page(url) — downloads a page, strips the boilerplate with BeautifulSoup, truncates to 8,000 chars, and refuses any domain outside the allowlist

agent = create_deep_agent(
    model=ChatOllama(model="gemma4:12b-mlx", num_ctx=16384, temperature=0.1),
    tools=[search_sites, fetch_page],
    system_prompt=SYSTEM_PROMPT,
)

That one-liner hides the two bugs that cost me the most time.

Lesson 2: Ollama silently truncates your context

The first agent run produced… nothing. No tool calls, no answer, no error. The culprit: Ollama defaults to a 4,096-token context window, and deepagents injects a large system prompt (planning tool, filesystem tools, subagent machinery). The prompt was being silently truncated, and the model never saw its own instructions. num_ctx=16384 fixed it. If your agent behaves as if it never read the system prompt — it probably didn’t.

Lesson 3: agents need low temperature

The second silent failure was intermittent: same script, same question, sometimes a perfect research loop, sometimes an empty response. The model was running at its default temperature of 1.0 — fine for creative writing, terrible for emitting well-formed tool calls. At temperature=0.1 the flakiness disappeared.

Lesson 4: small models repeat themselves — cache the fetches

Watching the agent loop, one run fetched the same CISA advisory three times. A three-line URL cache in fetch_page solves it, and the cache-hit message doubles as feedback to the model:

if url in _page_cache:
    return "[NOTE: you already fetched this page — do not fetch it again]\n" + _page_cache[url]

The real saving is not the 0.1s HTTP round-trip — it’s not stuffing 8K duplicate characters into the context three times.

Part 3: Same Agent, Two Backends

Since oLMX exposes OpenAI-compatible APIs, pointing the same agent at a different runtime is a five-line change:

model = ChatOpenAI(
    model="gemma-4-12B-it-4bit",
    base_url="http://localhost:8000/v1",
    api_key=api_key,
)

Same question (“What is Log4Shell and what mitigations does CISA recommend?”), same tools, same Gemma 4 12B weights, two runtimes:

Both produced correct summaries with real references from the CISA advisory. Interesting detail: the oLMX run planned with the todo tool and worked more economically; the Ollama run was faster per token but wasted calls re-fetching the same page (pre-cache). With n=1 runs and different warm-up states this is an anecdote, not a benchmark — but that’s exactly the kind of comparison a two-backend setup makes cheap to explore.

Part 4: Multi-Agent — a Master and Three Analysts

The final experiment: a master agent that never researches anything itself. It delegates the same question to three analyst subagents — each a full deep research agent running a different model — then compares their reports and writes a weighted synthesis.

deepagents supports this natively — each subagent spec takes its own model (a string or a full model instance):

agent = create_deep_agent(
    model=make_model(MASTER_MODEL),
    tools=[],          # the master has NO research tools: it can only delegate
    subagents=analysts,  # each dict has its own model, tools, system_prompt
    system_prompt=master_prompt,
)

The trick that makes the pattern work is tools=[]: with no research tools of its own, the master’s only path to an answer is the task tool. Its system prompt then demands a specific output format: Consensus (facts at least two analysts agree on), Divergences (single-source claims, flagged as lower confidence), Summary, and a deduplicated References list.

And it genuinely worked. On the Log4Shell question the master correctly identified that all three analysts agreed on the CVE and the JNDI mechanism, and — more impressively — flagged that the analysts cited different patch versions (2.15.0 vs 2.17.1) and that only one of them mentioned network segmentation, correctly labeling both as lower-confidence divergences.

The cost: model swapping

Total time: 17m 56s. Almost all of it is Ollama swapping four models in and out of RAM (26B → 12B → 8B → 4B → 26B again for the synthesis). The actual inference is a fraction of that. On a memory-constrained laptop the levers are obvious: use a smaller master, use analysts closer in size, or split master and analysts across two runtimes (oLMX keeps its models resident independently of Ollama).

Takeaways

1. Local-first is a great teacher. Every failure mode was mine to debug — no rate limits or provider magic hiding the mechanics.
2. The errors are rarely where you look first. A crashing ingest was a batching problem; a mute agent was a context-window problem; a flaky agent was a temperature problem. None of them said so in the error message.
3. Share the tools, swap the brains. Once tools and prompts live in a common module, comparing backends and models becomes a config change — and cross-model comparison is exactly what makes the multi-agent pattern interesting.
4. Small models can orchestrate. A local 26B master reliably followed a delegate-then-synthesize protocol and produced a genuinely useful consensus/divergence analysis of its analysts’ reports.

All of it in a few hundred lines of Python, running on a laptop.