Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
This article dives headfirst into the contentious world of self-hosting LLMs, arguing for local inference to safeguard intellectual property from 'devious' frontier providers. It details the technical 'gotchas' of migrating large prompts to smaller, self-managed context windows, sparking a robust debate among Hacker News readers about prompt engineering, hardware feasibility, and the true capabilities of AI agents.
The Lowdown
This post critiques large AI providers like Anthropic and OpenAI, accusing them of being 'untrustworthy' and 'actively devious,' potentially training on user data and insights—a concern heightened by recent public dramas like the Navier-Stokes equation incident. The author champions self-hosting LLMs as the only verifiable solution for privacy and control.
- Motivation for Self-Hosting: To protect valuable intellectual property and insights, which the author likens to 'metadata about your sessions,' from frontier providers perceived as 'pirates' who might steal or train on user data.
- Critique of Frontier AI: The author lambasts providers for their 'ethical deflection,' 'grandstanding on cybersecurity,' and 'safety filters' that inadvertently hinder legitimate security research by preventing 'hacking' related discovery of vulnerabilities.
- Migration Challenges: The core technical challenge involves adapting large, complex prompts (e.g., 35kb) designed for the expansive context windows of frontier models (e.g., 1M tokens) to the much smaller capabilities of self-hosted setups (e.g., 65k tokens in Ollama).
- Technical 'Gotchas': With limited context, local models suffer from 'context exhaustion,' leading to 'thrashing' (repeated tool calls, re-reading files, losing awareness of past instructions).
- Solutions & Best Practices: The article proposes 'Single Objective Prompting,' advocating for breaking down complex prompts into smaller, declarative units, managing session state externally, explicitly tuning Ollama's context length, and reducing tool calls per agentic step.
- Failure Signals: Indicators of context exhaustion include identical tool calls, multiple file reads on the same file, agents restating objectives, and tool-call parse failures.
- The Faustian Bargain of Large Context: While large context windows enable 'Chain of Thought' and can mask poorly constructed prompts, they make users dependent on frontier providers and their data collection practices, reinforcing the need for 'Total Custody of Output' via self-hosting.
Ultimately, the article serves as a call to action for developers to embrace sovereign, self-hosted AI for security and privacy, providing practical guidance on navigating the associated technical hurdles.
The Gossip
Prompting Principles & Practical Pointers
Many commenters concurred that a 35kb prompt is likely symptomatic of poor prompt engineering, arguing that complex tasks should be broken down into smaller, single-objective units. They emphasized that even frontier models exhibit 'context rot' beyond certain token limits (often around 250k), despite advertised larger windows. The discussion highlighted strategies like 'divide and conquer' and using external 'harnesses' to manage state and context, preventing the LLM from 'forgetting' previous steps. Some also pointed out that system prompts might contain a lot of data rather than just human prose, which can bloat the context.
Hardware Hurdles & Home-grown Horsepower
A significant portion of the debate centered on the substantial hardware requirements for effective local LLM inference. While some lamented that even a $10,000 investment might yield 'mediocre performance,' others presented more optimistic views. They cited recent advancements in open-source models, particularly the Qwen family, and specialized hardware setups (e.g., dual R9700 or new Mac Studio Ultras), suggesting that a $4k-$5k investment can now provide excellent speed and quality for 27B parameter models, often matching or exceeding hosted provider performance for concurrent sessions with decent context windows.
Critiques & Content Conundrums
Several readers expressed disappointment, feeling the article merely reiterated well-known local LLM 'dead-ends' without offering novel solutions or deeper technical insights they hadn't already encountered. Some found the author's tone overly aggressive or the problem statement insufficiently developed. A specific point of contention was the author's choice of Ollama, with multiple comments suggesting `llama.cpp` as a superior alternative and linking to a 'Friends Don't Let Friends Use Ollama' discussion.
Autonomous Agents & Advanced Approaches
An interesting counter-narrative emerged from users describing successful experiences with frontier models acting as highly autonomous agents. One commenter detailed using models like Opus 5 to tackle 'ticket-level' coding tasks, with the LLM planning, deploying, and testing solutions over long sessions (600k-700k tokens) without explicit context management. This highlights a perceived gap between the constraints of local setups and the advanced, high-context agentic workflows some users achieve with powerful cloud models, prompting discussion on how much autonomy can be safely delegated to an AI.