QFM073: Machine Intelligence Reading List - July 2025

The paper proposes a set of design patterns for building LLM-based agents with provable resistance to prompt injection attacks, which exploit agents' reliance on natural language inputs to manipulate their behavior toward unauthorized actions. The authors systematically analyze these patterns' trade-offs between security and utility, demonstrating their real-world applicability across ten case studies ranging from OS function assistants to software engineering agents. The design patterns constrain agent actions to prevent solving arbitrary tasks while maintaining meaningful functionality without overly restricting capability.

Context engineering manages what information an LLM agent includes in its limited context window at each step, treating the context window like operating system RAM that must be strategically filled with instructions, knowledge, and tool feedback. As agents execute long-running tasks with accumulating feedback, they face problems like context poisoning, distraction, confusion, and clash that degrade performance and increase costs, making context engineering a critical challenge. Solutions fall into four categories: writing context outside the window (via scratchpads and cross-session memories), selecting relevant information, compressing context, and isolating separate concerns.

The author shipped Context, a native macOS app for debugging MCP servers, with approximately 95% of the 20,000-line codebase generated by Claude Code rather than hand-written. Claude Code proved particularly effective at SwiftUI development and excels when given detailed specifications and iterative feedback loops, though the author estimates it required substantial "prompt engineering" to maximize code quality and handle tasks beyond simple code generation like UI refinement and cross-cutting concerns.

In volatile markets with strong tailwinds, being early and riding market momentum ("levered beta") outweighs superior product quality or team capability—as evidenced by AI SDR companies like 11x and Lovable gaining market traction despite inferior products and leadership, simply because they captured early positioning in a category that benefits from the broader AI boom. The author argues that founders obsessed with building defensible competitive advantages ("alpha") miss that in today's explosive growth markets, mere correlation with market trends while maximizing leverage can generate outsized returns regardless of actual innovation or execution quality.

RAG systems like Kapa depend on high-quality, explicitly structured documentation to generate accurate AI responses, as the system retrieves discrete content chunks to answer user queries rather than reading narratives comprehensively. Documentation optimized for AI should be self-contained and contextually complete, with explicit relationships between sections and unambiguous information, since AI systems cannot infer unstated information or rely on implicit connections the way human readers can. The three-step retrieval process—chunking content, matching user questions to relevant sections, and generating responses—means that poor documentation directly degrades AI answer quality, creating a compounding problem where documentation improvements simultaneously benefit both human readers and machine comprehension.

Rather than building complex multi-agent systems that become brittle and difficult to debug, most LLM use cases are better served by simpler workflow patterns that keep humans in control of the task flow. The author argues that agent frameworks create the illusion of progress through complexity, but in practice agents fail due to tool selection errors and task management breakdowns—real problems he encountered building a three-agent research system—and that a deliberate progression from basic LLM augmentation through retrieval, tool use, and finally agentic control (only when necessary) prevents unnecessary failure modes.

The author argues that while AI and industry volatility present real challenges for coding careers, developers should view AI as an opportunity rather than a threat—AI will handle routine tasks, freeing humans to focus on creative problem-solving and novel ideas where human ingenuity remains irreplaceable. Success in this shifting landscape requires embracing continuous learning, contributing to shared knowledge commons, and mastering the skill of providing high-quality context to AI systems to amplify human capability rather than replace it.

Context engineering extends beyond prompt engineering to encompass the complete information payload—including examples, memory, retrieval, tools, state, and control flow—that an LLM receives at inference time. This GitHub repository provides a comprehensive first-principles handbook for designing, orchestrating, and optimizing context windows to provide models with precisely the right information for each inference step, drawing on research from major conferences like ICML and NeurIPS. Evidence shows that cognitive tools integrated into context can significantly improve model performance, such as increasing GPT-4's AIME2024 pass rate from 26.7% to 43.3%.

The post argues that the LLM industry is driven by unquantified, non-comparable anecdotal claims that lack critical context—including project type, codebase maturity, user expertise, and the non-deterministic nature of the systems—making it impossible to evaluate whether the technology actually works or simply appears to work in isolated cases. The author contends that hype and magical thinking dominate discourse, with high-profile claims receiving thousands of endorsements despite providing zero measurable details, and that questioning these claims brands skeptics as ignorant rather than appropriately critical.

Tech in Asia is a website focused on covering Asia's startup ecosystem, but requires JavaScript to be enabled for proper functionality.