SQLite is all you need for durable workflows
This post argues that for many durable workflow needs, particularly with AI agents, SQLite combined with Litestream is
The Lowdown
The author builds on the idea that a robust database can replace complex orchestration tiers, positing that SQLite—often dismissed for production—is sufficient for a large class of durable workflows. This approach simplifies infrastructure by leveraging SQLite's transactional local storage and Litestream for asynchronous cloud backup.
The core argument for SQLite’s suitability rests on several points:
- Decoupled Durability: Workflow state is the durable part, while compute can remain cheap and disposable, making local transactional storage ideal.
- Simplicity: SQLite provides transactional durability without requiring a separate database server, eliminating network hops, control planes, and additional operational overhead.
- Litestream Integration: Litestream streams SQLite changes to S3-compatible object storage, offering a simple way to back up and inspect workflow states, although replication is asynchronous.
- AI Agent Fit: This model is particularly attractive for bursty, experimental AI agents where self-contained, isolated state is beneficial, reducing complexity and improving fault isolation compared to large shared systems.
While SQLite offers significant advantages for certain use cases, the author acknowledges that Postgres remains the better choice for high availability, broad shared scalability, or when synchronous replication is a strict requirement. However, for many initial workflow systems, the local SQLite database plus Litestream provides a lightweight yet durable solution, especially for the evolving landscape of AI agents.
The Gossip
SQLite's Scalability Scrutiny
Commenters fiercely debated the suitability of SQLite for production, particularly concerning concurrency and scaling. Critics argued SQLite is fundamentally unsuitable for multi-process or multi-machine concurrency, viewing its proponents as inexperienced. Proponents countered that SQLite excels in single-node applications, offers surprising performance benefits over more complex databases like Postgres in certain scenarios, and is excellent when strong data partitioning exists. The consensus emerged that SQLite works well for its specific niche (local, single-writer, partitioned data) but falls short for highly concurrent, distributed systems.
Durability Doubts with Litestream
A significant point of contention was the article's claim of 'durable workflows' given Litestream's asynchronous replication. Many questioned the true durability when local writes could be lost before synchronization, especially during system failures. Concerns were raised about Litestream's production readiness, with users reporting past bugs related to runaway disk usage and data syncing issues. While some acknowledged that even Postgres requires specific configurations for synchronous replication, the asynchronous nature of Litestream was seen by many as a critical vulnerability for systems requiring strong durability guarantees.
Alternative Orchestration & Storage Options
The discussion expanded to compare the proposed SQLite/Litestream approach with other workflow orchestration tools and storage paradigms. Temporal was highlighted as a robust, more feature-rich alternative for complex, reliable workflows, with some suggesting it offers similar philosophical benefits but with a richer interface. Other commenters debated the merits of using traditional database servers like Postgres for high-availability needs, or even simpler flat files for less complex data storage, emphasizing that the 'right' solution depends heavily on specific use cases and scalability requirements.