DocETL: LLM-Powered Data Processing and ETL System

What changed

The ucbepic/docetl repository introduces a system for agentic LLM-powered data processing and ETL (Extract, Transform, Load). This open-source project, written in Python, aims to simplify the handling of large datasets, both structured and unstructured, by leveraging Large Language Models (LLMs). Instead of manually crafting individual LLM calls and orchestrating them, users can define operations in natural language, such as "pull out every complaint in this ticket." DocETL then provides the necessary operators like map, reduce, and filter, orchestrating them and parallelizing work across the data. A key feature is its automatic pipeline optimization, which aims to improve accuracy and reduce costs by dynamically swapping LLM models, rewriting prompts, decomposing operations, and replacing subtasks with code where feasible. The system is designed to return data in tabular formats, making it easy to query in databases.

DocETL offers multiple interfaces for defining and running data pipelines. The Python API is recommended for production code, notebooks, and scripting, providing a programmatic way to build and execute pipelines. For users preferring a low-code approach, a YAML configuration file can be used to declare pipeline steps and operations. Additionally, the DocWrangler UI offers a visual playground for interactive prompt development, allowing users to edit prompts and see results in real-time, accessible via a web interface or local deployment.

Installation is straightforward via pip (pip install docetl), and users need to set their LLM provider API keys (e.g., export OPENAI_API_KEY=your_key). The system supports various LLM providers and includes features for managing rate limits for LLM calls and tokens. For assistance in writing pipelines, users can utilize Claude Code by running docetl install-skill and describing their task, or by using ChatGPT or the Claude app with a specific prompt provided on the project's website.

Why it matters for builders

DocETL significantly lowers the barrier to entry for complex LLM-driven data processing. Developers can focus on defining the desired data transformations and insights using natural language, rather than getting bogged down in the technical details of API calls, prompt engineering, and model management. The automatic optimization capabilities, including model swapping and prompt rewriting, mean that pipelines can adapt to achieve better performance and cost-efficiency over time without constant manual intervention. This abstraction allows builders to integrate sophisticated LLM capabilities into their applications more rapidly and with greater confidence in the results.

Practical impact

Developers can start using DocETL by installing it via pip and setting up their LLM API keys. They can then begin building data processing pipelines using either the Python API or YAML configuration. For instance, the Python API example demonstrates classifying support tickets and then summarizing them by category, showcasing the map and reduce operations. The DocWrangler UI provides an interactive environment to experiment with prompts and visualize outputs, which can be particularly useful for rapid prototyping and prompt refinement. The project also offers a quick start guide for Claude Code, enabling users to generate pipelines by describing their tasks. The system's ability to output data into queryable tables facilitates seamless integration with existing data infrastructure.

Caveats and source limits

The provided source information details the functionality and usage of DocETL but does not include specific benchmark results comparing its performance or cost-efficiency against alternative ETL tools or manual LLM implementations. Pricing details for the system itself are not specified, though it relies on external LLM provider APIs, whose costs would apply. While the system claims automatic optimization, the exact mechanisms and effectiveness of these optimizations (e.g., MOAR - Multi-Objective Agentic Rewrites) are described in associated research papers (VLDB 2026) which are not fully detailed in the provided excerpts. The latest release version mentioned is 0.2.6. The project is associated with research from UC Berkeley, with multiple papers published or forthcoming (VLDB 2025, UIST 2025, VLDB 2026).