Google Cloud launches Data Agent Kit for Airflow pipelines

Google Cloud has introduced Data Agent Kit, an open-source set of tools for building and managing data pipelines from an IDE or command-line interface. The product ties directly into Google Cloud's Orchestration Pipelines framework.

The launch targets data engineers, analysts and machine learning teams that use Apache Airflow but want to avoid writing large amounts of Python code to define and run workflows. The kit adds a dedicated Data Engineering tab for pipeline management and an agent-based tool that can author, deploy and troubleshoot Airflow DAGs through natural-language prompts.

Data pipelines sit at the centre of many analytics and machine learning systems, but orchestrating them often falls to specialist teams because of the coding and operational overhead involved. Google is trying to lower that barrier by combining agent-based assistance with a declarative, YAML-based approach that separates workflow logic from the underlying execution environment.

The toolkit is available for common development environments, including VS Code and command-line tools. It also supports Google's orchestration framework inside those environments, letting users create, validate and monitor pipelines without constantly switching between browser-based management consoles and local development tools.

Example workflow

To show how the system works, Google outlined a supply-chain use case based on predicting shipping transit times from warehouse location, customer location and order characteristics. The example uses the public TheLook Ecommerce dataset in BigQuery and splits the data into training and inference sets for demonstration purposes.

In that scenario, the toolkit generated PySpark scripts, dbt settings and three YAML pipelines from a natural-language prompt. The first pipeline extracts completed order data from BigQuery, runs model training on Managed Service for Apache Spark serverless infrastructure and uploads the resulting model to Gemini Enterprise Agent Platform Model Registry.

A second pipeline handles daily inference. It queries in-transit orders in BigQuery, runs batch prediction through Gemini Enterprise Agent Platform and writes the results back to BigQuery so customer support teams can identify possible service-level breaches.

The third pipeline evaluates prediction quality. It runs dbt models to compare predicted and actual delivery times, calculates error measures and can trigger retraining automatically if the model misses a set threshold.

Deployment and monitoring

The toolkit also produces continuous integration workflows for deployment to Managed Service for Apache Airflow. That is intended to reduce the need for separate deployment scripts and manual environment configuration, both common sources of delay for data teams moving from development into production.

Google is also positioning the toolkit as an operational tool, not just a code-generation layer. Users can monitor Airflow runs from within their IDE, inspect visualisations of pipeline structure and use a built-in troubleshooting function when jobs fail.

According to Google, the troubleshooting function analyses the failure context and distinguishes between infrastructure constraints, such as quota limits or memory shortages, and bugs in user code. It then suggests fixes inside the development environment.

Broader push

The release adds to a wider contest among cloud providers to make data engineering and machine learning workflows easier to assemble with agent-based software tools. Vendors are increasingly trying to bring orchestration, deployment and monitoring tasks into developer environments where teams already write code, rather than forcing them to use separate control planes for each stage of a data pipeline.

For Google, the move also reinforces the role of Airflow as a common orchestration layer even as companies expand their use of machine learning systems. By focusing on YAML definitions and natural-language prompts, Google is trying to make workflow creation accessible to a broader group of data workers, including those who are not specialists in Airflow internals.

The product spans several parts of Google's data and AI portfolio, including BigQuery, dbt, Managed Service for Apache Spark and Gemini Enterprise Agent Platform. That integration means organisations already using Google Cloud services may be able to create workflows that cover data extraction, transformation, model training, inference and evaluation in a single process.

Google said traditional MLOps architectures that combine data extraction, transformations, machine learning compute, model registry functions and conditional retraining often take weeks to assemble because teams must write and maintain complex orchestration logic. In its example, Google said the same lifecycle could be authored, deployed and maintained in minutes.