With Genie Code, knowledge engineers can use pure language to generate production-ready knowledge pipelines, orchestrate them with jobs, and debug failures. Duties that used to take weeks – discovering knowledge, constructing transformations, stitching collectively jobs, and fixing failures – can now be executed in hours, whereas staying aligned with governance and operational requirements.
Under, we’ll stroll via how this works in apply: discovering knowledge, constructing pipelines, orchestrating jobs, and debugging failures, all from a single dialog.
Construct and orchestrate full, production-ready pipelines and jobs utilizing pure language
Genie Code can now take you from exploration to scheduled pipelines and jobs in a single thread, serving to you writer and function them end-to-end.
It accelerates the event of Lakeflow Spark Declarative Pipelines and simplifies how pipelines and notebooks are orchestrated and run via Lakeflow Jobs. Genie Code understands your pipeline and job context, accessing the code, configuration, and run outcomes.
Genie Code helps throughout key levels of the info engineering lifecycle:
- Search over knowledge belongings, not simply code: Genie Code makes use of reputation, lineage, code samples, and Unity Catalog metadata to establish probably the most related datasets in your activity. For instance, you possibly can ask Genie Code to clarify how tables relate or hint how knowledge flows via a pipeline. At SiriusXM, groups use Genie Code to know desk relationships extra shortly.
- Construct and modify pipelines: Begin by describing the pipeline you need in plain language, corresponding to a fraud detection pipeline constructed on a medallion structure. Genie Code generates a Spark Declarative Pipeline with Bronze, Silver, and Gold layers, together with sources, transformations, knowledge high quality expectations, and outputs. From there, you possibly can ask for modifications, assessment the proposed diffs, and run and check the pipeline.
- Outline and orchestrate jobs: No must manually outline and preserve orchestration logic. You describe the job you need, together with duties, dependencies, and schedule. Genie Code configures it for you, then helps modify, debug, and repair orchestration points in pure language.
- Lengthen and evolve present workflows: As necessities change, Genie Code helps you replace pipelines and jobs with new datasets and transformations. It understands the present construction and outcomes of your pipelines, and might prolong them by writing AutoCDC flows for change knowledge seize, configuring Auto Loader, making use of knowledge high quality expectations, and following the medallion structure.
- Embrace finest practices with Declarative Automation Bundles (DABs): Genie Code can work immediately inside your present DABs initiatives: including sources, updating configurations, validating bundles, and deploying to your targets. So you possibly can undertake software program engineering finest practices like supply management, testing, and CI/CD in your knowledge initiatives with out hand-writing YAML.
- Work sooner with out decreasing requirements: These capabilities cut back guide effort whereas maintaining workflows aligned with enterprise necessities. Pipelines stay ruled via Unity Catalog and comply with established patterns for efficiency and knowledge high quality, whereas jobs inherit constant configuration for scheduling, retries, and dependencies. Knowledge engineers keep in management, however spend much less time on repetitive work.
Monitor, diagnose, and debug pipelines and jobs
- Understanding and enhancing pipeline conduct: Genie Code can examine datasets and pipeline outputs that will help you perceive a pipeline end-to-end. For instance, it may summarize transformations, hint how knowledge flows into downstream tables, and spotlight surprising modifications in row counts or schemas.
- Debug and diagnose job and pipeline failures: When a pipeline or job fails, Genie Code helps you’re employed via the difficulty. It analyzes errors, proposes updates throughout the related recordsdata, and reveals you the diffs earlier than making use of any modifications. You may assessment every replace and determine what strikes ahead. This turns lengthy, guide debug cycles into sooner guided iterations.
- Lengthen and customise Genie Code: Genie Code isn’t restricted to built-in capabilities. Groups can prolong it with customized directions, agent expertise and combine exterior programs via MCP servers, permitting Genie Code to function on domain-specific logic, inside instruments, and customized workflows. This ensures Genie Code adapts to your atmosphere and area data.
What’s subsequent
Extra capabilities are coming to increase Genie Code throughout pipelines, jobs, and the broader platform. One thrilling function on the horizon is AI-optimized workloads. Sooner or later, you possibly can enable Genie Code to additionally run within the background to maintain your platform working effectively, so you possibly can hand off these repetitive and time-consuming duties. This contains responding to job failures and managing routine upgrades, but in addition mechanically right-sizing cluster use.
Curious to study extra about these updates and finest practices? Make certain to register for Knowledge+AI Summit the place we’ve a whole bunch of periods protecting Genie Code, Lakeflow and far more!
Strive Genie Code’s knowledge engineering capabilities
Open Genie Code in agent mode and ask it that will help you construct or replace your pipelines and jobs. Take a look at the demo for extra particulars .
Assessment the documentation to study extra.