At present marks an essential milestone for open requirements and the broader knowledge group. We’re working with Microsoft to combine the info managed each in Unity Catalog and OneLake utilizing open requirements. The Unity Catalog Open API specification gives an interface to alternate knowledge each methods, zero copy, to open up entry to your entire lakehouse ecosystem for engines in both platform.
For practically a decade, Databricks and Microsoft have partnered to assist organizations unlock the worth of their knowledge by means of the openness and adaptability of the lakehouse. At present, that partnership takes one other step ahead, giving prospects extra selection in how they entry and govern knowledge throughout platforms.
The Path to Open Interoperability
As open desk codecs like Delta Lake and Iceberg gained adoption, new silos emerged on the catalog-level. To handle this, Databricks launched Unity Catalog Open APIs, establishing an open normal for unified governance throughout knowledge and AI, whereas additionally absolutely supporting the Iceberg REST catalog API for tabular knowledge. This dual-API strategy empowers organizations to undertake best-in-class options and keep away from vendor lock-in whereas seamlessly sharing knowledge throughout many knowledge platforms and SaaS Purposes. We additionally launched catalog federation to offer a single ruled view throughout all knowledge platforms. This mannequin, open APIs powering catalog federation, has grow to be the trade normal for interoperability.
At present, we’re excited to see this open imaginative and prescient validated by our companions at Microsoft.
Microsoft provides help for Unity Catalog Open APIs
As we introduced with our companions at Microsoft at this time, our collaboration is deepening.
Microsoft has already added help for studying from Azure Databricks with OneLake Mirroring, permitting Material engines to entry the entire knowledge managed in Unity Catalog seamlessly. Not like different OneLake integrations with proprietary platforms, this makes all knowledge in Unity Catalog – together with the best efficiency tables – accessible in OneLake with out copying, altering code, or extra prices.
At present, Microsoft and Databricks are extending their openness by integrating much more natively. With Microsoft OneLake including help for the Unity Catalog Open API normal, Azure Databricks customers will have the ability to seamlessly learn knowledge from OneLake.
With OneLake including help for open catalog APIs together with Iceberg REST and Unity Catalog Open APIs, Microsoft is making OneLake structured and unstructured knowledge from providers like Dynamics, Dataverse, Blob retailer, and so on. natively accessible to your entire open lakehouse ecosystem, together with Databricks.
Databricks Builds on the Open Basis
Leveraging this expanded functionality, Databricks is including help to make all OneLake knowledge seamlessly accessible in Unity Catalog.
This allows zero-copy studying of OneLake knowledge from inside Databricks guaranteeing seamless interoperability and entry throughout each platforms, giving Databricks prospects entry to all their knowledge within the Microsoft ecosystem to energy their most superior AI, machine studying, and large-scale knowledge engineering workloads.
Catalog Integration
- Studying from Unity Catalog in Material – Typically Accessible Now
Microsoft has launched Azure Databricks mirroring. Clients can mirror Azure Databricks knowledge into OneLake, by means of Unity Catalog guaranteeing that ALL knowledge managed by Azure Databricks is immediately accessible throughout Microsoft Material workloads. Each platforms can work over the identical copy of information with no knowledge motion. - Studying from OneLake in Azure Databricks – EOY 2025
This new functionality, permits Azure Databricks to learn knowledge from OneLake through their Unity Catalog Open APIs and permits all your OneLake knowledge to be leveraged seamlessly in Databricks. Clients can create Unity Catalog Connections to OneLake primarily based on Managed Identities from EntraID, and might seamlessly mount knowledge objects from any Material workspace, straight as a catalog in Unity.
Connecting to OneLake knowledge in Azure Databricks
The Future is Open
This milestone displays how far the trade has come towards an open, interoperable knowledge ecosystem. Clients now not want to decide on between innovation and interoperability—they will have each.
With Microsoft now embracing the open catalog normal, we’re excited to proceed constructing the way forward for knowledge and AI on this basis. Keep tuned for extra bulletins and roadmap updates, together with help for Azure Databricks writing knowledge straight in OneLake, as we deepen our integration.