How Kythera Labs, a Databricks Constructed-On Companion, saves $2M+/yr utilizing Delta Sharing

Healthcare techniques generate monumental quantities of delicate knowledge, however shifting, sharing, and analyzing that knowledge securely throughout organizations continues to be a serious problem. On this publish, we’ll have a look at how we at Kythera Labs use Databricks and Delta Sharing to handle greater than 300 million affected person information and help collaborations throughout healthcare and life sciences. The weblog will cowl the sensible points with older knowledge‑sharing strategies, why we adopted Delta Sharing, and the affect it’s had on our storage prices, effectivity, and actual‑time collaboration.

Making Information Work in Healthcare: Kythera’s Method

Kythera Labs is an information know-how firm that empowers healthcare and life sciences organizations with a unified, high-fidelity healthcare knowledge platform for evaluation. As a built-on Databricks Companion, we selected Databricks and Delta Sharing not only for inside knowledge sharing but in addition to help seamless knowledge alternate with exterior companions. At this time, greater than 80% of our prospects use merchandise constructed on the platform. We additionally help exterior collaborations, together with organizations like Actual Sciences, utilizing Delta Sharing throughout 50 lively buyer workspaces.

Why Delta Sharing?

Kythera Labs selected Delta Sharing to beat important challenges in securely sharing healthcare knowledge. With over 300 million affected person information spanning a decade of scientific historical past, conventional strategies required creating and shifting a number of full copies of datasets, driving storage prices into the a whole lot of 1000’s of {dollars} and slowing supply.

Delta Sharing modifications that by enabling safe, actual‑time entry to reside knowledge with out creating duplicate copies. As an alternative of storing and sustaining separate datasets for every accomplice or setting, we are able to share a single, ruled supply of reality instantly. This strategy has allowed us to energy inside groups and exterior collaborations with simply 3.5 PB of storage, somewhat than the 20‑plus PB in any other case required.

One other complexity is assembly our prospects the place they’re on the cloud. Healthcare suppliers usually function in Azure, whereas many pharmaceutical corporations run on AWS or GCP. With no know-how like Delta Sharing, delivering giant datasets throughout clouds would imply pricey transfers, advanced ETL work, and a number of stale copies scattered throughout clouds. With Delta Sharing, we are able to immediately present safe entry to the identical reside dataset — regardless of the cloud — whereas sustaining compliance and eliminating pointless copies.

This not solely streamlines our inside workflows (shifting from growth to testing to manufacturing with out re‑copying knowledge) but in addition makes it simple for patrons to behave quicker, like immediately updating a most cancers remedy mannequin with the latest knowledge.

Changing Legacy Approaches

Given the exponential progress in knowledge quantity and complexity, conventional knowledge sharing strategies like SFTP servers are now not viable for contemporary wants. Shifting giant information backwards and forwards introduces delays, provides safety dangers, and requires storage of a number of redundant datasets.

Whereas APIs might be a useful resource, they’re inadequate for sharing the huge oceans of knowledge that organizations like Kythera handle. Counting on APIs to share the immense volumes of knowledge we handle could be like making an attempt to fill a swimming pool with a backyard hose—it’s technically attainable, however too gradual and inefficient for our wants.

Operationally, we deal with 7–10 million transactions every day whereas making certain compliance by way of our customized “Vault Structure” constructed on Delta Sharing. Prospects profit from real-time updates by way of view sharing with out guide intervention.

By adopting Delta Sharing, we’ve fully moved away from these legacy strategies and gained operational effectivity whereas enabling seamless collaboration throughout clouds and organizations.

Delta Sharing ROI

Delta Sharing has allowed us to eradicate legacy data-sharing strategies, lower storage wants by over 80%, and save greater than $2 million within the final 2 years. — Jeff McDonald, CEO, Kythera Labs

Delta Sharing helped Kythera lower storage wants from a projected 24 PB to simply 3.5 PB. Over three years, storage demand dropped from 17 PB/month in 2024 to 12 PB/month in 2023 and 6 PB/month in 2022. These reductions add as much as hundreds of thousands in financial savings. For context, giant pharmaceutical corporations can spend as a lot as $14 million every month simply on storage.

Storage is simply a part of the story. The compute prices for performing the ETL copies might be much more important, starting from equal to the storage financial savings to doubtlessly many instances larger, relying on the use instances.

Key Takeaways

Delta Sharing has remodeled our data-sharing capabilities by decreasing prices, bettering effectivity, and enabling real-time collaboration throughout clouds and organizations. The mix of Delta Sharing, Unity Catalog, and liquid clustering ensures scalability whereas sustaining compliance with healthcare knowledge requirements, exemplifying how open, fashionable knowledge platforms can revolutionize healthcare analytics.