We’re excited to announce that we are wrapping up Sprint 1 for the Auto Data Warehouse PostgreSQL extension. It’s been a busy past two weeks! What have we done? We learned how to create Postgres extensions in Rust, so we can take our automations to a whole new level. Additionally, we've refined our vision, updated our script, and crafted a concise set of functions to make building a data warehouse easier than ever before.
1-Click DW Magic - Really, it’s one function GO!
We reimagined the standard UI typically used for DW design. Making functions straightforward and user-friendly while allowing for a multitude of future complexities. This led to our “1-click” data warehouse implementation. Really, it’s a single function “go.” Our idea is you install the PG_AUTO_DW PostgreSQL extension, run the function auto_dw.go(), and your whole data warehouse is built for you!
/* Build me a Data Warehouse for tables that are Ready-to-Deploy*/ SELECT auto_dw.go();This is the ideal, and if it can’t convert some source tables, you’ll get some helpful hints on how to either institute data governance best practices or provide context to the LLM. But first, let’s take a step back and look at the goals for this project as stated on the GitHub README page:
Goals: This extension will enable users to: - Automate the DW Build - Automate DW Maintenance - Understand DW Health - Support Data Governance All these capabilities will be delivered through a small set of intuitive functions.
As seen above, our last goal is to “support data governance.” This creates a huge opportunity and a feedback loop for the Auto DW system. Why? As a good Data/BI Engineer would tell you, the best place to issue data governance is as close to the source data as possible. Moreover, the better your data governance, the better your source system, and the better the LLM will be able to understand your data.
All this can get a little confusing even though we try to make it as simple as possible. So, we created a Demo with 2 Acts. Please check out our GitHub page for more details! Some functionality already converts source tables into data warehouse tables, but most functions serve as illustrative purposes only, hinting at what’s to come in subsequent sprints.
We hope this makes you think about getting involved with the PostgreSQL Auto Data Warehouse project or inspires you to start developing your own PostgreSQL extension!