◎ attribution: Bob Jones, licensed 'CC BY-SA 2.0'
A few years ago, when data was supposedly still “the new oil”, Data Lake projects sprung up in every major enterprise. Most of these projects were deemed as strategic and transformative for those organizations. Otherwise, no board would have committed the massive chunks of money that were poured into these endeavors. Large investments were put up because (pre-cloud) Data Lakes needed dedicated infrastructure, used a broad range of half-heartedly integrated emerging technologies and required very specific know-how. So, while the intent was business-driven, the implementation was technology-driven.
This led to a death-march-like fallacy: The technologist failed to build the Data Lake platform in time, requiring more money, even more tech experts and blocking the business from validating that mystical Data Lake thingy early. Support in business and other departments, which was readied early on when the project started, began to fade away because of delays.
I’ve been confronted with this kind of situation multiple times in my career: Overly relying on technology as a success factor of change. The overcommitment of tech and the lack of the business side to direct the change leads to the absence of actual benefits for the company and, without proper corrections, to complete failures.
One symptom I repeatedly observed especially struck me with Data Lake projects: When the Data Lake was declared “ready” there was no way to use it from a end-user perspective. There’s tools, the famous “Hadoop zoo”, there’s even a lot of precious data already loaded. But, no end user accounts, no login, no portal, no ports exposed, no business processes, no usage access! It’s like a library building full of great books without doors to get in.
So in 2019, I started to write it all down and build a concept for a user-centric portal for Data Lakes called “𝞳-teams”, putting the user first. It is built on Kubernetes, which at that time was definitely not a platform for Data Lakes and the discussion if persistent storage should live on kubernetes was still ongoing.
Around that time I gained more insights into how Data Scientists actually work (bummer: they don’t need Data Lakes per se, who could’ve thought that?). So 𝞳-teams was adopted a little bit more to Data Science. Then life happened, but 𝞳-teams is now available as a preview.
◎ attribution: Ann-Sophie Qvarnström, licensed 'CC BY-SA 4.0'
The benefits of 𝞳-teams start paying back for every data-centric org with more than one or two teams. With the Data Mesh hype cycle in full swing, a team-centric solution is needed. And while no tool can solve everything, 𝞳-teams is a piece to the puzzle. In fact, I think 𝞳-teams is the frame of the puzzle, outlining the boundaries and providing a good way to start different parts of the puzzle..
So, while Data Lake projects were killed by bad technology integration, missed deadlines, exceeded budgets and the failure to provide benefits for the potential users and the business as a whole, the idea to make use of data is still alive.
And if your organization is doing a team-first, data-first, technology-second approach, “𝞳-teams” is ready to get you started.
◎ attribution: public domain, licensed 'CC0'
Learn more now:
- Homepage: https://kteams.info
- Documentation: http://doc.kteams.info
- Youtube: https://www.youtube.com/channel/UC8ABOFbIpPAbwtXQf7TaZNw