Untangling Data Mesh, Fabric, and Lakehouse

[Three virtual half day sessions from 9:00 until 13:00 hrs CET Amsterdam time zone.]

The data warehouse is now middle-aged. BI has turned twenty and the data lake ten. So, is it time for something new? Something better? In fact, three new frameworks have recently emerged: data fabric, data mesh, and data lakehouse.

But what are they? Are they truly novel or simply marketing hype? In essence, all three are competing technological responses to the growing needs of digital transformation. If you are facing the urgent challenges of delivering high-value, consistent, and near real-time information across Cloud and on-premises environments, understanding these approaches and their differences is critical. Learn if you should move beyond your existing data warehouse or lake and, if so, how.

In this course, Barry Devlin explains and positions data fabric, mesh, and lakehouse, as well as other concepts, old and new, using a logical digital information systems architecture framework. Exploring existing and emerging technologies as well as organisational issues, methodologies, and implementation approaches, Barry will help you decide if one of these new approaches is right for your business needs, existing technical environment, and current skills.

The seminar

Three new data delivery and information preparation approaches have recently emerged to challenge current data warehouse and data lake thinking. Each has its pros and cons, strengths and weaknesses. Data fabric proposes active metadata and knowledge graphs to power a logical data warehouse approach. Data mesh suggests that a domain-oriented, self-service approach based on microservices thinking should be adopted, eliminating data copies almost entirely. Data lakehouse, as the name implies, attempts to combine the best of data warehouse and data lake, as well as promising transactional consistency within its scope.

All these new and old terms, with partially overlapping scopes, preferred technology bases, and different promoters, are types of digital information systems. Their aim is to prepare, deliver, and manage data / information to all decision-making and action-taking business processes.

Today’s business, with its conflicting needs for data timeliness vs. consistency, immediate vs. correct decisions, and information-informed competition, places extensive new demands on digital information systems. These demands have led to the emergence of the lakehouse, fabric, and mesh approaches honed for the complex distributed and network-centric environments that are already common. However, the prevalence of legacy systems, historical data management issues, as well as existing and evolving complications in information meaning and usage mean that traditional approaches and knowledge cannot be readily abandoned.

The key questions, therefore, are if and how data lakehouse, fabric, and mesh address these new needs, how differently each does that, where they improve on existing approaches or create new problems, and how they can coexist with or replace established data warehouses and lakes.

To answer these questions, Barry Devlin compares and contrasts all these approaches, old and new, using as a foundation the Digital Information Systems Architecture (DISA) first defined in “Business unintelligence.” Existing and emerging technologies for data storage, preparation, and virtualization; data catalogs and knowledge graphs; and other tools, both on-premises and Cloud, are described and analysed. Also explored are a wide variety of organisational issues, methodologies, and implementation approaches that are often as important in assessing solutions as the underlying technologies.

In his new book Cloud Data Warehousing, Dr. Barry Devlin builds the architectural and systems foundations for data lakehouse, data fabric, and data mesh, as well as the base cloud data warehouse. Barry offers initial answers to key questions: Why are there so many approaches to cloud data warehousing? How do they differ? Which may be best? And why? Drawing lessons from the history of data warehousing, he defines an all-embracing architecture and draws specific architectural design patterns for each of these modern approaches. And he discusses the various choices and paths from current systems to the different solutions.
As “the book of the course,” Barry is offering a special discount to course attendees.

What you will learn

• Origins, drivers, meaning, and detailed functionality of data lakehouse, fabric, and mesh
• Benefits—business and technical—and lessons learned for each approach
• An introduction to the technical rationale, structure and components of the conceptual and logical Digital Information Systems Architecture (DISA) and its business and technical uses
• An in-depth comparison of data fabric and mesh with data warehouse, lake, hub, and lakehouse using DISA as a basis
• Possibilities and challenges of new database and data management technologies in Cloud, on-premises, and hybrid environments
• The central role of context-setting information, knowledge systems, and metadata
• Adaptive Processes as the basis for data preparation, information creation, and insight discovery
• Using data virtualization and preparation as tools for integration of all types of content and data in Cloud, on-premises, and hybrid environments
• Practical planning and implementation steps from data warehouse / lake to data lakehouse, fabric, or mesh.

DISA is the new name for the conceptual and logical architectures introduced in “Business unIntelligence”

Who should attend?

• Enterprise, systems, solutions and data warehouse architects
• Systems, strategy and business intelligence managers
• Data warehouse, data lake and IT systems designers and developers
• Data and database administrators
• Tech-savvy business analysts.