Implementing Data Mesh Katowice

As data continues to grow and become more complex, organizations seek innovative solutions to manage their data effectively. Data mesh is one solution that provides a new approach to managing data in complex organizations. This practical guide offers step-by-step guidance on how to implement data mesh in your organization. In this book, Jean-Georges Perrin and Eric Broda focus on the key components of data mesh and provide practical advice supported by code. Data engineers, architects, and analysts will explore a simple and intuitive process for identifying key data mesh components and data products. Youll learn a consistent set of interfaces and access methods that make data products easy to consume. This approach ensures that your data products are easily accessible and the data mesh ecosystem is easy to navigate. This book helps you: Identify, define, and build data products that interoperate within an enterprise data mesh Build a data mesh fabric that binds data products together Build and deploy data products in a data mesh Establish the organizational structure to operate data products, data platforms, and data fabric Learn an innovative architecture that brings data products and data fabric together into the data mesh About the authors Jean-Georges "JG" Perrin is a technology leader focusing on building innovative and modern data platforms. Eric Broda is a technology executive, practitioner, and founder of a boutique consulting firm that helps global enterprises realize value from data. Spis treści: Foreword Preface Who Is This Book For? Overview of the Parts and Chapters What This Book Isnt Conventions Used in This Book OReilly Online Learning How to Contact Us Acknowledgments I. The Basics 1. Understanding Data Mesh: The Essentials Making Data Agile Local Autonomy + Speed = Agility Solving Todays Data Challenges Bridging Data Silos Shifting Toward Higher Quality Data Transforming Data Governance Data Volume, Variety, and Variability Turning Principles into Practice Summary 2. Applying Data Mesh Principles Data Mesh Principles Data as a Product Decentralized Domain Ownership Self-Serve Data Platform Federated Computational Governance Defining a Good Data Product Defining a Principled Data Product Defining a FAIR Data Product Defining an Enterprise-Grade Data Product Defining a Valuable Data Product Defining a Balanced Data Product Defining a Modern Data ProductMore than Just Regular Data Defining a Practical Data Product Lifecycle Defining a Practical Data Mesh Roadmap A Good Data Product Has an Empowered Data Product Owner Identifying a Data Product Summary 3. Our Case Study: Climate Quantum Inc. Making Climate Data Easier to Find, Consume, Share, and Trust Introducing Climate Quantum Inc. Climate Quantum Data Landscape Applying Climate Quantum Inc. to Your Enterprise Summary II. Designing, Building, and Deploying Data Mesh 4. Defining the Data Mesh Architecture Data Product Architecture Data Product Artifacts Development Architecture Components Data product definition Artifact definition Policy definition Runtime Architecture Components Data product gateway Ingestion interfaces Consumption interfaces Operations Architecture Components Discoverability interfaces Observability interfaces Governance interfaces Control interfaces Data Mesh Architecture Data Mesh Marketplace and Registry Data Mesh Backbone Services Climate Quantum Use Case Considerations Summary 5. Driving Data Products with Data Contracts Bringing Value Through Trust Navigating the Data Contract Going Through the Theory Stacking Up Good Information Its All About Proper Versioning Keeping It Simple and Semantic Walking Through an Example: Complementing Tribal Knowledge Documenting in a slightly better way Creating a human lineage What is Data QoS and Why Is It Critical? Representation Periods Groups Why Does It Matter? Why Data Quality Is Not Enough Accuracy (Ac) Completeness (Cp) Conformity (Cf) Consistency (Cs) Coverage (Cv) Timeliness (Tm) Uniqueness (Uq) Service Levels Complement Quality Availability (Av) Throughput (Th) Error rate (Er) General availability (Ga) End of support (Es) End of life Retention (Re) Frequency of update (Fy) Latency (Ly) Time to detect (an issue) (Td) Time to notify (Tn) Time to repair (Tr) Applying Data QoS to the Data Contract Checking Conformity of Measurements Completeness Accuracy Engaging Service Levels Summary 6. Building Your First Data Product Anatomy of a Data Product The Data Contract to the Rescue Connecting Your Data Sources Ensuring Higher Data Quality with Observability Getting Faster Data Discovery Enabling Operability and Control Building and Packaging Summary 7. Aligning with the Experience Planes The Three Planes The Infrastructure Plane Remains Key The Data Product Experience Plane Is Independent The Mesh Experience Plane Is About Synergy Building a Capability Model for Each Plane Capabilities of the Infrastructure Experience Plane Capabilities of the Data Product Experience Plane Capabilities of the Mesh Experience Plane Gathering Key Metrics Through Feedback Loops Summary 8. Meshing Your Data Products Registering Your Data Product Connecting to Your Products in the Mesh Meshing Data Products Together Describing Your Data Lineage Notification Through the Mesh Summary III. GenAI, Teams, Operating Model, and Roadmap for Data Mesh 9. Running and Operating Your Data Mesh Making Data Products Discoverable, Observable, and Secure Data Product Discovery Interface Data Product Observability Interface Data Product Control Interface Data Product Security Climate Quantum Use Case Considerations Toward Dynamic Data Products The Power of Dynamic Data Products Summary 10. Creating a Data Mesh Marketplace Challenges with Traditional Data Catalogs The Data Mesh Marketplace A Window into Data Mesh The Consumer User Journey The Producer User Journey Self-Publishing of Data Products The Power of Discoverability Bootstrapping Your Marketplace Climate Quantums Data Mesh Marketplace Summary 11. Establishing Data Mesh Governance Data Governance Data Product Certification Federated Data Product Certification Implementing Data Product Certification Data Contracts and Certification Climate Quantums Certification Approach Summary 12. Understanding Data Product Supply Chains The Modern Manufacturing Supply Chain The Modern Software Factory The Data Product Factory The Data Product Supply Chain Climate Quantums Data Product Supply Chain Summary 13. Integrating Data Mesh and Generative AI Generative AI Background Large Language Models Embeddings Vector Databases Prompts Challenges Data Mesh and Generative AI Improve Data Quality with Generative AI Elevate Models as Primary Artifacts Consume Models Inside Data Products Generate Data Product Code Complement Data Product Capabilities Climate Quantum and Generative AI Simplifying Climate Data Search Analyzing and Summarizing Climate Data Creating Disclosure Reports Generating Code to Simplify Climate Data Processing Summary 14. Establishing Data Mesh Teams Team Topologies in Data Mesh The Data Product Team Key roles and responsibilities Data product owner Release management Metadata engineering Data and security engineering Consumption engineering Ingestion engineering Platform Teams Enabling Teams Climate Quantum Teams Summary 15. Defining a Data Mesh Operating Model Characteristics of an Operating Model The Operating Model Continuum Centralized Organizations Matrixed Organizations Federated Organizations Distributed Organizations Data Mesh Operating Model Data Mesh Maturity Model Climate Quantums Data Mesh Operating Model Summary 16. Establishing a Practical Data Mesh Roadmap Roadmap Structure Strategy and Roadmap Stream Technology Stream Foundational phase deliverables Industrialization phase deliverables Factory Stream Operating Model Stream Socialization Stream Rollout Stream Climate Quantums Data Mesh Roadmap Summary Index