Monitoring with Graphite. Tracking Dynamic Host Katowice

Graphite has become one of the most powerful monitoring tools available today, due to its ease of use, rapid graph prototyping abilities, and a friendly rendering API. With this practical guide, system administrators and engineers will learn how to use this open source tool to track operational data you need to monitor your systems, as well as application-level metrics for profiling your services.Author Jason Dixon, member of the Graphite project, provides a thorough introduction of Graphite from the basics to the skills and tools you need for troubleshooting and scaling out its software components. If you want to learn more about monitoring systems, services, or applications, this is the book you need.Get an introduction to monitoring, including important concepts and terminologyExamine the features and functionality of key Graphite components, including Carbon and WhisperLearn the typical user workflow necessary to create a basic line chartBuild complex charts with chained functions and multiple axes that interact directly with the rendering APIUnderstand how to use the native Graphite dashboard, as well as the more popular third-party dashboardsMaster the art of scaling and troubleshooting high-performance or highly available Graphite clusters Spis treści: Preface Who Should Read This Book Why I Wrote This Book A Word on Monitoring Today Navigating This Book Conventions Used in This Book OReilly Safari How to Contact Us Acknowledgments 1. What Is Graphite? What Is Time-Series Data? Time-Series Databases Storage Considerations Prioritizing Workloads What Is the History of Graphite? What Makes Graphite Unique? Simple Metrics Format Graphing API Rapid Prototyping Rich Statistical Library Chained Functions Case Studies: Who Uses Graphite in Production? Booking.com GitHub Etsy Electronic Arts Why Should I Use Graphite? 2. Monitoring Conventions Three Tenets of Monitoring Fault Detection Alerting Capacity Planning Rethinking the Poll/Pull Model Pull Model Push Model Where Does Graphite Fit into the Picture? Composable Monitoring Systems Telemetry Collection agents and sensors Application instrumentation Logging Business telemetry Metrics Router Aggregation State Engine Notification Routers Storage Engine Visualization Conclusion 3. Graphite Components: The Moving Parts Carbon carbon-cache carbon-relay Consistent-hashing Rules-based routing Aggregated-consistent-hashing carbon-aggregator Filtering Metrics Internal Statistics Network Security Considerations Whisper How Do Whisper Files Get Created? Retention Policies and Archives The Laws of Whisper Archives Calculating Whisper File Sizes Deconstructing a Whisper File Which Archive Handles My Query? Aggregation Methods xFilesFactor Planning Your Namespaces Performance Considerations Graphite-Web Django Framework Web Server Database SQLite PostgreSQL versus MySQL Memcached Events Storage Backends RRD Ceres Cyanite Putting It All Together Basic Setup Vertical Scaling Horizontal Scaling Multisite Replication A Final Thought 4. Building Your First Graphite Server Quick Start with Synthesize Running Synthesize in Vagrant Running Synthesize manually Next steps with Synthesize Where Does Graphite Store All My Files? Are Packages Available for My Distro? What Installation Methods Are Available? Should I Use Virtualenv? Using sudo Effectively Dependencies Debian/Ubuntu RHEL/Fedora/CentOS Installing from Source Preparing Your Web Database Configuring Carbon carbon.conf storage-schemas.conf storage-aggregation.conf Some Final Preparations Starting Your Carbon Daemons Configuring Graphite-Web local_settings.py Setting Up Apache Verifying Your Graphite Installation Carbon Statistics Feeding New Data to Carbon Building Your First Graph Loading the web interface Adding metrics to a graph 5. The Graphite User Interface Finding Metrics Navigating the Tree Metrics MyGraphs and user graphs Using the Search Feature Working Smarter with the Auto-Completer Wildcards The Graphite Composer Window The Embedded Chart The Toolbar Selecting Recent Data Refreshing the Graph Selecting a Date Range Exporting a Short URL Loading a Graph from URL Saving to My Graphs Deleting from My Graphs The Graph Options Menu Adding a Graph Title Overriding the Graph Legend Toggling Axes and the Grid Applying a Graph Template Line Chart Modes Slope line Staircase line Connected line Connected line limit Draw null as zero Area and Stacked Graphs First only Stacked All Filled area alpha value Tweaking the Y-Axis A not-so-quick example Y-axis labels Upper and lower boundaries The Graph Data Dialog What Are Targets Anyway? Building a Carbon Performance Graph Adding metric targets Adding functions to targets Removing functions from targets Sharing Your Work 6. The Render API Working with Functions Starting with the Basics Math and Statistical Transforms Filtering and Sorting Grouping on Wildcards Data Smoothing and Forecasting Adjusting Metric Labels Alternate Output Formats 7. Dashboards Why Do I Need a Dashboard? Graphite Dashboard Third-Party Dashboards Grafana Tasseo Dusk Do-It-Yourself Dashing Rickshaw and D3.js Populating Rickshaw with Graphite data Dealing with CORS Conclusion 8. Troubleshooting Graphite Performance First, the Basics The Troubleshooting Toolbelt Generating Metrics and Benchmarking CPU Utilization Disk Performance (I/O) iostat iotop Networking ping tcpdump netstat iftop Inspecting Metrics whisper-info.py whisper-fetch.py whisper-dump.py API text formats Configuration Settings Carbon Noisy logging Cache and Whisper performance Message queueing and forwarding Graphite-Web Logging Carbon console.log creates.log updates.log query.log listener.log clients.log aggregator.log Graphite-Web exception.log rendering.log cache.log info.log Kernel messages Failure Scenarios The Full Disk Orphaned Whisper files Corrupted Whisper files Logfile rotation CPU Saturation Rendering Problems Networking failures Mismatched schema Slow rendering or timeouts Taking It to the Next Level 9. Scaling Graphite What Makes It Hard to Scale Graphite? Peters Graphite Story The Beginning The Pains of Popularity Clearing the Next Hurdle Try, Try Again Maximizing Resources Avoiding Outages Metrics replication Sending clients to the nearest Graphite Shared Web Database Limitations of SQLite Installing PostgreSQL dependencies Creating the web database Importing a brand new schema Migrating from SQLite to PostgreSQL Scaling in Both Directions Adding horizontal capacity Under the microscope Distributing requests across the cluster Some Final Thoughts Summary A. Carbon and Graphite-Web Internal Statistics Carbon Common to All Carbon types carbon-cache carbon-relay carbon-aggregator Graphite-Web Index