Learning OpenCV 3 - Adrian Kaehler Katowice

Get started in the rapidly expanding field of computer vision with this practical guide. Written by Adrian Kaehler and Gary Bradski, creator of the open source OpenCV library, this book provides a thorough introduction for developers, academics, roboticists, and hobbyists. Youll learn what it takes to build applications that enable computers to "see" and make decisions based on that data. With over 500 functions that span many areas in vision, OpenCV is used for commercial applications such as security, medical imaging, pattern and face recognition, robotics, and factory product inspection. This book gives you a firm grounding in computer vision and OpenCV for building simple or sophisticated vision applications. Hands-on exercises in each chapter help you apply what youve learned. This volume covers the entire library, in its modern C++ implementation, including machine learning tools for computer vision. Learn OpenCV data types, array types, and array operations Capture and store still and video images with HighGUI Transform images to stretch, shrink, warp, remap, and repair Explore pattern recognition, including face detection Track objects and motion through the visual field Reconstruct 3D images from stereo vision Discover basic and advanced machine learning techniques in OpenCV Spis treści: Preface Purpose of This Book Who This Book Is For What This Book Is Not About the Programs in This Book Prerequisites How This Book Is Best Used Conventions Used in This Book Using Code Examples OReilly Safari Wed Like to Hear from You Acknowledgments Thanks for Help on OpenCV Thanks for Help on This Book Adrian Adds... Gary Adds... 1. Overview What Is OpenCV? Who Uses OpenCV? What Is Computer Vision? The Origin of OpenCV OpenCV Block Diagram Speeding Up OpenCV with IPP Who Owns OpenCV? Downloading and Installing OpenCV Installation Windows Linux Mac OS X Getting the Latest OpenCV via Git More OpenCV Documentation Supplied Documentation Online Documentation and the Wiki OpenCV Contribution Repository Downloading and Building Contributed Modules Portability Summary Exercises 2. Introduction to OpenCV Include Files Resources First ProgramDisplay a Picture Second ProgramVideo Moving Around A Simple Transformation A Not-So-Simple Transformation Input from a Camera Writing to an AVI File Summary Exercises 3. Getting to Know OpenCV Data Types The Basics OpenCV Data Types Overview of the Basic Types Basic Types: Getting Down to Details The point classes The cv::Scalar class The size classes The cv::Rect class The cv::RotatedRect class The fixed matrix classes The fixed vector classes The complex number classes Helper Objects The cv::TermCriteria class The cv::Range class The cv::Ptr template and Garbage Collection 101 The cv::Exception class and exception handling The cv::DataType<> template The cv::InputArray and cv::OutputArray classes Utility Functions cv::alignPtr() cv::alignSize() cv::allocate() cv::deallocate() cv::fastAtan2() cvCeil() cv::cubeRoot() cv::CV_Assert() and CV_DbgAssert() cv::CV_Error() and CV_Error_() cv::error() cv::fastFree() cv::fastMalloc() cvFloor() cv::format() cv::getCPUTickCount() cv::getNumThreads() cv::getOptimalDFTSize() cv::getThreadNum() cv::getTickCount() cv::getTickFrequency() cvIsInf() cvIsNaN() cvRound() cv::setNumThreads() cv::setUseOptimized() cv::useOptimized() The Template Structures Summary Exercises 4. Images and Large Array Types Dynamic and Variable Storage The cv::Mat Class: N-Dimensional Dense Arrays Creating an Array Accessing Array Elements Individually The N-ary Array Iterator: NAryMatIterator Accessing Array Elements by Block Matrix Expressions: Algebra and cv::Mat Saturation Casting More Things an Array Can Do The cv::SparseMat Class: Sparse Arrays Accessing Sparse Array Elements Functions Unique to Sparse Arrays Template Structures for Large Array Types Summary Exercises 5. Array Operations More Things You Can Do with Arrays cv::abs() cv::absdiff() cv::add() cv::addWeighted() cv::bitwise_and() cv::bitwise_not() cv::bitwise_or() cv::bitwise_xor() cv::calcCovarMatrix() cv::cartToPolar() cv::checkRange() cv::compare() cv::completeSymm() cv::convertScaleAbs() cv::countNonZero() cv::cvarrToMat() cv::dct() cv::dft() cv::cvtColor() cv::determinant() cv::divide() cv::eigen() cv::exp() cv::extractImageCOI() cv::flip() cv::gemm() cv::getConvertElem() and cv::getConvertScaleElem() cv::idct() cv::idft() cv::inRange() cv::insertImageCOI() cv::invert() cv::log() cv::LUT() cv::magnitude() cv::Mahalanobis() cv::max() cv::mean() cv::meanStdDev() cv::merge() cv::min() cv::minMaxIdx() cv::minMaxLoc() cv::mixChannels() cv::mulSpectrums() cv::multiply() cv::mulTransposed() cv::norm() cv::normalize() cv::perspectiveTransform() cv::phase() cv::polarToCart() cv::pow() cv::randu() cv::randn() cv::randShuffle() cv::reduce() cv::repeat() cv::scaleAdd() cv::setIdentity() cv::solve() cv::solveCubic() cv::solvePoly() cv::sort() cv::sortIdx() cv::split() cv::sqrt() cv::subtract() cv::sum() cv::trace() cv::transform() cv::transpose() Summary Exercises 6. Drawing and Annotating Drawing Things Line Art and Filled Polygons cv::circle() cv::clipLine() cv::ellipse() cv::ellipse2Poly() cv::fillConvexPoly() cv::fillPoly() cv::line() cv::rectangle() cv::polyLines() cv::LineIterator Fonts and Text cv::putText() cv::getTextSize() Summary Exercises 7. Functors in OpenCV Objects That Do Stuff Principal Component Analysis (cv::PCA) cv::PCA::PCA() cv::PCA::operator()() cv::PCA::project() cv::PCA::backProject() Singular Value Decomposition (cv::SVD) cv::SVD() cv::SVD::operator()() cv::SVD::compute() cv::SVD::solveZ() cv::SVD::backSubst() Random Number Generator (cv::RNG) cv::theRNG() cv::RNG() cv::RNG::operator T(), where T is your favorite type cv::RNG::operator() cv::RNG::uniform() cv::RNG::gaussian() cv::RNG::fill() Summary Exercises 8. Image, Video, and Data Files HighGUI: Portable Graphics Toolkit Working with Image Files Loading and Saving Images Reading files with cv::imread() Writing files with cv::imwrite() A Note About Codecs Compression and Decompression Compressing files with cv::imencode() Uncompressing files with cv::imdecode() Working with Video Reading Video with the cv::VideoCapture Object Reading frames with cv::VideoCapture::read() Reading frames with cv::VideoCapture::operator>>() Reading frames with cv::VideoCapture::grab() and cv::VideoCapture::retrieve() Camera properties: cv::VideoCapture::get() and cv::VideoCapture::set() Writing Video with the cv::VideoWriter Object Writing frames with cv::VideoWriter::write() Writing frames with cv::VideoWriter::operator<<() Data Persistence Writing to a cv::FileStorage Reading from a cv::FileStorage cv::FileNode Summary Exercises 9. Cross-Platform and Native Windows Working with Windows HighGUI Native Graphical User Interface Creating a window with cv::namedWindow() Drawing an image with cv::imshow() Updating a window and cv::waitKey() An example displaying an image Mouse events Sliders, trackbars, and switches Surviving without buttons Working with the Qt Backend Getting started The actions menu The text overlay Writing your own text into the status bar The properties window Trackbars revisited Creating buttons with cv::createButton() Text and fonts Setting and getting window properties Saving and recovering window state Interacting with OpenGL Integrating OpenCV with Full GUI Toolkits An example of OpenCV and Qt An example of OpenCV and wxWidgets An example of OpenCV and the Windows Template Library Summary Exercises 10. Filters and Convolution Overview Before We Begin Filters, Kernels, and Convolution Anchor points Border Extrapolation and Boundary Conditions Making borders yourself Manual extrapolation Threshold Operations Otsus Algorithm Adaptive Threshold Smoothing Simple Blur and the Box Filter Median Filter Gaussian Filter Bilateral Filter Derivatives and Gradients The Sobel Derivative Scharr Filter The Laplacian Image Morphology Dilation and Erosion The General Morphology Function Opening and Closing Morphological Gradient Top Hat and Black Hat Making Your Own Kernel Convolution with an Arbitrary Linear Filter Applying a General Filter with cv::filter2D() Applying a General Separable Filter with cv::sepFilter2D Kernel Builders cv::getDerivKernel() cv::getGaussianKernel() Summary Exercises 11. General Image Transforms Overview Stretch, Shrink, Warp, and Rotate Uniform Resize cv::resize() Image Pyramids cv::pyrDown() cv::buildPyramid() cv::pyrUp() The Laplacian pyramid Nonuniform Mappings Affine Transformation cv::warpAffine(): Dense affine transformations cv::getAffineTransform(): Computing an affine map matrix cv::transform(): Sparse affine transformations cv::invertAffineTransform(): Inverting an affine transformation Perspective Transformation cv::warpPerspective(): Dense perspective transform cv::getPerspectiveTransform(): Computing the perspective map matrix cv::perspectiveTransform(): Sparse perspective transformations General Remappings Polar Mappings cv::cartToPolar(): Converting from Cartesian to polar coordinates cv::polarToCart(): Converting from polar to Cartesian coordinates LogPolar cv::logPolar() Arbitrary Mappings cv::remap(): General image remapping Image Repair Inpainting Denoising Basic FNLMD with cv::fastNlMeansDenoising() FNLMD on color images with cv::fastNlMeansDenoisingColor() FNLMD on video with cv::fastNlMeansDenoisingMulti() and cv::fastNlMeansDenoisingColorMulti() Histogram Equalization cv::equalizeHist(): Contrast equalization Summary Exercises 12. Image Analysis Overview Discrete Fourier Transform cv::dft(): The Discrete Fourier Transform cv::idft(): The Inverse Discrete Fourier Transform cv::mulSpectrums(): Spectrum Multiplication Convolution Using Discrete Fourier Transforms cv::dct(): The Discrete Cosine Transform cv::idct(): The Inverse Discrete Cosine Transform Integral Images cv::integral() for Standard Summation Integral cv::integral() for Squared Summation Integral cv::integral() for Tilted Summation Integral The Canny Edge Detector cv::Canny() Hough Transforms Hough Line Transform cv::HoughLines(): The standard and multiscale Hough transforms cv::HoughLinesP(): The progressive probabilistic Hough transform Hough Circle Transform cv::HoughCircles(): the Hough circle transform Distance Transformation cv::distanceTransform() for Unlabeled Distance Transform cv::distanceTransform() for Labeled Distance Transform Segmentation Flood Fill Watershed Algorithm Grabcuts Mean-Shift Segmentation Summary Exercises 13. Histograms and Templates Histogram Representation in OpenCV cv::calcHist(): Creating a Histogram from Data Basic Manipulations with Histograms Histogram Normalization Histogram Threshold Finding the Most Populated Bin Comparing Two Histograms Correlation method (cv::COMP_CORREL) Chi-square method (cv::COMP_CHISQR_ALT) Intersection method (cv::COMP_INTERSECT) Bhattacharyya distance method (cv::COMP_BHATTACHARYYA) Histogram Usage Examples Some More Sophisticated Histograms Methods Earth Movers Distance Back Projection Basic back projection: cv::calcBackProject() Template Matching Square Difference Matching Method (cv::TM_SQDIFF) Normalized Square Difference Matching Method (cv::TM_SQDIFF_NORMED) Correlation Matching Methods (cv::TM_CCORR) Normalized Cross-Correlation Matching Method (cv::TM_CCORR_NORMED) Correlation Coefficient Matching Methods (cv::TM_CCOEFF) Normalized Correlation Coefficient Matching Method (cv::TM_CCOEFF_NORMED) Summary Exercises 14. Contours Contour Finding Contour Hierarchies Finding contours with cv::findContours() Drawing Contours A Contour Example Another Contour Example Fast Connected Component Analysis More to Do with Contours Polygon Approximations Polygon approximation with cv::approxPolyDP() The Douglas-Peucker algorithm explained Geometry and Summary Characteristics Length using cv::arcLength() Upright bounding box with cv::boundingRect() A minimum area rectangle with cv::minAreaRect() A minimal enclosing circle using cv::minEnclosingCircle() Fitting an ellipse with cv::fitEllipse() Finding the best line fit to your contour with cv::fitLine() Finding the convex hull of a contour using cv::convexHull() Geometrical Tests Testing if a point is inside a polygon with cv::pointPolygonTest() Testing whether a contour is convex with cv::isContourConvex() Matching Contours and Images Moments Computing moments with cv::moments() More About Moments Central moments are invariant under translation Normalized central moments are also invariant under scaling Hu invariant moments are invariant under rotation Computing Hu invariant moments with cv::HuMoments() Matching and Hu Moments Using Shape Context to Compare Shapes Structure of the shape module The shape context distance extractor Hausdorff distance extractor Summary Exercises 15. Background Subtraction Overview of Background Subtraction Weaknesses of Background Subtraction Scene Modeling A Slice of Pixels Frame Differencing Averaging Background Method Accumulating Means, Variances, and Covariances Computing the mean with cv::Mat::operator+=() Computing the mean with cv::accumulate() Variation: Computing the mean with cv::accumulateWeighted() Finding the variance with the help of cv::accumulateSquare() Finding the covariance with cv::accumulateWeighted() A brief note on model testing and cv::Mahalanobis() A More Advanced Background Subtraction Method Structures Learning the Background Learning with Moving Foreground Objects Background Differencing: Finding Foreground Objects Using the Codebook Background Model A Few More Thoughts on Codebook Models Connected Components for Foreground Cleanup A Quick Test Comparing Two Background Methods OpenCV Background Subtraction Encapsulation The cv::BackgroundSubtractor Base Class KaewTraKuPong and Bowden Method cv::BackgroundSubtractorMOG Zivkovic Method cv::BackgroundSubtractorMOG2 Summary Exercises 16. Keypoints and Descriptors Keypoints and the Basics of Tracking Corner Finding Finding corners using cv::goodFeaturesToTrack() Subpixel corners Introduction to Optical Flow Lucas-Kanade Method for Sparse Optical Flow How Lucas-Kanade works Pyramid Lucas-Kanade code: cv::calcOpticalFlowPyrLK() A worked example Generalized Keypoints and Descriptors Optical Flow, Tracking, and Recognition How OpenCV Handles Keypoints and Descriptors, the General Case The cv::KeyPoint object The (abstract) class that finds keypoints and/or computes descriptors for them: cv::Feature2D The cv::DMatch object The (abstract) keypoint matching class: cv::DescriptorMatcher Core Keypoint Detection Methods The Harris-Shi-Tomasi feature detector and cv::GoodFeaturesToTrackDetector Keypoint finder Additional functions A brief look under the hood The simple blob detector and cv::SimpleBlobDetector Keypoint finder The FAST feature detector and cv::FastFeatureDetector Keypoint finder The SIFT feature detector and cv::xfeatures2d::SIFT Keypoint finder and feature extractor The SURF feature detector and cv::xfeatures2d::SURF Keypoint finder and feature extractor Additional functions provided by cv::xfeatures2d::SURF The Star/CenSurE feature detector and cv::xfeatures2d::StarDetector Keypoint finder The BRIEF descriptor extractor and cv::BriefDescriptorExtractor Feature extractor The BRISK algorithm Keypoint finder and feature extractor Additional functions provided by cv::BRISK The ORB feature detector and cv::ORB Keypoint finder and feature extractor Additional functions provided by cv::ORB The FREAK descriptor extractor and cv::xfeatures2d::FREAK Feature extractor Dense feature grids and the cv::DenseFeatureDetector class Keypoint finder Keypoint Filtering The cv::KeyPointsFilter class Matching Methods Brute force matching with cv::BFMatcher Fast approximate nearest neighbors and cv::FlannBasedMatcher Linear indexing with cv::flann::LinearIndexParams KD-tree indexing with cv::flann::KDTreeIndexParams Hierarchical k-means tree indexing with cv::flann::KMeansIndexParams Combining KD-trees and k-means with cv::flann::CompositeIndexParams Locality-sensitive hash (LSH) indexing with cv::flann::LshIndexParams Automatic index selection with cv::flann::AutotunedIndexParams FLANN search parameters and cv::flann::SearchParams Displaying Results Displaying keypoints with cv::drawKeypoints Displaying keypoint matches with cv::drawMatches Summary Exercises 17. Tracking Concepts in Tracking Dense Optical Flow The Farnebäck Polynomial Expansion Algorithm Computing dense optical flow with cv::calcOpticalFlowFarneback The Dual TV-L1 Algorithm Computing dense optical flow with cv::createOptFlow_DualTVL1 The Simple Flow Algorithm Computing Simple Flow with cv::optflow::calcOpticalFlowSF() Mean-Shift and Camshift Tracking Mean-Shift Camshift Motion Templates Estimators The Kalman Filter What goes in and what comes out Assumptions required by the Kalman filter Information fusion Systems with dynamics Kalman equations Tracking in OpenCV with cv::KalmanFilter Kalman filter example code A Brief Note on the Extended Kalman Filter Summary Exercises 18. Camera Models and Calibration Camera Model The Basics of Projective Geometry Rodrigues Transform Lens Distortions Calibration Rotation Matrix and Translation Vector Calibration Boards Finding chessboard corners with cv::findChessboardCorners() Subpixel corners on chessboards and cv::cornerSubPix() Drawing chessboard corners with cv::drawChessboardCorners() Circle-grids and cv::findCirclesGrid() Homography Camera Calibration How many chess corners for how many parameters? Whats under the hood? Calibration function Computing extrinsics only with cv::solvePnP() Computing extrinsics only with cv::solvePnPRansac() Undistortion Undistortion Maps Converting Undistortion Maps Between Representations with cv::convertMaps() Computing Undistortion Maps with cv::initUndistortRectifyMap() Undistorting an Image with cv::remap() Undistortion with cv::undistort() Sparse Undistortion with cv::undistortPoints() Putting Calibration All Together Summary Exercises 19. Projection and Three-Dimensional Vision Projections Affine and Perspective Transformations Birds-Eye-View Transform Example Three-Dimensional Pose Estimation Pose Estimation from a Single Camera Computing the pose of a known object with cv::solvePnP() Stereo Imaging Triangulation Epipolar Geometry The Essential and Fundamental Matrices Essential matrix math Fundamental matrix math How OpenCV handles all of this Computing Epipolar Lines Stereo Calibration Stereo Rectification Uncalibrated stereo rectification: Hartleys algorithm Calibrated stereo rectification: Bouguets algorithm Rectification map Stereo Correspondence The stereo matching classes: cv::StereoBM and cv::StereoSGBM Block matching Computing stereo depths with cv::StereoBM Semi-global block matching Computing stereo depths with cv::StereoSGBM Stereo Calibration, Rectification, and Correspondence Code Example Depth Maps from Three-Dimensional Reprojection Structure from Motion Fitting Lines in Two and Three Dimensions Summary Exercises 20. The Basics of Machine Learning in OpenCV What Is Machine Learning? Training and Test Sets Supervised and Unsupervised Learning Generative and Discriminative Models OpenCV ML Algorithms Using Machine Learning in Vision Variable Importance Diagnosing Machine Learning Problems Cross-validation, bootstrapping, ROC curves, and confusion matrices Cost of misclassification Mismatched feature variance Legacy Routines in the ML Library K-Means Problems and solutions K-means code Mahalanobis Distance Using the Mahalanobis distance to condition input data Using the Mahalanobis distance for classification Summary Exercises 21. StatModel: The Standard Model for Learning in OpenCV Common Routines in the ML Library Training and the cv::ml::TrainData Structure Constructing cv::ml::TrainData Constructing cv::ml::TrainData from stored data Secret sauce and cv::ml::TrainDataImpl Splitting training data Accessing cv::ml::TrainData Prediction Machine Learning Algorithms Using cv::StatModel Nave/Normal Bayes Classifier The nave/normal Bayes classifier and cv::ml::NormalBayesClassifier Binary Decision Trees Regression impurity Classification impurity OpenCV implementation Decision tree usage Decision tree results Boosting AdaBoost Boosting code Random Trees Random trees code Using random trees Expectation Maximization Expectation maximization with cv::EM() K-Nearest Neighbors Using K-nearest neighbors with cv::ml::KNearest() Multilayer Perceptron Back propagation The Rprop algorithm Using artificial neural networks and back propagation with cv::ml::ANN_MLP Parameters for training Support Vector Machine About kernels Handling outliers Multiclass extension of SVM One-class SVM Support vector regression Using support vector machines and cv::ml::SVM() Additional members of cv::ml::SVM Summary Exercises 22. Object Detection Tree-Based Object Detection Techniques Cascade Classifiers Haar-like features Local binary pattern features Training and pretrained detectors Supervised Learning and Boosting Theory Boosting in the Haar cascade Rejection cascades Viola-Jones classifier summary The cv::CascadeClassifer object Searching an image with detectMultiScale() Face detection example Learning New Objects Detailed arguments to createsamples Detailed arguments to traincascade Object Detection Using Support Vector Machines Latent SVM for Object Detection Object detection with cv::dpm::DPMDetector Other methods of cv::dpm::DPMDetector Where to get models for cv::dpm::DPMDetector The Bag of Words Algorithm and Semantic Categorization Training with cv::BOWTrainer K-means and cv::BOWKMeansTrainer Categorization with cv::BOWImgDescriptorExtractor Putting it together using a support vector machine Summary Exercises 23. Future of OpenCV Past and Present OpenCV 3.x How Well Did Our Predictions Go Last Time? Future Functions Current GSoC Work Community Contributions OpenCV.org Some AI Speculation Afterword A. Planar Subdivisions Delaunay Triangulation, Voronoi Tesselation Creating a Delaunay or Voronoi Subdivision Navigating Delaunay Subdivisions Points from edges Locating a point within a subdivision Orbiting around a vertex Rotating an edge Identifying the bounding triangle Identifying the bounding triangle or edges on the convex hull and walking the hull Usage Examples Exercises B. opencv_contrib An Overview of the opencv_contrib Modules Contents of opencv_contrib C. Calibration Patterns Calibration Patterns Used by OpenCV Bibliography Index