Linux Shell Scripting Cookbook. Don't neglect the shell ‚Äì this book will empower you to use simple command Dąbrowa Górnicza

The shell remains one of the most powerful tools on a computer system ‚Äî yet a large number of users are unaware of how much one can accomplish with it. Using a combination of simple commands, we will see how to solve complex problems in day to day computer usage.Linux Shell Scripting Cookbook, Second Edition will take you through useful real-world recipes designed to make your daily life easy when working with the shell. The book shows the reader how to effectively use the shell to accomplish complex tasks with ease.The book discusses basics of using the shell, general commands and proceeds to show the reader how to use them to perform complex tasks with ease.Starting with the basics of the shell, we will learn simple commands with their usages allowing us to perform operations on files of different kind. The book then proceeds to explain text processing, web interaction and concludes with backups, monitoring and other sysadmin tasks.Linux Shell Scripting Cookbook, Second Edition serves as an excellent guide to solving day to day problems using the shell and few powerful commands together to create solutions. Spis treści: Linux Shell Scripting Cookbook Table of Contents Linux Shell Scripting Cookbook Credits About the Authors About the Reviewers www.PacktPub.com Support files, eBooks, discount offers and more Why Subscribe? Free Access for Packt account holders Preface What this book covers What you need for this book Who this book is for Conventions Reader feedback Customer support Downloading the example code Errata Piracy Questions 1. Shell Something Out Introduction Printing in the terminal How to do it... How it works... Theres more... Escaping newline in echo Printing a colored output Playing with variables and environment variables Getting ready How to do it... There's more... Finding the length of a string Identifying the current shell Checking for super user Modifying the Bash prompt string (username@hostname:~$) Function to prepend to environment variables How to do it... How it works... Math with the shell Getting ready How to do it... Playing with file descriptors and redirection Getting ready How to do it... How it works... There's more... Redirection from a file to a command Redirecting from a text block enclosed within a script Custom file descriptors Arrays and associative arrays Getting ready How to do it... There's more... Defining associative arrays Listing of array indexes Visiting aliases How to do it... There's more... Escaping aliases Grabbing information about the terminal Getting ready How to do it... Getting and setting dates and delays Getting ready How to do it... How it works... There's more... Producing delays in a script Debugging the script How to do it... How it works... There's more... Shebang hack Functions and arguments How to do it... There's more... The recursive function Exporting functions Reading the return value (status) of a command Passing arguments to commands Reading the output of a sequence of commands in a variable Getting ready How to do it... There's more... Spawning a separate process with subshell Subshell quoting to preserve spacing and the newline character Reading n characters without pressing the return key How to do it... Running a command until it succeeds How to do it... How it works... There's more... A faster approach Adding a delay Field separators and iterators Getting ready How to do it... Comparisons and tests How to do it... 2. Have a Good Command Introduction Concatenating with cat How to do it... How it works There's more... Getting rid of extra blank lines Displaying tabs as ^I Line numbers Recording and playing back of terminal sessions Getting ready How to do it... How it works... Finding files and file listing Getting ready How to do it... There's more... Search based on filename or regular expression match Negating arguments Search based on the directory depth Search based on file type Search on file times Search based on file size Deleting based on the file matches Match based on the file permissions and ownership Executing commands or actions with find Skipping specified directories when using the find command Playing with xargs Getting ready How to do it... How it works There's more... Passing formatted arguments to a command by reading stdin Using xargs with find Counting the number of lines of C code in a source code directory While and subshell trick with stdin Translating with tr Getting ready How to do it... How it works There's more... Deleting characters using tr Complementing character set Squeezing characters with tr Character classes Checksum and verification Getting ready How to do it... How it works... There's more... Checksum for directories Cryptographic tools and hashes How to do it... Sorting unique and duplicates Getting ready How to do it... How it works There's more... Sorting according to the keys or columns uniq Temporary file naming and random numbers How to do it... How it works Splitting files and data How to do it... There's more Specifying a filename prefix for the split files Slicing filenames based on extension How to do it How it works Renaming and moving files in bulk Getting ready How to do it... How it works Spell checking and dictionary manipulation How to do it... How it works... Automating interactive input Getting ready How to do it... How it works There's more... Automating with expect Making commands quicker by running parallel processes How to do it... How it works... 3. File In, File Out Introduction Generating files of any size How to do it... The intersection and set difference (A-B) on text files Getting ready How to do it... How it works... Finding and deleting duplicate files Getting ready How to do it... How it works... Working with file permissions, ownership, and the sticky bit How to do it... There's more... Changing ownership Setting sticky bit Applying permissions recursively to files Applying ownership recursively Running an executable as a different user (setuid) Making files immutable Getting ready How to do it... Generating blank files in bulk Getting ready How to do it... Finding symbolic links and their targets How to do it... How it works... Enumerating file type statistics Getting ready How to do it... How it works... Using loopback files How to do it... How it works... There's more... Creating partitions inside loopback images Quicker way to mount loopback disk images with partitions Mounting ISO files as loopback Flush changing immediately with sync Creating ISO files and hybrid ISO Getting ready How to do it... There's more... Hybrid ISO that boots off a flash drive or hard disk Burning an ISO from the command line Playing with the CD-ROM tray Finding the difference between files, patching How to do it... There's more... Generating difference against directories Using head and tail for printing the last or first 10 lines How to do it... Listing only directories alternative methods Getting ready How to do it... How it works... Fast command-line navigation using pushd and popd Getting ready How to do it... There's more... Most frequently used directory switching Counting the number of lines, words, and characters in a file How to do it... Printing the directory tree Getting ready How to do it... There's more... HTML output for tree 4. Texting and Driving Introduction Using regular expressions How to do it... How it works... There's more... Treatment of special characters Visualizing regular expressions Searching and mining a text inside a file with grep How to do it... There's more... Recursively search many files Ignoring case of pattern grep by matching multiple patterns Including and excluding files in a grep search Using grep with xargs with zero-byte suffix Silent output for grep Printing lines before and after text matches Cutting a file column-wise with cut How to do it... There's more Specifying the range of characters or bytes as fields Using sed to perform text replacement How to do it There's more... Removing blank lines Performing replacement directly in the file Matched string notation (&) Substring match notation (\1) Combination of multiple expressions Quoting Using awk for advanced text processing Getting ready... How to do it How it works There's more Special variables Passing an external variable to awk Reading a line explicitly using getline Filtering lines processed by awk with filter patterns Setting delimiter for fields Reading the command output from awk Using loop inside awk String manipulation functions in awk Finding the frequency of words used in a given file Getting ready How to do it... How it works... See also Compressing or decompressing JavaScript Getting ready How to do it... How it works... See also Merging multiple files as columns How to do it... See also Printing the nth word or column in a file or line How to do it... See also Printing text between line numbers or patterns Getting ready How to do it... See also Printing lines in the reverse order Getting ready How to do it... How it works... Parsing e-mail addresses and URLs from text How to do it... How it works... See also Removing a sentence in a file containing a word Getting ready How to do it... How it works... See also Replacing a pattern with text in all the files in a directory How to do it... How it works... There's more... Text slicing and parameter operations How to do it... See also 5. Tangled Web? Not At All! Introduction Downloading from a web page Getting ready How to do it... How it works... There's more... Restricting the download speed Resume downloading and continue Copying a complete website (mirroring) Accessing pages with HTTP or FTP authentication Downloading a web page as plain text How to do it... A primer on cURL Getting ready How to do it How it works... There's more... Continuing and resuming downloads Setting the referer string with cURL Cookies with cURL Setting a user agent string with cURL Specifying a bandwidth limit on cURL Specifying the maximum download size Authenticating with cURL Printing response headers excluding the data See also Accessing Gmail e-mails from the command line How to do it... How it works... See also Parsing data from a website How to do it... How it works... See also Image crawler and downloader How to do it... How it works... See also Web photo album generator Getting ready How to do it... How it works... See also Twitter command-line client Getting ready How to do it... How it works... See also Creating a "define" utility by using the Web backend Getting ready How to do it... How it works... See also Finding broken links in a website Getting ready How to do it... How it works... See also Tracking changes to a website Getting ready How to do it... How it works... See also Posting to a web page and reading the response Getting ready How to do it... How it works... See also 6. The Backup Plan Introduction Archiving with tar Getting ready How to do it... How it works... There's more... Appending files to an archive Extracting files and folders from an archive stdin and stdout with tar Concatenating two archives Updating files in an archive with a timestamp check Comparing files in the archive and file system Deleting files from the archive Compression with the tar archive Excluding a set of files from archiving Excluding version control directories Printing total bytes See also Archiving with cpio How to do it... How it works... Compressing data with gzip How to do it... There's more... Gzip with tarball zcat - reading gzipped files without extracting Compression ratio Using bzip2 Using lzma See also Archiving and compressing with zip How to do it... How it works... Faster archiving with pbzip2 Getting ready How to do it... How it works... There's more... Manually specifying the number of CPUs Specifying the compression ratio Creating filesystems with compression Getting ready How to do it... There's more... Excluding files while creating a squashfs file Backup snapshots with rsync How to do it... How it works... There's more... Excluding files while archiving with rsync Deleting non-existent files while updating rsync backup Scheduling backups at intervals Version control-based backup with Git Getting ready How to do it... Creating entire disk images using fsarchiver Getting ready How to do it... How it works... 7. The Old-boy Network Introduction Setting up the network Getting ready How to do it... There's more... Printing the list of network interfaces Displaying IP addresses Spoofing the hardware address (MAC address) Name server and DNS (Domain Name Service) DNS lookup Showing routing table information See also Let us ping! How to do it... There's more Round trip time Limiting the number of packets to be sent Return status of the ping command Traceroute Listing all the machines alive on a network Getting ready How to do it... How it works... There's more... Parallel pings Using fping See also Running commands on a remote host with SSH Getting ready How to do it... There's more... SSH with compression Redirecting data into stdin of remote host shell commands Running graphical commands on a remote machine See also Transferring files through the network Getting ready How to do it... There's more... Automated FTP transfer SFTP (Secure FTP) The rsync command SCP (secure copy program) Recursive copying with SCP See also Connecting to a wireless network Getting ready How to do it... How it works... See also Password-less auto-login with SSH Getting ready How to do it... Port forwarding using SSH How to do it... There's more... Non-interactive port forward Reverse port forwarding Mounting a remote drive at a local mount point Getting ready How to do it... See also Network traffic and port analysis Getting ready How to do it... How it works... There's more... Opened port and services using netstat Creating arbitrary sockets Getting ready How to do it... There's more... Quickly copying files over the network Sharing an Internet connection Getting ready How to do it... Basic firewall using iptables How to do it... How it works... There's more... 8. Put on the Monitor's Cap Introduction Monitoring disk usage Getting ready How to do it... There's more... Displaying disk usage in KB, MB, or Blocks Displaying the grand total sum of disk usage Printing files in specified units Excluding files from the disk usage calculation Finding the 10 largest size files from a given directory Disk free information Calculating the execution time for a command How to do it... How it works... Collecting information about logged in users, boot logs, and boot failures Getting ready How to do it... Listing the top 10 CPU consuming processes in an hour Getting ready How to do it... How it works... See also Monitoring command outputs with watch How to do it... There's more Highlighting the differences in the watch output Logging access to files and directories Getting ready How to do it... How it works... Logfile management with logrotate Getting ready How to do it... How it works... Logging with syslog Getting ready How to do it... See also Monitoring user logins to find intruders Getting ready How to do it How it works Remote disk usage health monitor Getting ready How to do it How it works See also Finding out active user hours on a system Getting ready How to do it How it works Measuring and optimizing power usage Getting ready How to do it... Monitoring disk activity Getting ready How to do it... Checking disks and filesystems for errors Getting ready How to do it... How it works... 9. Administration Calls Introduction Gathering information about processes Getting ready How to do it... How it works... There's more... top Sorting the ps output with respect to a parameter Finding the process ID when given command names Filters with ps for real user or ID, effective user or ID TTY filter for ps Information about process threads Specifying output width and columns to be displayed Showing environment variables for a process About which, whereis, file, whatis, and load average See also Killing processes and send or respond to signals Getting ready How to do it... There's more... The kill family of commands Capturing and responding to signals Sending messages to user terminals Getting ready How to do it... How it works... Gathering system information How to do it... Using /proc for gathering information How to do it... Scheduling with cron Getting ready How to do it How it works... There's more Specifying environment variables Running commands at system start up/boot Viewing the cron table Removing the cron table Writing and reading the MySQL database from Bash Getting ready How to do it How it works User administration script How to do it How it works Bulk image resizing and format conversion Getting ready How to do it.. How it works See also Taking screenshots from the terminal Getting ready How to do it... Managing multiple terminals from one Getting ready How to do it... Index