Skip to main content

0106 | More Utilities

More Utilities​

Searching and Processing Text​

grep;sort;uniq;wc​

  • searching
    • search for "error" in multiple log files
      • grep error /var/log/*.log
    • display lines surrounding the matches
      • grep error -B 3 -A 2 /var/log/*.log
  • sorting
    • sorting a file (alphabetically)
      • sort random-words.txt
    • sorting a file (numerically decreasing order)
      • sort -nr random-numbers.txt
  • removing duplicates | making unique
    • for uniq, the lines MUST be adjacent --> sort first
      • sort random-words.txt | uniq
  • counting | wc
    • full output
      • wc wordlist.txt;
    • lines only
      • grep bob wordlist.txt | wc -l
  • example
    • grep -v e random-words.txt | sort | uniq | wc -l

Manipulating Text​

seq; awk;tr​

  • sed | stream editor for filtering and transforming text
    • manipulates text as it flows by -- modifies text line-by-line
    • substitution | s
      • substitute all occurrences -- one/line -- s
        • replace all occurrences of "Suite" with "Ste"`
        • sed 's/Suite/Ste/' sampl.txt`
        • only the first if multiple in on line
      • only the first if multiple in on line -- s
        • echo Suite Suite | sed 's/Suite/Ste/'
      • global substituion -- with g
        • echo Suite Suite | sed 's/Suite/Ste/g'
        • all of them are replaced -- even in the same line
      • replace only the last occurrence -- with $
        • sed '$s/Suite/Ste/' sample.txt
      • conditional replacement
        • replace only if double e in line
        • /ee/ is the matching criteria
        • sed '/ee/ s/Suite/Ste/g' sample.txt
        • search for lines with double e-s, substitute them
      • example | separate lines by new line and substitute commas with new lines
        • sed 's/$/\n/g' sample.txt | sed 's/,/\n/g'
        • or with a single command and multiple expressions
        • sed -e 's/$/\n/g' -e 's/,/\n/g' sample.txt
    • removal | d
      • remove the line with the occurrence
        • sed '/Suite/d' sample.txts
  • awk | pattern scanning and text processing language
    • breaks each line of input into separate fields using specific delimiters
    • default delimiter: space
    • examples -- separating by comma and space 
      # will output bob
      echo linux bob sally | awk '{print $2}'
      # or
      echo linux bob sally | awk '{print $3, "likes", $1}'
      # sally likes linux

      # or with comma as separator -- firs, lastname
      awk -F ',' '{print $1}' sample.txt
      # change name order -- last, firstname
      awk -F ',' '{print $1}' sample.txt | awk '{print $2 ", " $1}'

      # filter by 'dakota' -- display names
      awk -F ',' '/Dakota/ {print $1}' sample.txt
      # display which lines they are
      awk -F ',' '/Dakota/ {print NR,$1}' sample.txt
  • tr | translate or delete characters
    • reads from stdin and writes to stdout
    • replace commas with tabs
      • cat sample.txt | tr ',' '\t'
    • replace lower-case with upper-case
      • cat sample.txt | tr 'a-z' 'A-Z'
      • or with SETS
      • cat sample.txt | tr '[:lower:]' '[:upper:]'

Networking at the Command Line​

ping;ifconfig;ip​

  • testing network connection
    • ping google.com
    • ping -c 3 google.com
  • configuring network connection
    • ifconfig -- ip address
    • display statistics too
      • ip -s link
    • use builtin help
      • ip address help; ip link help
    • bring down/up the internet connection
      • dev -- device
      • sudo ip link set dev enp0s3 down
      • sudo ip link set dev enp0s3 up
  • routing tables
    • check routing table settings
      • ip route; route
    • add/delete a route to the table
      • sudo ip route add 10.0.3.0/24 via 10.0.2.1
      • sudo ip route delete 10.0.3.0/24 via 10.0.2.1
  • dns lookup | nslookup;dig
    • nslookup google.com
    • dig google.com
    • reverse lookup
      • dig -x 8.8.8.8
  • network statistics | netstat
    • check open tcp connections
      • netstat -at
    • check listening tcp ports
      • netstat -lt -- only default ports in general
      • start python server to verify
        • python3 -m http.server

File Transfer Utilities​

scp;rsync​

  • OpenSSH secure file copy | scp
    • copy local file/folder to remote computer
      • scp file.txt 192.168.100.4:/home/bob/
      • scp -r files 192.168.100.4:/home/bob/
    • copy remote file/folder from remote to local machine
      • scp 192.168.100.4:/home/bob/remote-file.txt backup/
      • scp -r 192.168.100.4:/home/bob/remote-files backup/
    • copying with a different user name
  • a fast, versatile, remote (and local) file-copying tool | rsync
    • computes the difference between the source and destination files, and only transfers the difference
    • copy file from local to remote
      • a:archive mode
        • recursively copying directories and preserving user permissions and ownership
      • v:verbose; z:compressing file data; h:human readable
      • rsync -avzh file2.txt 192.168.100.4:/home/bob

Converting Text Files​

  • different line terminator for win/macos and linux -- end of the line symbol
    • windows/dos
      • ctrl and line feed -- "/r/n" -- CRLF
    • macos
      • ctrl -- "/r" -- "CR"
    • linux/unix
      • line feed -- "/n"
  • check file formats
    • file *.txt
  • change line terminator | vim
    • win/dos example 
      vim sample-dos-file.txt
      :e ++ff=unix
      # each line (except the last one) will be 
      # terminated with ^M = control char
      # the new line is still displayed since LF is 
      # used = same as linux
    • macos example 
      vim sample-macos-file.txt
      # only the ^M -s are displayed -- no new lines
  • convert file types | dos2unix;unix2dos
    • unix->dos
      • modify original file
        • unix2dos temp.txt
    • unix->dos
      • create a new file
        • unix2dos -n sample-unix-file.txt temp.txt
    • unix->macos
      • unix2dos -c mac temp.txt
    • dos->unix
      • dos2unix temp.txt
    • macos->unix
      • dos2unix -c mac temp.txt