Basics of using bash, and shell tools for covering several of the most common tasks

Introduction

‍

Most shells have their own scripting language with variables, control flow and its own syntax.

‍

Shell scripting optimized for performing shell-related tasks.

‍

Creating command pipelines
Saving results into files
Reading from standard input are primitives in shell scripting

‍

Catalog

‍

Basics of using bash, and shell tools for covering several of the most common tasks

‍

Assign variables in bash

‍

Syntax:

‍

Assign variables: variableName=value
Access variables' value: $variableName

‍

Note: variableName = value will not work if it is interpreted as calling the variableName program with arguments = and value

Added: the space character will perform argument splitting in shell.

‍

Differences between `'` and `"`

‍

Strings in bash can be defined with ' and " delimiters, they are not equivalent.

‍

' : Only the literal strings
" : Will substitute variable values that strings contain

‍

Use functions in bash

‍

mcd () {

    mkdir -p "$1"

    cd "$1"

}

‍

$0 - Name of the script
$1 to $9 - Arguments to the script. $1 is the first argument and so on.
$@ - All the arguments
$# - Number of arguments
$? - Return code of the previous command
$$ - Process identification number (PID) for the current script
!! - Entire last command, including arguments. A common pattern is to execute a command only for it to fail due to missing permissions; you can quickly re-execute the command with sudo by doing sudo !!
$_ - Last argument from the last command. If you are in an interactive shell, you can also quickly get this value by typing Esc followed by . or Alt+.

‍

Write the scripts

‍

We can type it directly in our shell, but we always prefer to write those commands into a file and execute it.

After writing it into a script(e.g. mcd.sh), we could use source to execute and load it.

Next, give the arguments to the script (e.g. mcd test), and it will work.

‍

bash manner

‍

if

‍

if [ condition ]

then

  commands

elif [ condition ]

then

  commands

else

  commands

fi

‍

case

‍

case variable in

  pattern1)

    commands

    ;;

  pattern2)

    commands

    ;;

  *)

    commands

    ;;

esac

‍

select

‍

select variable in list

do

  commands

  break

done

‍

for loop

‍

for variable in list

do

  commands

done

‍

while loop

‍

while condition

do

  commands

done

‍

until loop

‍

until condition

do

  commands

done

‍

Stream

‍

output: STDOUT >
errors : STDERR 2>

‍

Return code or exit status

‍

value of 0 -> OK;
anything different from 0 -> error occurred

‍

Operator

‍

and operator: &&
or operator: ||
seperate commands: ;
true: 0 return code
false: 1 return code
-ne : not equal

‍

Get the output of a command as a variable

‍

Substitutions

‍

command substitution: $(CMD)

process substitution: <(CMD)

Execute CMD and place the output in a temporary file and substitute the <() with that file’s name.

‍

/dev/null : You can write sth. to it, but all the written things will be discarded.

‍

No need to give arguments one by one

‍

Sometimes, you don't need to give arguments one by one, with the usage of globbing(e.g. *, ?, {}) .

In general, you can just use it by way of using the small regex.

Also, you can use another language(e.g. Python) to interact with the Shell.

‍

The usage of shebang

‍

We can use the shebang (e.g. #!/usr/local/bin/python) to let the Shell know how to run this program.

But, python may not be installed at this path, and you may can't know where they actually live in your machine, we can use env to use the PATH environment variable to resolve such a kind of question.(e.g. #!/usr/bin/env python)

‍

As writing bash scripts can be tricky and unintuitive, we can use tools like shellcheck to help us to find errors in our sh/bash scripts.

‍

Differences between shell functions and scripts

‍

Language
- Shell functions: The same language as the shell
- Scripts: Any language(So it's important to include a shebang for scripts)
Load
- Shell functions: Loaded once (Slightly fast to load, but whenever you change them you will have to reload their definition)
- Scripts: Loaded whenever they are executed
execute
- Shell functions: In the current shell environment(Can modify environment variables)
- Scripts: In their own process(Can't modify environment variables)
  
  Scripts will be passed by value environment variables that have been exported using export
As with any programming language, functions are a powerful construct to achieve modularity, code reuse, and clarity of shell code
Often shell scripts will include their own function definitions.

‍

Shell Tools

‍

Finding how to use commands

‍

man provides a manual page (called manpage) for a command you specify.
-h or --help flags
:help or ?(For interactive tools such as the ones based on ncurses)
tldr focuses on giving example use cases of a command so you can quickly figure out which options to use.

‍

Finding files

‍

find
- -name
- -path pattern
- -type
  - d : directory
  - f : file
- -mtime
- -exec
fd

A simple, fast, and user-friendly alternative to find.
locate

locate uses a database that is updated using updatedb
- In most systems, updatedb is updated daily via cron.
- A more in-depth comparison between find and locate can be found here.

‍

Finding code(the content of the file)

‍

grep

grep is an incredibly valuable shell tool that we will cover in greater detail during the data wrangling lecture.
- -R: go through the entire directory
  - Many grep alternatives have been developed, including ack, ag, and rg.
  - riggrep : more colorful
    - -u: not ignore hidden files
    - -t: type
    - -C: show lines around the matched
    - --files-without-match
    - --stats

‍

Finding shell commands(already used)

‍

up
history
- history | grep find
fzf bindings.

‍

Directory Navigation

‍

tree
broot
nnn

‍

Shortcut key

‍

Ctl-A

Moves cursor to beginning of line of text (on the command-line).
Ctl-B

Backspace (nondestructive).
Ctl-C

Break. Terminate a foreground job.
Ctl-D

Log out from a shell (similar to exit).

EOF (end-of-file). This also terminates input from stdin.

When typing text on the console or in anxterm window, Ctl-D erases the character under the cursor. When there are no characters present, Ctl-D logs out of the session, as expected. In an xterm window, this has the effect of closing the window.
Ctl-E

Moves cursor to end of line of text (on the command-line).
Ctl-F

Moves cursor forward one character position (on the command-line).
Ctl-G

BEL. On some old-time teletype terminals, this would actually ring a bell. In an xterm it might beep.
Ctl-H

Rubout (destructive backspace). Erases characters the cursor backs over while backspacing.
Ctl-I

Horizontal tab.
Ctl-J

Newline (line feed). In a script, may also be expressed in octal notation -- '\012' or in hexadecimal -- '\x0a'.
Ctl-K

Vertical tab.

When typing text on the console or in anxterm window, Ctl-K erases from the character under the cursor to end of line. Within a script, Ctl-K may behave differently, as in Lee Lee Maschmeyer's example, below.
Ctl-L

Formfeed (clear the terminal screen). In a terminal, this has the same effect as the clear command. When sent to a printer, a Ctl-L causes an advance to end of the paper sheet.
Ctl-M

Carriage return.
Ctl-N

Erases a line of text recalled from history buffer [8] (on the command-line).
Ctl-O

Issues a newline (on the command-line).
Ctl-P

Recalls last command from history buffer (on the command-line).
Ctl-Q

Resume (XON).

This resumes stdin in a terminal.
Ctl-R

Backwards search for text in history buffer (on the command-line).
Ctl-S

Suspend (XOFF).

This freezes stdin in a terminal. (Use Ctl-Q to restore input.)
Ctl-T

Reverses the position of the character the cursor is on with the previous character (on the command-line).
Ctl-U

Erase a line of input, from the cursor backward to beginning of line. In some settings, Ctl-U erases the entire line of input, regardless of cursor position.
Ctl-V

When inputting text, Ctl-V permits inserting control characters.
Ctl-W

When typing text on the console or in an xterm window, Ctl-W erases from the character under the cursor backwards to the first instance of whitespace. In some settings, Ctl-W erases backwards to first non-alphanumeric character.
Ctl-X

In certain word processing programs, Cuts highlighted text and copies to clipboard.
Ctl-Y

Pastes back text previously erased (with Ctl-U or Ctl-W).
Ctl-Z

Pauses a foreground job.

Substitute operation in certain word processing applications.

EOF (end-of-file) character in the MSDOS filesystem.

‍

Exercises

‍

ls -laht --color
1. ls -a
2. ls -h
3. ls -lh
4. ls -lt
5. ls --color

bash functions marco and polo

marco.sh

#!/bin/bash

marco(){

        path_dir=$(pwd)

        export path_dir

        echo "set $path_dir"

}

polo.sh

#!/bin/bash

polo(){

        cd "$path_dir"

        echo -n 'cd to ' && echo "$path_dir"

}

runUFail

fail.sh

#!/usr/bin/env bash

n=$(( RANDOM % 100 ))

if [[ n -eq 42 ]]; then

	echo "Something went wrong"

	>&2 echo "The error was using magic numbers"

	exit 1

fi

echo "Everything went according to plan"

runUFail

#!/bin/bash

runUFail(){

	./fail.sh > output.txt 2> error.txt

	code=$?

	fail_times=0

	while [ $code -ne 1 ]

	do

		fail_times=$((fail_times+1))

		./fail.sh >> output.txt 2>> error.txt

		code=$?

	done

	echo "fail times:$fail_times"

	echo "output.txt:" && cat output.txt

	echo "error.txt:" && cat error.txt

}

xargs command which will execute a command using STDIN as arguments.

touch ex{0,1,2,3,4,5,asb,'a b','bb'}.html

find -path "*html" -type f -print0 | ls | xargs -I {} zip output.zip {}
find . -type f -mtime -1 | ls -lt

‍

References

‍

Advanced Bash-Scripting Guide: https://tldp.org/LDP/abs/html/special-chars.html
The Missing Semester of Your CS Education: https://missing.csail.mit.edu/2020/shell-tools/

‍