Handy shell scripts

shell.md

Change all files and folders permissions of a directory to 644/755

chmod -R u+rwX,go+rX,go-w /foo

How to count all the lines of code in a directory recursively?

find . -name '*.php' | xargs wc -l

Rsync to AWS EC2 Using .PEM key

rsync -rave "ssh -i PEMKEYFILE.pem" /path/to/local/files/* ec2-user@EC2_INSTANCE_HOSTNAME:/path/to/remote/files

rsync through ssh tunnel

rsync -av -e "ssh -A root@proxy ssh" ./src root@target:/dst

fast rsync over ssh

rsync -avpP -e "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" ./src root@target:/dst

Bash string difference

diff  <(echo "$string1" ) <(echo "$string2")

Replace whole line containing a string using Sed

sed -i '/TEXT_TO_BE_REPLACED/c\This line is removed by the admin.' /tmp/foo

Generating a new SSH key

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

Select random lines from a file in bash

shuf -n N input > output

Stream a file

tail -f file

Stream multiple files

tail -f file1 -f file2

SSH and execute in background

nohup myprogram > foo.out 2> foo.err < /dev/null &

Save PID of nohup

nohup ./myprogram.sh > /dev/null 2>&1 & echo $! > run.pid

Reboot to Windows from Ubuntu under dual boot

function my_reboot_to_windows {
    WINDOWS_TITLE=`grep -i "^menuentry 'Windows" /boot/grub/grub.cfg|head -n 1|cut -d"'" -f2`
    sudo grub-reboot "$WINDOWS_TITLE"
    sudo reboot
}

Get full path of file

readlink -f relative/path/to/file

Vim write with sudo

:w !sudo tee %

Count open files per process

ls /proc/$pid/fd/ | wc -l

See limit of process

cat /proc/<PID>/limits

Finding which process was killed by Linux OOM killer

egrep -i -r 'killed process' /var/log/

scp remote to remote through local

scp -3 host1:a host2:

How can I sort du -h output by size

du -hs * | sort -h

Find large docker logs

docker ps -qa | xargs docker inspect --format='{{.LogPath}}' | xargs ls -hl
sudo find /var/lib/docker/ -name "*.log" -exec ls -sh {} \; | sort -h -r | head -20

Delete many files quickly

mkdir empty
rsync -a --delete empty/ dir/

grep-like tool that colors/boldens but not filter

egrep 'pattern|^' file

Keep process running

apt install -y run-one
keep-one-running my_process

Edit large files / Concat files

cat file2 file3 >> file1

strace summary of system calls

strace -cfp $PID

Simulate network latency

# Add latency
tc qdisc add dev eth0 root netem delay 10ms
# Remove latency
tc qdisc del dev eth0 root

Piping command but not use exit code of last

$ false | tee /dev/null ; echo $?
0
$ set -o pipefail
$ false | tee /dev/null ; echo $?
1
# restore
$ set +o pipefail

tcpdump a UNIX socket

sudo mv /path/to/sock /path/to/sock.original
sudo socat -t100 -x -v UNIX-LISTEN:/path/to/sock,mode=777,reuseaddr,fork UNIX-CONNECT:/path/to/sock.original

Get lines unique to file A

comm -23 <(sort a.txt) <(sort b.txt)

Intersection of lines in 2 files

join <(sort file1) <(sort file2)

Get last field in cut

echo 'maps.google.com' | rev | cut -d'.' -f 1 | rev

Get timing of each line of output

sudo apt install moreutils
./some_command | ts

Use netcat for p2p file transfer over a network

# receiver:
nc -l -p 9999 > foo
# sender:
nc -q 0 192.168.0.123 9999 < foo

Git

Git replace branch from another

git checkout seotweaks
git merge -s ours master
git checkout master
git merge seotweaks

How could I use git bisect to find the first GOOD commit?

git bisect start --term-new=fixed --term-old=unfixed
git bisect fixed master
git bisect unfixed $some-old-sha1

Stripping comments from yaml

yq '... comments=""' sample.yml

op