Linux

Linux in general

• Give read & execute rights to all users for a folder and its subfolders

chmod -R 755 $folder_name

• Find file in current folder and subfolders

find . -name "$file_name"

• Print lines (i.e. rows) x to y in a text file

head -y $file_name | tail -n $((y-x+1))

• Extract columns x and y from a csv file

cat $file_name.csv | cut -d ',' -f x,y

• Rename part of file name(s)

rename "s/${old_part}/${new_part}/" $file_name

• List size of folders/files in ascending order

du -sh * | sort -h

• Make sure the correct shell (e.g. bash) is used for make installation

make SHELL=/bin/bash install

MacOS Terminal (uname = Darwin)

• After any major/semi-major update, the command line tool should be updated with

xcode-select --install

• Fix broken links in dylib

otool -L $dylib_file
sudo install_name_tool $dylib_file -change $old_link $new_link

• Insert the plus/minus sign ±: shift + alt (options) + =

Git related

• Create a new branch based on the develop branch

git checkout -b $branch_name develop

• Fetch a remote branch (which does not exist locally)

git switch $branch_name

• Reset last commit

git reset --soft HEAD~1 # keeping work done
git reset --hard HEAD~1 # reverting changes

• Revert uncommitted changes

git reset
git checkout -- .

• Stop tracking previously tracked file/folder

git rm -r --cached $folder
git commit -m "Stop tracking $folder"

• Create and push a tag

git tag -a $version -m "message"
git push origin $version

• Delete a branch from both local and remote repositories

git branch -d $branch_name
git push origin --delete $branch_name

• Update a forked repository with feature branch $branch_name(assuming no conflict)

git checkout develop
git pull upstream develop
git push origin -f develop
git checkout $branch_name
git rebase develop

• Changed a cloned copy to a forked copy & make current branch track a forked branch (e.g. master branch)

git remote rename origin upstream
git remote add origin $fork_address
git branch -u origin/master

• Check changes in a file before committing

(unstaged) git diff $file_name
(staged) git diff --cached $file_named

• Check directory sizes based on annex files

git annex info --fast data/* --json | jq -j '."local annex size", "\t", .directory, "\t", "\n"'

Annex file/directory that are already commited to git

git rm --cached --ignore-unmatch -r $file_or_directory
git -c annex.verify=false annex add $file_or_directory

• Check git configurations and their origins

git config --list --show-origin

Bash

• Extract part of a string (e.g. getting 0843 from Sub0843_Ses1)

string=Sub0843_Ses1
string_cut_end=${string%_Ses1}
string_cut_front=${string_cut_end#Sub}

• Read values from a csv column into an array

arr=($(cat $file_name.csv | cut -d ',' -f $column))

• Remove Windows/DOS CRLF line endings from an array element

arr1=${arr[1]%$'\r'}

• Remove all leading spaces from zsh console output

string="    Sub0843_Ses1"
string_cut=${string##*[[:blank:]]}

• Floating point operations (e.g. x+y/365)

echo x y 365 | awk '{printf "%4.3f", $1+$2/$3}'

• Suppress 1) warnings and errors 2) all outputs from a command

1) $command 2> /dev/null
2) $command 1> /dev/null 2>&1

• The above might not work inside a script, in which case the exec command can be used before the commands generating warnings/outputs.

1) exec 2> /dev/null
2) exec 1> /dev/null 2>&1

SSH related

• Save host and user name in ~/.ssh/config

Host $new_host_name
	Hostname $host_address
	User $user_name

• Automatic login: paste content of local ~/.ssh/id_rsa.pub to remote ~/.ssh/authorized_keys

• File transfer between local machine and SSH server

(local to server) rsync -avh --progress $file_path user@host:$destination
(server to local) rsync -avh --progress user@host:$file_path $destination

Python

• Since Python 3.4, it’s preferable to use pip with

python3 -m pip list
python3 -m pip install --upgrade pip --user
python3 -m pip install numpy

• Find out which python (e.g. in Jupyter-lab)

import sys
sys.executable

• Install a package of a specific version

(new installation) python3 -m pip install -Iv package_name==version
(existing installation) python3 -m pip install -I package_name==version

• Create virtual environment (Python >= 3.4)

python3 -m venv $directory
source $directory/bin/activate

• Print elapsed time in hour-minute-second format

import time
time.strftime("%H:%M:%S", time.gmtime(elapsed_time))

• Create function handle (i.e. equivalent to myfunc = @(x) func(10, x) in Matlab)

myfunc = lambda x: func(10, x)

• Call function with dynamic name

(from another module) output = getattr(module_name, 'func_name')(parameters)
(from the same module) output = globals()['func_name'](parameters)

• Reload a module (to reflect changes made to the module)

from importlib import reload
reload(module_name)

• Write stdout to a log file at real time (from command line)

python -u $file_name.py > $log_name.txt

• Creating a Jupyterlab kernel based on a virtual environment

  source ~/.venv/$venv/bin/activate
  python3 -m pip install ipykernel
  python3 -m ipykernel install --user --name=$venv
  

• Install cutom Python version in home directory without root privilege (e.g., on HPC)

mkdir $HOME/python && cd $HOME/python
wget https://www.python.org/ftp/python/$version/Python-$version.tgz
tar -xvf Python-$version.tgz && cd Python-$version
./configure --prefix $HOME/python
make && make install
echo "export PATH=$HOME/python/Python-$version/:$PATH" >> $HOME/.zshrc

Matlab

• Write long values to a csv file (e.g. with a precision of 6)

dlmwrite('file_name.csv', variable_name, 'precision', 6);

Vi(m)

• turn off search (#) highlighting

:noh

Jekyll

• Install bundles according to specification in Gemfile

bundle install --path vendor/bundle

• Build a site with Jekyll locally

bundle exec jekyll serve
(in the browser) http://localhost:4000