from contextlib import contextmanager import time @contextmanager def timer(label): start = time.time() try: yield finally: end = time.time() print(f'{label}: {end - start} seconds') # Usage: with timer('Model training'): # Your code block here, e.g., train your model
Gliding Phenomena
Python contextlib for Timing Python code
Linux HTOP settings/configuration
vi ~/.config/htop/htoprc
# Beware! This file is rewritten by htop when settings are changed in the interface. # The parser is also very primitive, and not human-friendly. fields=0 48 17 18 38 39 40 2 46 47 49 1 sort_key=46 sort_direction=-1 tree_sort_key=0 tree_sort_direction=1 hide_kernel_threads=1 hide_userland_threads=0 shadow_other_users=0 show_thread_names=0 show_program_path=1 highlight_base_name=0 highlight_megabytes=1 highlight_threads=1 highlight_changes=0 highlight_changes_delay_secs=5 find_comm_in_cmdline=1 strip_exe_from_cmdline=1 show_merged_command=0 tree_view=0 tree_view_always_by_pid=0 header_margin=1 detailed_cpu_time=0 cpu_count_from_one=0 show_cpu_usage=1 show_cpu_frequency=0 show_cpu_temperature=0 degree_fahrenheit=0 update_process_names=0 account_guest_in_cpu_meter=0 color_scheme=0 enable_mouse=1 delay=15 left_meters=LeftCPUs8 Memory Swap left_meter_modes=1 1 1 right_meters=RightCPUs8 Tasks LoadAverage Uptime right_meter_modes=1 2 2 2 hide_function_bar=0
Disk Utilities in Linux
Linux, renowned for its robustness, is a preferred choice among software professionals. Navigating the expansive world of disk management, Linux offers a spectrum of utilities. In this concise guide, we'll explore essential tools tailored for professionals, shedding light on three practical examples for each.
df (Disk Free):
Example 1: Display disk space usage for all mounted filesystems.
df -h
Example 2: Show the total, used, and available space for a specific directory.
df -h /path/to/directory
Example 3: Highlight human-readable disk space information in 1K blocks.
df -kh
du (Disk Usage):
Example 1: Determine the size of a directory and its subdirectories.
du -h /path/to/directory
Example 2: Display the sizes of all files in a directory.
du -h --max-depth=1
Example 3: Sort and list top disk space-consuming directories.
du -h --max-depth=1 | sort -rh
fdisk:
Example 1: List all available disk partitions.
fdisk -l
Example 2: Create a new partition on a specific disk.
fdisk /dev/sdX
Example 3: Delete a partition from a disk.
fdisk /dev/sdX -d
parted:
Example 1: Display partition information for a specific disk.
parted /dev/sdX print
Example 2: Resize a partition to a specific size.
parted /dev/sdX resizepart N 50G
Example 3: Create a new partition on a disk.
parted /dev/sdX mkpart primary ext4 0% 100%
gparted:
Example 1: Launch the graphical partition editor.
gparted
Example 2: Resize a partition using the GUI interface.
gparted /dev/sdX
Example 3: Format a partition with a specific file system.
gparted /dev/sdX - create new ext4 partition
badblocks:
Example 1: Check for bad blocks on a specific disk.
badblocks -s /dev/sdX
Example 2: Print a list of bad blocks found during testing.
badblocks -l /dev/sdX
Example 3: Non-destructive read-write test for bad blocks.
badblocks -n /dev/sdX
smartctl:
Example 1: Display SMART information for a specific disk.
smartctl -a /dev/sdX
Example 2: Run a short self-test on a disk.
smartctl -t short /dev/sdX
Example 3: View the test result and overall health status.
smartctl -l selftest /dev/sdX
rsync:
Example 1: Copy files and directories to a remote server.
rsync -av /local/path/ user@remote:/remote/path/
Example 2: Synchronize two directories, updating only changed files.
rsync -av --update /source/directory/ /destination/directory/
Example 3: Backup a directory and exclude specific files.
rsync -av --exclude='*.log' /source/directory/ /backup/directory/
lvm (Logical Volume Manager):
Example 1: Display information about logical volumes.
lvs
Example 2: Create a new logical volume.
lvcreate -L 10G -n myvolume myvg
Example 3: Extend the size of a logical volume.
lvextend -L +5G /dev/myvg/myvolume
fstrim:
Example 1: Trim all mounted filesystems.
fstrim -a
Example 2: Trim a specific mount point.
fstrim /path/to/mount/point
Example 3: Dry-run trim to check for discardable blocks.
fstrim -v --dry-run /path/to/mount/point
duf:
Example 1: Display disk usage with a colorful and user-friendly interface.
duf -c
Example 2: Show disk usage for a specific directory.
duf /path/to/directory
Example 3: Display disk usage with detailed information.
duf -l
ncdu:
Example 1: Launch the NCurses Disk Usage tool for interactive disk exploration.
ncdu
Example 2: Scan and analyze disk usage for a specific directory.
ncdu /path/to/directory
Example 3: Display disk usage in a human-readable format.
ncdu -x
fstab (File System Table):
Example 1: Edit the fstab file for automatic mounting of partitions at boot.
nano /etc/fstab
Example 2: View the current entries in the fstab file.
cat /etc/fstab
Example 3: Check for errors in the fstab file syntax.
mount -a --test
Get PIP package versions and licenses within a virtual environment
pip install pip-licenses
pip-licenses --format=csv \
--with-urls \
--with-authors \
--with-maintainers \
--with-description > packages.csv
# The following will produce main packages only without nested dependencies. pip install pipdeptree pipdeptree -f | grep -P '^\w+' | cut -d = -f 1 > main-packages.txt
Ubuntu Error constructing proxy for org.gnome.Terminal
If the terminal fails to start when using Ubuntu desktop via VNC Server. If you see the below error in ~/.vnc/<some-ip>.log.
Ubuntu org.gnome.Terminal:/org/gnome/Terminal/Factory0: Error calling StartServiceByName
Error constructing proxy for org.gnome.Terminal: /org/gnome/Terminal/Factory0:
Error calling StartServiceByName for org.gnome.Terminal:
GDBus.Error:org.freedesktop.DBus.Error.Spawn.ChildExited:
Process /usr/lib/gnome-terminal/gnome-terminal-server exited with status 8
sudo apt-get update sudo apt-get install rxvt-unicode
#!/bin/sh export XKL_XMODMAP_DISABLE=1 unset SESSION_MANAGER unset DBUS_SESSION_BUS_ADDRESS [ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup [ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources xsetroot -solid grey vncconfig -iconic & gnome-panel & gnome-settings-daemon & metacity & nautilus & gnome-terminal & urxvt &
vncserver -kill :1 vncserver :1
Install iGPU / Intel integrated GPU drivers for OpenVINO Inference
To enable Intel integrated GPU / iGPU for running deep learning inference with OpenVINO, we need to install the Intel Graphics Compute Runtime for oneAPI Level Zero and OpenCL™ Driver on Linux.
Intel Graphics Compute Runtime: https://github.com/intel/compute-runtime/releases
See : https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html
More info: https://dgpu-docs.intel.com/driver/client/overview.html
# After Installing Intel Graphics Compute Runtime
sudo apt install hwinfo clinfo -y
After installation, add video and render user groups.
sudo usermod -aG video $USER sudo usermod -aG render $USER
# Logout and login
Build OpenVINO from source - Linux
Official Instructions - Click Here
Build instructions for OpenVINO from Source with the Python API Wrapper
OpenVINO 2022.1.0 and later require GLIBC 2.27+, check with `ldd --version`
Here instructions are for Python 3.7, you can change it to 3.6 as well.
- GCC 7.5 or higher to build OpenVINO Runtime
- Python 3.6 or higher for OpenVINO Runtime Python API
- (Optional) Install Intel® Graphics Compute Runtime for OpenCL™ Driver package 19.41.14441 to enable inference on Intel integrated GPUs.
$ sudo apt-get install python3.7-dev $ pip install cython numpy $ cd ~ #openvino will be installed in ~/openvino $ git clone https://github.com/openvinotoolkit/openvino.git $ cd openvino $ git submodule update --init --recursive $ chmod +x install_dependencies.sh $ ./install_dependencies.sh $ mkdir build && cd build $ cmake -DCMAKE_BUILD_TYPE=Release \
-DENABLE_INTEL_GNA=OFF -DENABLE_INTEL_MYRIAD_COMMON=OFF \
-DENABLE_PYTHON=ON \ -DPYTHON_EXECUTABLE=`which python3.7` \ -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.7m.so \ -DPYTHON_INCLUDE_DIR=/usr/include/python3.7 ..
$ make --jobs=$(nproc --all) $ export PYTHONPATH=$PYTHONPATH:~/openvino/bin/intel64/Release/lib/python_api/python3.7/ $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/openvino/bin/intel64/Release/lib/
# OR Install the wheel with PIP
$ pip install ~/openvino/build/wheel/*.whl
# TEST BUILD
$ python3.7 >>> from openvino.inference_engine import IENetwork, IECore # Test Benchmark app $ alias benchmark_app=~/openvino/bin/intel64/Release/benchmark_app $ benchmark_app -h
Install Python3.6 or Python3.7 in Ubuntu 20.04; ImportError: libpython3.6m.so.1.0: cannot open shared object file
See below to install Python 3.6 or 3.7
sudo -E add-apt-repository ppa:deadsnakes/ppa sudo apt-get update sudo apt-get install python3.6 sudo apt-get install -y libpython3.6-dev
ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory
Debugging Python with pudb
1. Install pudb pip package..
$ pip install pudb
import pudb; pudb.set_trace()
$ python -m pudb my_script.py
TMUX Scrolling with Mouse
TMUX Scrolling with Mouse
# Add these in ~/.tmux.conf
set -g mouse on #For tmux version 2.1 and up set -g @scroll-down-exit-copy-mode "off" set -g terminal-overrides 'xterm*:smcup@:rmcup@' # Sane scrolling
# RELOAD the conf file:
# tmux source-file ~/.tmux.conf
# If you get Error: protocol version mismatch (client 8, server 7)
# sudo killall -9 tmux
# If conf error, make sure you have tmux version >2.1
# tmux -V
# Update tmux. Ex: sudo yum install tmux
Install ClamAV on Amazon Linux 2
Install ClamAV Scan on Amazon Linux 2
sudo amazon-linux-extras install epel sudo yum install clamav clamd -y sudo sed -i -e "s/Example/#Example/" /etc/freshclam.conf sudo sed -i -e "s:#DatabaseDirectory /var/lib/clamav:DatabaseDirectory /var/lib/clamav:" /etc/freshclam.conf sudo sed -i -e "s:#UpdateLogFile /var/log/freshclam.log:UpdateLogFile /var/log/freshclam.log:" /etc/freshclam.conf sudo sed -i -e "s/#DatabaseOwner clamupdate/DatabaseOwner clamupdate/" /etc/freshclam.conf sudo vi /etc/clamd.d/scan.conf # Change MaxThreads if desired. sudo freshclam clamscan -r / 2>&1 | tee openvinoami_clamav_scan.txt # If problems arise, restart sudo pkill freshclam sudo freshclam clamscan -r / 2>&1 | tee openvinoami_clamav_scan.txt
Dynamic Port Forwarding using Firefox Browser
[bob@workstation ~]$ ssh -D 1080 bastion.securecorp.io [bob@bastion ~]$
SOCKS5 configuration in Firefox:
Source: https://www.redhat.com/sysadmin/ssh-dynamic-port-forwarding
Host a static website in Ubuntu using Nginx
Install Ngnix Web Server
sudo apt-get install nginx
Controlling Nginx
Firewall
Nginx Configuration Directory
include /etc/nginx/sites-enabled/*;
cd sites-available
sudo nano default
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /home/ravi/www/html;
index index.html;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
}
Restart Server
Manage logging in python. Suppress and Set Log Level in Python
import logging
logging.getLogger('tensorflow').setLevel(logging.ERROR)
To get all the list of available loggers:
import logging
for key in logging.Logger.manager.loggerDict:
print(key)
Check if MKL is enabled in Tensorflow installation
python -c "import tensorflow; print(tensorflow.pywrap_tensorflow.IsMklEnabled())"
Python contextlib for Timing Python code
If you've ever found yourself needing to measure the execution time of specific portions of your Python code, the `contextlib` module o...
-
If you get the following error while installing FFMPEG or other packages on CentOS 7. Requires: libva.so.1(VA_API_0.33.0)(64bit) Follow...
-
Common error: fallocate: fallocate failed: Text file busy in ubuntu OS Check the allocation: free -m or sudo swapon --show Make sure ...
-
How to fix Warning: mysql_connect(): Can't connect to local MySQL server through socket '/var/mysql/mysql.sock' Apparently, Ma...