Python contextlib for Timing Python code

If you've ever found yourself needing to measure the execution time of specific portions of your Python code, the `contextlib` module offers an elegant solution. In particular, we can use it to create a timer that functions as a context manager, simplifying the process of tracking performance without cluttering your code with timing logic. 

Consider the following example of a timer implemented as a context manager: 

from contextlib import contextmanager
import time

@contextmanager
def timer(label):
    start = time.time()
    try:
        yield
    finally:
        end = time.time()
        print(f'{label}: {end - start} seconds')

# Usage:
with timer('Model training'):
    # Your code block here, e.g., train your model

In this example, the `timer` function is decorated with `@contextmanager`, transforming it into a context manager. When you use this timer within a `with` statement, it automatically measures the time taken to execute the enclosed code block. 

This feature proves exceptionally useful for profiling the performance of various parts of your codebase. For instance, in the context of machine learning, you can easily measure the duration of model training or inference without inserting cumbersome timing statements throughout your code. 

By adopting Python's context manager and the `contextlib` module, you can streamline the process of performance profiling, making your code more readable and focused on its primary logic. Try incorporating this timer into your projects to gain valuable insights into the execution time of critical code segments.

Linux HTOP settings/configuration

vi ~/.config/htop/htoprc 

Update with the following:
 
# Beware! This file is rewritten by htop when settings are changed in the interface.
# The parser is also very primitive, and not human-friendly.
fields=0 48 17 18 38 39 40 2 46 47 49 1
sort_key=46
sort_direction=-1
tree_sort_key=0
tree_sort_direction=1
hide_kernel_threads=1
hide_userland_threads=0
shadow_other_users=0
show_thread_names=0
show_program_path=1
highlight_base_name=0
highlight_megabytes=1
highlight_threads=1
highlight_changes=0
highlight_changes_delay_secs=5
find_comm_in_cmdline=1
strip_exe_from_cmdline=1
show_merged_command=0
tree_view=0
tree_view_always_by_pid=0
header_margin=1
detailed_cpu_time=0
cpu_count_from_one=0
show_cpu_usage=1
show_cpu_frequency=0
show_cpu_temperature=0
degree_fahrenheit=0
update_process_names=0
account_guest_in_cpu_meter=0
color_scheme=0
enable_mouse=1
delay=15
left_meters=LeftCPUs8 Memory Swap
left_meter_modes=1 1 1
right_meters=RightCPUs8 Tasks LoadAverage Uptime
right_meter_modes=1 2 2 2
hide_function_bar=0

Following will be the view:


Disk Utilities in Linux

Linux, renowned for its robustness, is a preferred choice among software professionals. Navigating the expansive world of disk management, Linux offers a spectrum of utilities. In this concise guide, we'll explore essential tools tailored for professionals, shedding light on three practical examples for each.

  1. df (Disk Free):

    • Example 1: Display disk space usage for all mounted filesystems.

      df -h
      
    • Example 2: Show the total, used, and available space for a specific directory.

      df -h /path/to/directory
      
    • Example 3: Highlight human-readable disk space information in 1K blocks.

      df -kh
      
  2. du (Disk Usage):

    • Example 1: Determine the size of a directory and its subdirectories.

      du -h /path/to/directory
      
    • Example 2: Display the sizes of all files in a directory.

      du -h --max-depth=1
      
    • Example 3: Sort and list top disk space-consuming directories.

      du -h --max-depth=1 | sort -rh
      
  3. fdisk:

    • Example 1: List all available disk partitions.

      fdisk -l
      
    • Example 2: Create a new partition on a specific disk.

      fdisk /dev/sdX
      
    • Example 3: Delete a partition from a disk.

      fdisk /dev/sdX -d
      
  4. parted:

    • Example 1: Display partition information for a specific disk.

      parted /dev/sdX print
      
    • Example 2: Resize a partition to a specific size.

      parted /dev/sdX resizepart N 50G
      
    • Example 3: Create a new partition on a disk.

      parted /dev/sdX mkpart primary ext4 0% 100%
      
  5. gparted:

    • Example 1: Launch the graphical partition editor.

      gparted
      
    • Example 2: Resize a partition using the GUI interface.

      gparted /dev/sdX
      
    • Example 3: Format a partition with a specific file system.

      gparted /dev/sdX - create new ext4 partition
      
  6. badblocks:

    • Example 1: Check for bad blocks on a specific disk.

      badblocks -s /dev/sdX
      
    • Example 2: Print a list of bad blocks found during testing.

      badblocks -l /dev/sdX
      
    • Example 3: Non-destructive read-write test for bad blocks.

      badblocks -n /dev/sdX
      
  7. smartctl:

    • Example 1: Display SMART information for a specific disk.

      smartctl -a /dev/sdX
      
    • Example 2: Run a short self-test on a disk.

      smartctl -t short /dev/sdX
      
    • Example 3: View the test result and overall health status.

      smartctl -l selftest /dev/sdX
      
  8. rsync:

    • Example 1: Copy files and directories to a remote server.

      rsync -av /local/path/ user@remote:/remote/path/
      
    • Example 2: Synchronize two directories, updating only changed files.

      rsync -av --update /source/directory/ /destination/directory/
      
    • Example 3: Backup a directory and exclude specific files.

      rsync -av --exclude='*.log' /source/directory/ /backup/directory/
      
  9. lvm (Logical Volume Manager):

    • Example 1: Display information about logical volumes.

      lvs
      
    • Example 2: Create a new logical volume.

      lvcreate -L 10G -n myvolume myvg
      
    • Example 3: Extend the size of a logical volume.

      lvextend -L +5G /dev/myvg/myvolume
      
  10. fstrim:

    • Example 1: Trim all mounted filesystems.

      fstrim -a
      
    • Example 2: Trim a specific mount point.

      fstrim /path/to/mount/point
      
    • Example 3: Dry-run trim to check for discardable blocks.

      fstrim -v --dry-run /path/to/mount/point
      
  11. duf:

    • Example 1: Display disk usage with a colorful and user-friendly interface.

      duf -c
      
    • Example 2: Show disk usage for a specific directory.

      duf /path/to/directory
      
    • Example 3: Display disk usage with detailed information.

      duf -l
      
  12. ncdu:

    • Example 1: Launch the NCurses Disk Usage tool for interactive disk exploration.

      ncdu
      
    • Example 2: Scan and analyze disk usage for a specific directory.

      ncdu /path/to/directory
      
    • Example 3: Display disk usage in a human-readable format.

      ncdu -x
      
  13. fstab (File System Table):

    • Example 1: Edit the fstab file for automatic mounting of partitions at boot.

      nano /etc/fstab
      
    • Example 2: View the current entries in the fstab file.

      cat /etc/fstab
      
    • Example 3: Check for errors in the fstab file syntax.

      mount -a --test

Get PIP package versions and licenses within a virtual environment

pip install pip-licenses

pip-licenses --format=csv \
--with-urls \
--with-authors \
--with-maintainers \
--with-description > packages.csv

# The following will produce main packages only without nested dependencies.
pip install pipdeptree
pipdeptree -f | grep -P '^\w+' | cut -d = -f 1 > main-packages.txt

Ubuntu Error constructing proxy for org.gnome.Terminal

If the terminal fails to start when using Ubuntu desktop via VNC Server. If you see the below error in ~/.vnc/<some-ip>.log.

Ubuntu org.gnome.Terminal:/org/gnome/Terminal/Factory0: Error calling StartServiceByName

Error constructing proxy for org.gnome.Terminal: /org/gnome/Terminal/Factory0: 
Error calling StartServiceByName for org.gnome.Terminal: 
GDBus.Error:org.freedesktop.DBus.Error.Spawn.ChildExited: 
Process /usr/lib/gnome-terminal/gnome-terminal-server exited with status 8 

Instead of using gnome-terminal use urxvt

sudo apt-get update
sudo apt-get install rxvt-unicode

Launch urxvt by adding it to the `~/.vnc/xstartup` vim ~/.vnc/xstartup
#!/bin/sh

export XKL_XMODMAP_DISABLE=1
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey

vncconfig -iconic &
gnome-panel &
gnome-settings-daemon &
metacity &
nautilus &
gnome-terminal &
urxvt &

Restart VNCSERVER session
vncserver -kill :1

vncserver :1

Install iGPU / Intel integrated GPU drivers for OpenVINO Inference

To enable Intel integrated GPU / iGPU for running deep learning inference with OpenVINO, we need to install the Intel Graphics Compute Runtime for oneAPI Level Zero and OpenCL™ Driver on Linux.

More info: https://dgpu-docs.intel.com/driver/client/overview.html

The script required to run is: openvino/scripts/install_dependencies/install_NEO_OCL_driver.sh


wget https://raw.githubusercontent.com/openvinotoolkit/openvino/master/scripts/install_dependencies/install_NEO_OCL_driver.sh

chmod +x ./install_NEO_OCL_driver.sh

sudo ./install_NEO_OCL_driver.sh

sudo usermod -aG video $USER

sudo usermod -aG render $USER

Build OpenVINO from source - Linux

Official Instructions - Click Here

Build instructions for OpenVINO from Source with the Python API Wrapper

OpenVINO 2022.1.0 and later require GLIBC 2.27+, check with `ldd --version`

Here instructions are for Python 3.7, you can change it to 3.6 as well.

Software Requirements:

- CMake 3.13 or higher
- GCC 7.5 or higher to build OpenVINO Runtime
- Python 3.6 or higher for OpenVINO Runtime Python API
- (Optional) Install Intel® Graphics Compute Runtime for OpenCL™ Driver package 19.41.14441 to enable inference on Intel integrated GPUs.


$ sudo apt-get install python3.7-dev

$ pip install cython numpy

$ cd ~ #openvino will be installed in ~/openvino

$ git clone https://github.com/openvinotoolkit/openvino.git
$ cd openvino
$ git submodule update --init --recursive
$ chmod +x install_dependencies.sh
$ ./install_dependencies.sh
$ mkdir build && cd build

$ cmake -DCMAKE_BUILD_TYPE=Release \
-DENABLE_INTEL_GNA=OFF -DENABLE_INTEL_MYRIAD_COMMON=OFF \
-DENABLE_PYTHON=ON \
-DPYTHON_EXECUTABLE=`which python3.7` \
-DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.7m.so \
-DPYTHON_INCLUDE_DIR=/usr/include/python3.7 ..

$ make --jobs=$(nproc --all)

$ export PYTHONPATH=$PYTHONPATH:~/openvino/bin/intel64/Release/lib/python_api/python3.7/
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/openvino/bin/intel64/Release/lib/

# OR Install the wheel with PIP
$ pip install ~/openvino/build/wheel/*.whl

# TEST BUILD
$ python3.7
>>> from openvino.inference_engine import IENetwork, IECore

# Test Benchmark app

$ alias benchmark_app=~/openvino/bin/intel64/Release/benchmark_app 
$ benchmark_app -h

Install Python3.6 or Python3.7 in Ubuntu 20.04; ImportError: libpython3.6m.so.1.0: cannot open shared object file

Ubuntu 20.04 has Python 3.8 by default. 
See below to install Python 3.6 or 3.7

sudo -E add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.6
sudo apt-get install -y libpython3.6-dev

Install like above if you get below error

ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory


Debugging Python with pudb

1. Install pudb pip package..

 $ pip install pudb

2. Add the following statement in python code.
import pudb; pudb.set_trace()
3. Or, Launch the debugger like below
$ python -m pudb my_script.py

TMUX Scrolling with Mouse

TMUX Scrolling with Mouse



# Add these in ~/.tmux.conf
set -g mouse on    #For tmux version 2.1 and up  
set -g @scroll-down-exit-copy-mode "off"
set -g terminal-overrides 'xterm*:smcup@:rmcup@' # Sane scrolling


# RELOAD the conf file:  
# tmux source-file ~/.tmux.conf

# If you get Error: protocol version mismatch (client 8, server 7)
# sudo killall -9 tmux

# If conf error, make sure you have tmux version >2.1
# tmux -V
# Update tmux. Ex: sudo yum install tmux

Install ClamAV on Amazon Linux 2

Install ClamAV Scan on Amazon Linux 2

sudo amazon-linux-extras install epel
sudo yum install clamav clamd -y
sudo sed -i -e "s/Example/#Example/" /etc/freshclam.conf
sudo sed -i -e "s:#DatabaseDirectory /var/lib/clamav:DatabaseDirectory /var/lib/clamav:" /etc/freshclam.conf
sudo sed -i -e "s:#UpdateLogFile /var/log/freshclam.log:UpdateLogFile /var/log/freshclam.log:" /etc/freshclam.conf
sudo sed -i -e "s/#DatabaseOwner clamupdate/DatabaseOwner clamupdate/" /etc/freshclam.conf
sudo vi /etc/clamd.d/scan.conf 
# Change MaxThreads if desired.

sudo freshclam
clamscan -r / 2>&1 | tee openvinoami_clamav_scan.txt

# If problems arise, restart
sudo pkill freshclam
sudo freshclam
clamscan -r / 2>&1 | tee openvinoami_clamav_scan.txt

Host a static website in Ubuntu using Nginx

 Install Ngnix Web Server

sudo apt-get install nginx

Controlling Nginx

sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx
sudo systemctl reload nginx

Firewall

sudo ufw allow 'Nginx Full'
sudo ufw reload

Nginx Configuration Directory

cd /etc/nginx
sudo cp nginx.conf nginx.conf.bak

sudo nano nginx.conf

Verify this line is uncommented, include /etc/nginx/sites-enabled/*;

cd sites-available
sudo nano default

server { listen 80 default_server; listen [::]:80 default_server; server_name _; root /home/ravi/www/html; index index.html; location / { # First attempt to serve request as file, then # as directory, then fall back to displaying a 404. try_files $uri $uri/ =404; } }

Restart Server

sudo systemctl restart nginx


Manage logging in python. Suppress and Set Log Level in Python

Example to supress Tensorflow logging

import logging
logging.getLogger('tensorflow').setLevel(logging.ERROR)

To get all the list of available loggers:

import logging
for key in logging.Logger.manager.loggerDict:
    print(key)

Check if MKL is enabled in Tensorflow installation

Check if MKL is enabled in Tensorflow



python -c "import tensorflow; print(tensorflow.pywrap_tensorflow.IsMklEnabled())"




Python contextlib for Timing Python code

If you've ever found yourself needing to measure the execution time of specific portions of your Python code, the `contextlib` module o...