Gliding Phenomena

Linux HTOP settings/configuration

vi ~/.config/htop/htoprc

Update with the following:

# Beware! This file is rewritten by htop when settings are changed in the interface.
# The parser is also very primitive, and not human-friendly.
fields=0 48 17 18 38 39 40 2 46 47 49 1
sort_key=46
sort_direction=-1
tree_sort_key=0
tree_sort_direction=1
hide_kernel_threads=1
hide_userland_threads=0
shadow_other_users=0
show_thread_names=0
show_program_path=1
highlight_base_name=0
highlight_megabytes=1
highlight_threads=1
highlight_changes=0
highlight_changes_delay_secs=5
find_comm_in_cmdline=1
strip_exe_from_cmdline=1
show_merged_command=0
tree_view=0
tree_view_always_by_pid=0
header_margin=1
detailed_cpu_time=0
cpu_count_from_one=0
show_cpu_usage=1
show_cpu_frequency=0
show_cpu_temperature=0
degree_fahrenheit=0
update_process_names=0
account_guest_in_cpu_meter=0
color_scheme=0
enable_mouse=1
delay=15
left_meters=LeftCPUs8 Memory Swap
left_meter_modes=1 1 1
right_meters=RightCPUs8 Tasks LoadAverage Uptime
right_meter_modes=1 2 2 2
hide_function_bar=0

Following will be the view:

Disk Utilities in Linux

Linux, renowned for its robustness, is a preferred choice among software professionals. Navigating the expansive world of disk management, Linux offers a spectrum of utilities. In this concise guide, we'll explore essential tools tailored for professionals, shedding light on three practical examples for each.

df (Disk Free):
- Example 1: Display disk space usage for all mounted filesystems.
```
df -h
```
- Example 2: Show the total, used, and available space for a specific directory.
```
df -h /path/to/directory
```
- Example 3: Highlight human-readable disk space information in 1K blocks.
```
df -kh
```
du (Disk Usage):
- Example 1: Determine the size of a directory and its subdirectories.
```
du -h /path/to/directory
```
- Example 2: Display the sizes of all files in a directory.
```
du -h --max-depth=1
```
- Example 3: Sort and list top disk space-consuming directories.
```
du -h --max-depth=1 | sort -rh
```
fdisk:
- Example 1: List all available disk partitions.
```
fdisk -l
```
- Example 2: Create a new partition on a specific disk.
```
fdisk /dev/sdX
```
- Example 3: Delete a partition from a disk.
```
fdisk /dev/sdX -d
```
parted:
- Example 1: Display partition information for a specific disk.
```
parted /dev/sdX print
```
- Example 2: Resize a partition to a specific size.
```
parted /dev/sdX resizepart N 50G
```
- Example 3: Create a new partition on a disk.
```
parted /dev/sdX mkpart primary ext4 0% 100%
```
gparted:
- Example 1: Launch the graphical partition editor.
```
gparted
```
- Example 2: Resize a partition using the GUI interface.
```
gparted /dev/sdX
```
- Example 3: Format a partition with a specific file system.
```
gparted /dev/sdX - create new ext4 partition
```
badblocks:
- Example 1: Check for bad blocks on a specific disk.
```
badblocks -s /dev/sdX
```
- Example 2: Print a list of bad blocks found during testing.
```
badblocks -l /dev/sdX
```
- Example 3: Non-destructive read-write test for bad blocks.
```
badblocks -n /dev/sdX
```
smartctl:
- Example 1: Display SMART information for a specific disk.
```
smartctl -a /dev/sdX
```
- Example 2: Run a short self-test on a disk.
```
smartctl -t short /dev/sdX
```
- Example 3: View the test result and overall health status.
```
smartctl -l selftest /dev/sdX
```
rsync:
- Example 1: Copy files and directories to a remote server.
```
rsync -av /local/path/ user@remote:/remote/path/
```
- Example 2: Synchronize two directories, updating only changed files.
```
rsync -av --update /source/directory/ /destination/directory/
```
- Example 3: Backup a directory and exclude specific files.
```
rsync -av --exclude='*.log' /source/directory/ /backup/directory/
```
lvm (Logical Volume Manager):
- Example 1: Display information about logical volumes.
```
lvs
```
- Example 2: Create a new logical volume.
```
lvcreate -L 10G -n myvolume myvg
```
- Example 3: Extend the size of a logical volume.
```
lvextend -L +5G /dev/myvg/myvolume
```
fstrim:
- Example 1: Trim all mounted filesystems.
```
fstrim -a
```
- Example 2: Trim a specific mount point.
```
fstrim /path/to/mount/point
```
- Example 3: Dry-run trim to check for discardable blocks.
```
fstrim -v --dry-run /path/to/mount/point
```
duf:
- Example 1: Display disk usage with a colorful and user-friendly interface.
```
duf -c
```
- Example 2: Show disk usage for a specific directory.
```
duf /path/to/directory
```
- Example 3: Display disk usage with detailed information.
```
duf -l
```
ncdu:
- Example 1: Launch the NCurses Disk Usage tool for interactive disk exploration.
```
ncdu
```
- Example 2: Scan and analyze disk usage for a specific directory.
```
ncdu /path/to/directory
```
- Example 3: Display disk usage in a human-readable format.
```
ncdu -x
```
fstab (File System Table):
- Example 1: Edit the fstab file for automatic mounting of partitions at boot.
```
nano /etc/fstab
```
- Example 2: View the current entries in the fstab file.
```
cat /etc/fstab
```
- Example 3: Check for errors in the fstab file syntax.
```
mount -a --test
```

Get PIP package versions and licenses within a virtual environment

pip install pip-licenses

pip-licenses --format=csv \

--with-urls \

--with-authors \

--with-maintainers \

--with-description > packages.csv

# The following will produce main packages only without nested dependencies.
pip install pipdeptree
pipdeptree -f | grep -P '^\w+' | cut -d = -f 1 > main-packages.txt

Ubuntu Error constructing proxy for org.gnome.Terminal

If the terminal fails to start when using Ubuntu desktop via VNC Server. If you see the below error in ~/.vnc/<some-ip>.log.

Ubuntu org.gnome.Terminal:/org/gnome/Terminal/Factory0: Error calling StartServiceByName

Error constructing proxy for org.gnome.Terminal: /org/gnome/Terminal/Factory0:

Error calling StartServiceByName for org.gnome.Terminal:

GDBus.Error:org.freedesktop.DBus.Error.Spawn.ChildExited:

Process /usr/lib/gnome-terminal/gnome-terminal-server exited with status 8

Instead of using gnome-terminal use urxvt

sudo apt-get update
sudo apt-get install rxvt-unicode

Launch urxvt by adding it to the `~/.vnc/xstartup` vim ~/.vnc/xstartup

#!/bin/sh

export XKL_XMODMAP_DISABLE=1
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey

vncconfig -iconic &
gnome-panel &
gnome-settings-daemon &
metacity &
nautilus &
gnome-terminal &
urxvt &

Restart VNCSERVER session

vncserver -kill :1

vncserver :1

Install iGPU / Intel integrated GPU drivers for OpenVINO Inference

To enable Intel integrated GPU / iGPU for running deep learning inference with OpenVINO, we need to install the Intel Graphics Compute Runtime for oneAPI Level Zero and OpenCL™ Driver on Linux.

Intel Graphics Compute Runtime: https://github.com/intel/compute-runtime/releases

See : https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html

More info: https://dgpu-docs.intel.com/driver/client/overview.html

# After Installing Intel Graphics Compute Runtime

sudo apt install hwinfo clinfo -y

After installation, add video and render user groups.

sudo usermod -aG video $USER
sudo usermod -aG render $USER

# Logout and login

Build OpenVINO from source - Linux

Official Instructions - Click Here

Build instructions for OpenVINO from Source with the Python API Wrapper

OpenVINO 2022.1.0 and later require GLIBC 2.27+, check with `ldd --version`

Here instructions are for Python 3.7, you can change it to 3.6 as well.

Software Requirements:

- CMake 3.13 or higher
- GCC 7.5 or higher to build OpenVINO Runtime
- Python 3.6 or higher for OpenVINO Runtime Python API
- (Optional) Install Intel® Graphics Compute Runtime for OpenCL™ Driver package 19.41.14441 to enable inference on Intel integrated GPUs.

$ sudo apt-get install python3.7-dev

$ pip install cython numpy

$ cd ~ #openvino will be installed in ~/openvino

$ git clone https://github.com/openvinotoolkit/openvino.git
$ cd openvino
$ git submodule update --init --recursive
$ chmod +x install_dependencies.sh
$ ./install_dependencies.sh
$ mkdir build && cd build

$ cmake -DCMAKE_BUILD_TYPE=Release \

-DENABLE_INTEL_GNA=OFF -DENABLE_INTEL_MYRIAD_COMMON=OFF \

-DENABLE_PYTHON=ON \
-DPYTHON_EXECUTABLE=`which python3.7` \
-DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.7m.so \
-DPYTHON_INCLUDE_DIR=/usr/include/python3.7 ..

$ make --jobs=$(nproc --all)

$ export PYTHONPATH=$PYTHONPATH:~/openvino/bin/intel64/Release/lib/python_api/python3.7/
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/openvino/bin/intel64/Release/lib/

# OR Install the wheel with PIP

$ pip install ~/openvino/build/wheel/*.whl

# TEST BUILD

$ python3.7
>>> from openvino.inference_engine import IENetwork, IECore

# Test Benchmark app

$ alias benchmark_app=~/openvino/bin/intel64/Release/benchmark_app 
$ benchmark_app -h

Install Python3.6 or Python3.7 in Ubuntu 20.04; ImportError: libpython3.6m.so.1.0: cannot open shared object file

Ubuntu 20.04 has Python 3.8 by default.
See below to install Python 3.6 or 3.7

sudo -E add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.6
sudo apt-get install -y libpython3.6-dev

Install like above if you get below error

ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

Debugging Python with pudb

1. Install pudb pip package..

$ pip install pudb

2. Add the following statement in python code.

import pudb; pudb.set_trace()

3. Or, Launch the debugger like below

$ python -m pudb my_script.py

TMUX Scrolling with Mouse

# Add these in ~/.tmux.conf

set -g mouse on    #For tmux version 2.1 and up  
set -g @scroll-down-exit-copy-mode "off"
set -g terminal-overrides 'xterm*:smcup@:rmcup@' # Sane scrolling

# RELOAD the conf file:

# tmux source-file ~/.tmux.conf

# If you get Error: protocol version mismatch (client 8, server 7)

# sudo killall -9 tmux

# If conf error, make sure you have tmux version >2.1

# tmux -V

# Update tmux. Ex: sudo yum install tmux

Install ClamAV on Amazon Linux 2

Install ClamAV Scan on Amazon Linux 2

sudo amazon-linux-extras install epel
sudo yum install clamav clamd -y
sudo sed -i -e "s/Example/#Example/" /etc/freshclam.conf
sudo sed -i -e "s:#DatabaseDirectory /var/lib/clamav:DatabaseDirectory /var/lib/clamav:" /etc/freshclam.conf
sudo sed -i -e "s:#UpdateLogFile /var/log/freshclam.log:UpdateLogFile /var/log/freshclam.log:" /etc/freshclam.conf
sudo sed -i -e "s/#DatabaseOwner clamupdate/DatabaseOwner clamupdate/" /etc/freshclam.conf
sudo vi /etc/clamd.d/scan.conf 
# Change MaxThreads if desired.

sudo freshclam
clamscan -r / 2>&1 | tee openvinoami_clamav_scan.txt

# If problems arise, restart
sudo pkill freshclam
sudo freshclam
clamscan -r / 2>&1 | tee openvinoami_clamav_scan.txt

Dynamic Port Forwarding using Firefox Browser

[bob@workstation ~]$ ssh -D 1080 bastion.securecorp.io [bob@bastion ~]$

SOCKS5 configuration in Firefox:

Source: https://www.redhat.com/sysadmin/ssh-dynamic-port-forwarding

Host a static website in Ubuntu using Nginx

Install Ngnix Web Server

sudo apt-get install nginx

Controlling Nginx

sudo systemctl start nginx

sudo systemctl stop nginx

sudo systemctl restart nginx

sudo systemctl reload nginx

Firewall

sudo ufw allow 'Nginx Full'

sudo ufw reload

Nginx Configuration Directory

cd /etc/nginx

sudo cp nginx.conf nginx.conf.bak

sudo nano nginx.conf

Verify this line is uncommented, include /etc/nginx/sites-enabled/*;

cd sites-available

sudo nano default

server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name _;
        root /home/ravi/www/html;
        index index.html;
        
        location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ =404;
        }
}

`Restart Server`

sudo systemctl restart nginx

Source: https://medium.com/@jasonrigden/how-to-host-a-static-website-with-nginx-8b2dd0c5b301

Manage logging in python. Suppress and Set Log Level in Python

Example to supress Tensorflow logging


import logging
logging.getLogger('tensorflow').setLevel(logging.ERROR)

To get all the list of available loggers:


import logging
for key in logging.Logger.manager.loggerDict:
    print(key)

Check if MKL is enabled in Tensorflow installation

Check if MKL is enabled in Tensorflow

python -c "import tensorflow; print(tensorflow.pywrap_tensorflow.IsMklEnabled())"

Start Jupyter or Jupyter-lab on startup of ec-2 instance.

Edit the crontab by running

sudo crontab -e

and add the following

@reboot su ec2-user -c "/home/ec2-user/.start_jupyter.sh"

Create a file

vi /home/ec2-user/.start_jupyter.sh

and add the following

cd /home/ec2-user

source /home/ec2-user/.bashrc

source /opt/intel/openvino/bin/setupvars.sh

/home/ec2-user/.local/bin/jupyter notebook --no-browser --NotebookApp.allow_password_change=False --NotebookApp.token="$INSTANCE_ID" --ip 0.0.0.0 --port 8888 > /tmp/jupyter.out 2>&1 &

We can leave this option to true, but changing password is not working...

--NotebookApp.allow_password_change=False because change password is not working. It is a bug in Jupyter. See https://github.com/jupyter/notebook/issues/3842

Give executable file permissions.

chmod 755 .start_jupyter.sh

Debugging

sudo cat /var/log/cron

Also,

cat /tmp/jupyter.out

If you want to set password randomly on EC2-Instance:

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`

INSTANCE_ID=`curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id`

cd /home/ec2-user

source /home/ec2-user/.bashrc

source /opt/intel/openvino/bin/setupvars.sh

/home/ec2-user/.local/bin/jupyter notebook --no-browser --NotebookApp.allow_password_change=False --NotebookApp.token="$INSTANCE_ID" --ip 0.0.0.0 --port 8888 > /tmp/jupyter.out 2>&1 &

See:

https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html#security

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

Gliding Phenomena

Linux HTOP settings/configuration

Disk Utilities in Linux

Get PIP package versions and licenses within a virtual environment

Ubuntu Error constructing proxy for org.gnome.Terminal

Install iGPU / Intel integrated GPU drivers for OpenVINO Inference

Build OpenVINO from source - Linux

Install Python3.6 or Python3.7 in Ubuntu 20.04; ImportError: libpython3.6m.so.1.0: cannot open shared object file

Debugging Python with pudb

TMUX Scrolling with Mouse

TMUX Scrolling with Mouse

Install ClamAV on Amazon Linux 2

Install ClamAV Scan on Amazon Linux 2

Dynamic Port Forwarding using Firefox Browser

Host a static website in Ubuntu using Nginx

Install Ngnix Web Server

Controlling Nginx

Firewall

Nginx Configuration Directory

`Restart Server`

Manage logging in python. Suppress and Set Log Level in Python

Check if MKL is enabled in Tensorflow installation

Start Jupyter or Jupyter-lab on startup of ec-2 instance.

Debugging

If you want to set password randomly on EC2-Instance:

Python contextlib for Timing Python code

Report Abuse

Followers