Manage logging in python. Suppress and Set Log Level in Python

Example to supress Tensorflow logging

import logging
logging.getLogger('tensorflow').setLevel(logging.ERROR)

To get all the list of available loggers:

import logging
for key in logging.Logger.manager.loggerDict:
    print(key)

Check if MKL is enabled in Tensorflow installation

Check if MKL is enabled in Tensorflow



python -c "import tensorflow; print(tensorflow.pywrap_tensorflow.IsMklEnabled())"




Start Jupyter or Jupyter-lab on startup of ec-2 instance.


Start Jupyter or Jupyter-lab on startup of ec-2 instance.

Edit the crontab by running

sudo crontab -e


and add the following

@reboot su ec2-user -c "/home/ec2-user/.start_jupyter.sh"


Create a file 

vi /home/ec2-user/.start_jupyter.sh

and add the following

cd /home/ec2-user

source /home/ec2-user/.bashrc

source /opt/intel/openvino/bin/setupvars.sh

/home/ec2-user/.local/bin/jupyter notebook --no-browser --NotebookApp.allow_password_change=False --NotebookApp.token="$INSTANCE_ID" --ip 0.0.0.0 --port 8888 > /tmp/jupyter.out 2>&1 &


We can leave this option to true, but changing password is not working...
--NotebookApp.allow_password_change=False because change password is not working. It is a bug in Jupyter. See https://github.com/jupyter/notebook/issues/3842

Give executable file permissions.

chmod 755 .start_jupyter.sh 

Debugging

sudo cat /var/log/cron

Also,

cat /tmp/jupyter.out 


If you want to set password randomly on EC2-Instance:

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`

INSTANCE_ID=`curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id`


cd /home/ec2-user

source /home/ec2-user/.bashrc

source /opt/intel/openvino/bin/setupvars.sh

/home/ec2-user/.local/bin/jupyter notebook --no-browser --NotebookApp.allow_password_change=False --NotebookApp.token="$INSTANCE_ID" --ip 0.0.0.0 --port 8888 > /tmp/jupyter.out 2>&1 &


See:

Set Python3 as default in CentOS or Amazon Linux2


# Install Python3
sudo yum install -y python3
# Start by registering python2 as an alternative
sudo alternatives --install /usr/bin/python python /usr/bin/python2 50

# Register python3.5 as an alternative
sudo alternatives --install /usr/bin/python python /usr/bin/python3.5 60

# Select which Python version to use
sudo alternatives --config python

Ubuntu 18.04: Create/Modify SWAP partition memory limit.

Common error:
fallocate: fallocate failed: Text file busy in ubuntu OS 

Check the allocation:
free -m
or
sudo swapon --show

Make sure you have enough free disk space by
df -h

Disable the use of swap, allocate new swap space, enable swap.
sudo swapoff -a

sudo fallocate -l 8G /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Check the allocation:
free -m
or
sudo swapon --show

For detailed information: https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-18-04

Connecting to Ubuntu Remote Desktop from Mac OS

Connecting to Ubuntu or any linux server via Mac OS:
-- Access the "Spotlight" by hitting CMD + Space-bar
-- Within the Spotlight field, enter vnc://your_server_ip:5900

If you get an error when connecting to Ubuntu from Mac OS using VNC screen sharing:
"The software on the remote computer appears to be incompatible with this version of Screen Sharing."
On Ubuntu 18.04:
$ gsettings set org.gnome.Vino require-encryption false
Verify the setting with
$ gsettings list-recursively org.gnome.Vino | grep encrypt

Fix wrong time in Ubuntu

Fix wrong time in Ubuntu .
sudo timedatectl set-local-rtc 1 --adjust-system-clock

BIOS settings to Install Ubuntu

If Ubuntu installation shows only the USB drive from which you are installing, make sure you have the BIOS settings as follows.
BIOS settings to Install Ubuntu alongside with Windows :
  1. Disable Secure Boot.
  2. Set SATA-controller to AHCI from RAID On.
  3. Set boot mode to legacy from UEFI.

How to fix error Requires: libva.so.1(VA_API_0.33.0)(64bit)

If you get the following error while installing FFMPEG or other packages on CentOS 7.

Requires: libva.so.1(VA_API_0.33.0)(64bit)

Follow the following steps to install libva

sudo yum install libwayland-client    

wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libva-1.8.3-1.el7.x86_64.rpm 

sudo rpm -i libva-1.8.3-1.el7.x86_64.rpm          


Sample error log when installing FFMPEG:

Error: Package: ffmpeg-libs-3.4.7-1.el7.x86_64 (rpmfusion-free-updates)
           Requires: libva.so.1(VA_API_0.33.0)(64bit)
Error: Package: ffmpeg-libs-3.4.7-1.el7.x86_64 (rpmfusion-free-updates)
           Requires: libva-x11.so.1()(64bit)
Error: Package: libmfx-1.21-2.el7.x86_64 (epel)
           Requires: libva-x11.so.1()(64bit)
Error: Package: libmfx-1.21-2.el7.x86_64 (epel)
           Requires: libva-drm.so.1()(64bit)
Error: Package: ffmpeg-libs-3.4.7-1.el7.x86_64 (rpmfusion-free-updates)
           Requires: libva-drm.so.1()(64bit)
Error: Package: ffmpeg-libs-3.4.7-1.el7.x86_64 (rpmfusion-free-updates)
           Requires: libva.so.1()(64bit)
Error: Package: libmfx-1.21-2.el7.x86_64 (epel)
           Requires: libva.so.1()(64bit)

Azure VM --os-disk-size-gb partition, format and mount

If you are using Azure VM --os-disk-size-gb parameter to increase the size of the OS disk, you need to partition, format and mount to use it.

Check the current status with
 fdisk -l


 df -Th 


Step 1: Create a partition (See sample output below)
 sudo fdisk /dev/sda
(use n to create new partition, and follow prompts)

Step 2: Refresh partition table
 sudo partprobe

Step 3: Format the new partition, get the name from step 1 or use fdisk -l 
 mkfs.xfs /dev/sda3 

Step 4: Mount 
 mkdir /mnt/shared
 mount /dev/sda3 /mnt/shared/  

Step 5: Verify  
 df -Th 

Step 6: Edit the /etc/fstab to automatically mount when restarted 


 vi  /etc/fstab
Append line /dev/sda3 /mnt/shared xfs defaults 0 0  . Please note the filesystem type "xfs" is used as it was formatted in "xfs" in Step 3. You could use other formats as well based on your requirements .
                                                                                                                               
#                                                                                                                              
# /etc/fstab                                                                                                                   
# Created by anaconda on Wed Dec 19 23:06:16 2018                                                                              
#                                                                                                                              
# Accessible filesystems, by reference, are maintained under '/dev/disk'                                                       
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info                                                    
#                                                                                                                              
UUID=fdb125e2-1ce5-4b7a-98db-88c0f66a86ee /                       xfs     defaults        0 0                                  
UUID=d4c5a046-3513-4ff0-a955-621311869210 /boot                   xfs     defaults        0 0                                  
/dev/sda3 /mnt/shared xfs defaults 0 0     


Sample Output:

[root@azvm azureusr]# fdisk /dev/sda

The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): n
Partition type:
   p   primary (2 primary, 0 extended, 2 free)
   e   extended
Select (default p): 
Using default response p
Partition number (3,4, default 3): 
First sector (62914560-125829119, default 62914560): 
Using default value 62914560
Last sector, +sectors or +size{K,M,G} (62914560-125829119, default 125829119): 
Using default value 125829119
Partition 3 of type Linux and of size 30 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@azvm azureusr]# partprobe
[root@azvm azureusr]# fdisk -l

Disk /dev/sda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x000eba6a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048    62914559    30944256   83  Linux
/dev/sda3        62914560   125829119    31457280   83  Linux

Disk /dev/sdb: 751.6 GB, 751619276800 bytes, 1468006400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x3fde6d89

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1             128  1468004351   734002112   83  Linux

[root@azvm azureusr]# df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/sda2      xfs        32G  5.8G   26G  19% /
devtmpfs       devtmpfs  186G     0  186G   0% /dev
tmpfs          tmpfs     186G     0  186G   0% /dev/shm
tmpfs          tmpfs     186G  9.5M  186G   1% /run
tmpfs          tmpfs     186G     0  186G   0% /sys/fs/cgroup
/dev/sda1      xfs       521M   68M  454M  13% /boot
/dev/sdb1      ext4      740G   76M  703G   1% /mnt/resource
tmpfs          tmpfs      38G     0   38G   0% /run/user/1000

[root@azvm azureusr]# mkfs.xfs /dev/sda3 
meta-data=/dev/sda3              isize=512    agcount=4, agsize=1966080 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=7864320, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=3840, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@azvm azureusr]# mkdir /mnt/shared

[root@azvm azureusr]# mount /dev/sda3 /mnt/shared/ 

[root@azvm azureusr]# df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/sda2      xfs        30G  5.4G   25G  19% /
devtmpfs       devtmpfs  174G     0  174G   0% /dev
tmpfs          tmpfs     174G     0  174G   0% /dev/shm
tmpfs          tmpfs     174G  9.1M  174G   1% /run
tmpfs          tmpfs     174G     0  174G   0% /sys/fs/cgroup
/dev/sda1      xfs       497M   65M  433M  13% /boot
/dev/sdb1      ext4      689G   73M  654G   1% /mnt/resource
tmpfs          tmpfs      35G     0   35G   0% /run/user/1000
/dev/sda3      xfs        30G   33M   30G   1% /mnt/shared



Unblock a specific port from the firewall in Centos 7

Following is a example which unblocks port 8888

firewall-cmd --permanent --add-port=8888/tcp
firewall-cmd --reload
firewall-cmd --list-all

opencv: Install opencv CentOS


sudo yum install opencv opencv-python opencv-devel-docs 

#OPTIONAL DEPENDENCIES:
sudo yum install -y python-devel gtk2-devel libdc1394-devel libv4l-devel ffmpeg-devel gstreamer-plugins-base-devel
sudo yum install -y libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel libtiff-devel libwebp-devel
sudo yum install -y gcc  cmake  git gtk2-devel pkgconfig  numpy  ffmpeg

If your using virtual environment, copy the relevant cv files to the virtual environment.

 cp /usr/lib64/python2.7/site-packages/cv* .venv/lib/python2.7/site-packages/

Caffe: Visualizing Caffe Network prototxt file using GraphViz: draw_net.py

Visualizing Caffe Net topology/ntwork/prototxt file using GraphViz
From the Caffe root directory, you can generate a graph in image format from a .prototxt model file:

#INSTALL PRE-REQS:
$ make pycaffe
$ pip install pydotplus
$ yum install graphviz

Visualize a CNN in left-to-right:

#goto CAFFE ROOT
$ python/draw_net.py mynetwrk.prototxt mynetwrk.png

Output formats could be : PNG, PDF, DOT or other GraphViz supported formats.
Visualize a CNN in top-to-bottom:

$ python/draw_net.py --rankdir TB mynetwrk.prototxt mynetwrk.png

Following errors can be eliminated if you install the PRE-REQS (pip install pydotplus; yum install graphviz)

pydotplus.graphviz.InvocationException: GraphViz's executables not found
Exception: "dot" not found in path.
ImportError: No module named pydot

Python contextlib for Timing Python code

If you've ever found yourself needing to measure the execution time of specific portions of your Python code, the `contextlib` module o...