import logging
logging.getLogger('tensorflow').setLevel(logging.ERROR)
import logging
for key in logging.Logger.manager.loggerDict:
print(key)
Manage logging in python. Suppress and Set Log Level in Python
Example to supress Tensorflow logging
Check if MKL is enabled in Tensorflow installation
Check if MKL is enabled in Tensorflow
python -c "import tensorflow; print(tensorflow.pywrap_tensorflow.IsMklEnabled())"
Start Jupyter or Jupyter-lab on startup of ec-2 instance.
Start Jupyter or Jupyter-lab on startup of ec-2 instance.
sudo crontab -e
and add the following
@reboot su ec2-user -c "/home/ec2-user/.start_jupyter.sh"
vi /home/ec2-user/.start_jupyter.sh
and add the following
cd /home/ec2-user
source /home/ec2-user/.bashrc
source /opt/intel/openvino/bin/setupvars.sh
/home/ec2-user/.local/bin/jupyter notebook --no-browser --NotebookApp.allow_password_change=False --NotebookApp.token="$INSTANCE_ID" --ip 0.0.0.0 --port 8888 > /tmp/jupyter.out 2>&1 &
We can leave this option to true, but changing password is not working...
--NotebookApp.allow_password_change=False because change password is not working. It is a bug in Jupyter. See https://github.com/jupyter/notebook/issues/3842Give executable file permissions.
chmod 755 .start_jupyter.sh
Debugging
sudo cat /var/log/cron
cat /tmp/jupyter.out
If you want to set password randomly on EC2-Instance:
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
INSTANCE_ID=`curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id`
cd /home/ec2-user
source /home/ec2-user/.bashrc
source /opt/intel/openvino/bin/setupvars.sh
/home/ec2-user/.local/bin/jupyter notebook --no-browser --NotebookApp.allow_password_change=False --NotebookApp.token="$INSTANCE_ID" --ip 0.0.0.0 --port 8888 > /tmp/jupyter.out 2>&1 &
Set Python3 as default in CentOS or Amazon Linux2
# Install Python3
sudo yum install -y python3# Start by registering python2 as an alternative
sudo alternatives --install /usr/bin/python python /usr/bin/python2 50
# Register python3.5 as an alternative
sudo alternatives --install /usr/bin/python python /usr/bin/python3.5 60
# Select which Python version to use
sudo alternatives --config pythonUbuntu 18.04: Create/Modify SWAP partition memory limit.
Common error:
Check the allocation:
Make sure you have enough free disk space by
Disable the use of swap, allocate new swap space, enable swap.
Check the allocation:
For detailed information: https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-18-04
fallocate: fallocate failed: Text file busy in ubuntu OS
Check the allocation:
free -m or sudo swapon --show
Make sure you have enough free disk space by
df -h
Disable the use of swap, allocate new swap space, enable swap.
sudo swapoff -a sudo fallocate -l 8G /swapfile sudo mkswap /swapfile sudo swapon /swapfile
Check the allocation:
free -m or sudo swapon --show
For detailed information: https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-18-04
Connecting to Ubuntu Remote Desktop from Mac OS
Connecting to Ubuntu or any linux server via Mac OS:
-- Access the "Spotlight" by hitting CMD + Space-bar
-- Within the Spotlight field, enter vnc://your_server_ip:5900
If you get an error when connecting to Ubuntu from Mac OS using VNC screen sharing:
"The software on the remote computer appears to be incompatible with this version of Screen Sharing."
On Ubuntu 18.04:
-- Access the "Spotlight" by hitting CMD + Space-bar
-- Within the Spotlight field, enter vnc://your_server_ip:5900
If you get an error when connecting to Ubuntu from Mac OS using VNC screen sharing:
"The software on the remote computer appears to be incompatible with this version of Screen Sharing."
On Ubuntu 18.04:
$ gsettings set org.gnome.Vino require-encryption falseVerify the setting with
$ gsettings list-recursively org.gnome.Vino | grep encrypt
Fix wrong time in Ubuntu
Fix wrong time in Ubuntu .
sudo timedatectl set-local-rtc 1 --adjust-system-clock
BIOS settings to Install Ubuntu
If Ubuntu installation shows only the USB drive from which you are installing, make sure you have the BIOS settings as follows.
BIOS settings to Install Ubuntu alongside with Windows :
- Disable Secure Boot.
- Set SATA-controller to AHCI from RAID On.
- Set boot mode to legacy from UEFI.
How to fix error Requires: libva.so.1(VA_API_0.33.0)(64bit)
If you get the following error while installing FFMPEG or other packages on CentOS 7.
Follow the following steps to install libva
Sample error log when installing FFMPEG:
Requires: libva.so.1(VA_API_0.33.0)(64bit)
Follow the following steps to install libva
sudo yum install libwayland-client
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libva-1.8.3-1.el7.x86_64.rpm
sudo rpm -i libva-1.8.3-1.el7.x86_64.rpm
Sample error log when installing FFMPEG:
Error: Package: ffmpeg-libs-3.4.7-1.el7.x86_64 (rpmfusion-free-updates)
Requires: libva.so.1(VA_API_0.33.0)(64bit)
Error: Package: ffmpeg-libs-3.4.7-1.el7.x86_64 (rpmfusion-free-updates)
Requires: libva-x11.so.1()(64bit)
Error: Package: libmfx-1.21-2.el7.x86_64 (epel)
Requires: libva-x11.so.1()(64bit)
Error: Package: libmfx-1.21-2.el7.x86_64 (epel)
Requires: libva-drm.so.1()(64bit)
Error: Package: ffmpeg-libs-3.4.7-1.el7.x86_64 (rpmfusion-free-updates)
Requires: libva-drm.so.1()(64bit)
Error: Package: ffmpeg-libs-3.4.7-1.el7.x86_64 (rpmfusion-free-updates)
Requires: libva.so.1()(64bit)
Error: Package: libmfx-1.21-2.el7.x86_64 (epel)
Requires: libva.so.1()(64bit)
Azure VM --os-disk-size-gb partition, format and mount
If you are using Azure VM --os-disk-size-gb parameter to increase the size of the OS disk, you need to partition, format and mount to use it.
Check the current status with
Step 1: Create a partition (See sample output below)
Step 2: Refresh partition table
sudo partprobe
Step 4: Mount
Step 5: Verify
Step 6: Edit the /etc/fstab to automatically mount when restarted
Append line /dev/sda3 /mnt/shared xfs defaults 0 0 . Please note the filesystem type "xfs" is used as it was formatted in "xfs" in Step 3. You could use other formats as well based on your requirements .
Sample Output:
Check the current status with
fdisk -l
df -Th
Step 1: Create a partition (See sample output below)
sudo fdisk /dev/sda
(use n to create new partition, and follow prompts)Step 2: Refresh partition table
sudo partprobe
Step 3: Format the new partition, get the name from step 1 or use fdisk -l
mkfs.xfs /dev/sda3
Step 4: Mount
mkdir /mnt/shared
mount /dev/sda3 /mnt/shared/
Step 5: Verify
df -Th
Step 6: Edit the /etc/fstab to automatically mount when restarted
vi /etc/fstab
#
# /etc/fstab
# Created by anaconda on Wed Dec 19 23:06:16 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=fdb125e2-1ce5-4b7a-98db-88c0f66a86ee / xfs defaults 0 0
UUID=d4c5a046-3513-4ff0-a955-621311869210 /boot xfs defaults 0 0
/dev/sda3 /mnt/shared xfs defaults 0 0
Sample Output:
[root@azvm azureusr]# fdisk /dev/sda
The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): n
Partition type:
p primary (2 primary, 0 extended, 2 free)
e extended
Select (default p):
Using default response p
Partition number (3,4, default 3):
First sector (62914560-125829119, default 62914560):
Using default value 62914560
Last sector, +sectors or +size{K,M,G} (62914560-125829119, default 125829119):
Using default value 125829119
Partition 3 of type Linux and of size 30 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@azvm azureusr]# partprobe
[root@azvm azureusr]# fdisk -l
Disk /dev/sda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x000eba6a
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 62914559 30944256 83 Linux
/dev/sda3 62914560 125829119 31457280 83 Linux
Disk /dev/sdb: 751.6 GB, 751619276800 bytes, 1468006400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x3fde6d89
Device Boot Start End Blocks Id System
/dev/sdb1 128 1468004351 734002112 83 Linux
[root@azvm azureusr]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 xfs 32G 5.8G 26G 19% /
devtmpfs devtmpfs 186G 0 186G 0% /dev
tmpfs tmpfs 186G 0 186G 0% /dev/shm
tmpfs tmpfs 186G 9.5M 186G 1% /run
tmpfs tmpfs 186G 0 186G 0% /sys/fs/cgroup
/dev/sda1 xfs 521M 68M 454M 13% /boot
/dev/sdb1 ext4 740G 76M 703G 1% /mnt/resource
tmpfs tmpfs 38G 0 38G 0% /run/user/1000
[root@azvm azureusr]# mkfs.xfs /dev/sda3
meta-data=/dev/sda3 isize=512 agcount=4, agsize=1966080 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=7864320, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=3840, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@azvm azureusr]# mkdir /mnt/shared
[root@azvm azureusr]# mount /dev/sda3 /mnt/shared/
[root@azvm azureusr]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 xfs 30G 5.4G 25G 19% /
devtmpfs devtmpfs 174G 0 174G 0% /dev
tmpfs tmpfs 174G 0 174G 0% /dev/shm
tmpfs tmpfs 174G 9.1M 174G 1% /run
tmpfs tmpfs 174G 0 174G 0% /sys/fs/cgroup
/dev/sda1 xfs 497M 65M 433M 13% /boot
/dev/sdb1 ext4 689G 73M 654G 1% /mnt/resource
tmpfs tmpfs 35G 0 35G 0% /run/user/1000
/dev/sda3 xfs 30G 33M 30G 1% /mnt/shared
Watch CPU Frequencies in real time
sudo watch -n 1 cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq
Unblock a specific port from the firewall in Centos 7
Following is a example which unblocks port 8888
firewall-cmd --permanent --add-port=8888/tcp
firewall-cmd --reload
firewall-cmd --list-all
opencv: Install opencv CentOS
sudo yum install opencv opencv-python opencv-devel-docs
#OPTIONAL DEPENDENCIES:
sudo yum install -y python-devel gtk2-devel libdc1394-devel libv4l-devel ffmpeg-devel gstreamer-plugins-base-devel
sudo yum install -y libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel libtiff-devel libwebp-devel
sudo yum install -y gcc cmake git gtk2-devel pkgconfig numpy ffmpeg
cp /usr/lib64/python2.7/site-packages/cv* .venv/lib/python2.7/site-packages/
Caffe: Visualizing Caffe Network prototxt file using GraphViz: draw_net.py
Visualizing Caffe Net topology/ntwork/prototxt file using GraphViz
From the Caffe root directory, you can generate a graph in image format from a .prototxt model file:
Visualize a CNN in left-to-right:
Output formats could be : PNG, PDF, DOT or other GraphViz supported formats.
Visualize a CNN in top-to-bottom:
Following errors can be eliminated if you install the PRE-REQS (pip install pydotplus; yum install graphviz)
From the Caffe root directory, you can generate a graph in image format from a .prototxt model file:
#INSTALL PRE-REQS:
$ make pycaffe
$ pip install pydotplus
$ yum install graphviz
Visualize a CNN in left-to-right:
#goto CAFFE ROOT
$ python/draw_net.py mynetwrk.prototxt mynetwrk.png
Output formats could be : PNG, PDF, DOT or other GraphViz supported formats.
Visualize a CNN in top-to-bottom:
$ python/draw_net.py --rankdir TB mynetwrk.prototxt mynetwrk.png
Following errors can be eliminated if you install the PRE-REQS (pip install pydotplus; yum install graphviz)
pydotplus.graphviz.InvocationException: GraphViz's executables not found
Exception: "dot" not found in path.
ImportError: No module named pydot
Subscribe to:
Posts (Atom)