Nhân Phan

View project on Github

VISION2020 aims at recovering a high resolution image from a low resolution one. The project is based largely on the excellent research of Xintao Wang, et al. on ESRGAN (2018) and their implementation using Pytorch. Inspired from the research, my version of ESRGAN is optimized and built entirely on Tensorflow 2.0. It successfully resizes the image up to x64 on square area.

Single image super-resolution (SISR), as a fundamental low-level vision problem, has attracted increasing attention in the research community and AI companies. SISR aims at recovering a high-resolution (HR) image from a single low-resolution (LR) one. Since the pioneer work of SRCNN proposed by Dong et al., deep convolution neural network (CNN) approaches have brought prosperous development. Various network architecture designs and training strategies have continuously improved the SR performance.

The Super-Resolution Generative Adversarial Network (SRGAN) is a seminal work that is capable of generating realistic textures during single image super-resolution. However, the hallucinated details are often accompanied with unpleasant artifacts. To further enhance the visual quality, we thoroughly study three key components of SRGAN - network architecture, adversarial loss and perceptual loss, and improve each of them to derive an Enhanced SRGAN (ESRGAN).

In particular, we introduce the Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit. Moreover, we borrow the idea from relativistic GAN to let the discriminator predict relative realness instead of the absolute value. Finally, we improve the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery. Benefiting from these improvements, the proposed ESRGAN achieves consistently better visual quality with more realistic and natural textures than SRGAN.

fig1 ⎯ (x4 per dimension) generated image successfully retains small detail like the strip at the shoulder area and the human head.

fig2 ⎯ (x4 per dimension) Natural features like eyes are well reconstructed.

fig3 ⎯ (x8 per dimension) Double challenging, then model successfully reconstruct pattern and lines.

fig4 ⎯ (x8 dimension) Letters are brought back to vision.

View project on Github 👾

Optical Character Recognition is one active field that bridges between computer vision and natural language processing. As much as the field emerges within machine learning community, it still performs poorly on local language, including Vietnamese with our distinctive symbol (ễ, ẩ, ứ for example). The lack of data is one of the main reason behind it. This project is a submission to a hackathon hosted by Cinnamon AI on that challenge of Vietnamese OCR. The dataset includes all the address written in Vietnamese. The model can be immediately apply in post service to alleviate the need of manual input.

⎯ EVALUATION

• Character Error Rate: 0.04
• Word Error Rate: 0.14
• Sentence Error Rate: 0.82

The hackathon's leader score is 0.1x on the Word Error Rate. Other metric results were not disclosed.

⎯ SAMPLE PREDICTIONS (T: True Label, P: Prediction)

⎯ DATA PREPROCESSING: Using OpenCV with 3 phases

1. Thresholding
2. Resize to 128x1024
3. Remove Recursive (reference to A. Vinciarelli and J. Luettin)

Raw data

Processed data

⎯ MODELLING

• CRNN + CTC Loss is used to solve this challenge.
• CNN blocks with skip connections (inspired by ResNet50) are used to extract the features from input images.
• The extracted feature map will be then passed through the LSTM layers.

Training Log

View project on Google Colaboratory 👾

This project is an entry of the corresponding Kaggle competition.

Every minute, the world loses an area of forest the size of 48 football fields. And deforestation in the Amazon Basin accounts for the largest share, contributing to reduced biodiversity, habitat loss, climate change, and other devastating effects. But better data about the location of deforestation and human encroachment on forests can help governments and local stakeholders respond more quickly and effectively.

From a dataset contains more than 40.000 images, this analysis uses Deep Learning to classify the spatial images of the Amazon forest taken by the satellite. From that, it hopes to shed a light on understanding how the forest has change naturally and manually. Thus, help preventing deforestation.

⎯ SAMPLE PREDICTION

Evaluation

⎯ SOME CHALLENGES

1. Multi-label: Each image is labeled with multiple tags (at least 2, at max 9). The tags fall into 17 categories, which are the forest landscape types. Since the tags in each label are mutually exclusive, they are treated as multiple binary classification problems. Thus, binary cross-entropy are chosen to be the loss function.

2. Imbalance: The dataset is severely imbalance with tags like Primary or Agriculture appear in 90% of the dataset. While other tags like Blooming or Conventional Mine can only be seen in less than 500 observations (even less than 100 for Burn Down).

   In the first base-line experiment, the model was totally bias toward the major tags. It predicts the major tags to appear in every data and almost never made a prediction with the minor tags. To tackle the problem of imbalance dataset, evaluation metrics must be chosen carefully. F2 is chosen to be the main metrics to evaluate the training. It watches over the harmonic mean between the Precision and Recall while favors Recall specifically. In other word, it is the attempt to reduce the number of False Negative, where the model fails to identify the absence of a tag.

3. Optimization: 400.000 images, a CNN model, and Google Colab's limited resource do not seem to mix well together. The training was slow at first and interupted often. Several improvements, mostly on the Tensorflow pipeline, were conducted to speed up the training: Using TFRecord to convert the raw images into byte-like data to reduce the amount of time spending on reading data from their paths.

   Using tf.data.Dataset with shuffle, map, batch, prefetch to optimize the reading data process by redistributing the tasks for agents to work concurrently, thus, avoid bottleneck. An attempt to use cache was also made but failed due to the limited RAM.

   Processing image with Tensorflow: The dataset contains images in JPG - RGBA. The built-in decode function tf.io.decode_jpeg only works on 1 or 3-channel image. Attempt on encoding a JPG RGBA image returns black black and black. We need a tensorflow encoding function to work in this part because the pipeline is built entirely on Tensor for the optimization purpose. To tackle the problem, the raw images were first read by Matplotlib then converted into byte-like and wrote into TFRecords. When reading the data from TF Record, instead of using the built-in decode image function, we use tf.io.parse_tensor following with reshaping.