JizhiziLi/animal-matting

Github repository for the paper End-to-end Animal Image Matting

06/11/20: Release the dataset BG-20k. Please fill out this agreement and send it to jili8515@uni.sydney.edu.au from your academic email address to request.

03/11/20: Publish the inference code and a pretrained model that can be used to test on your own animal images.

27/10/20: Publish a video demo contains motivation, network, datasets, and test results on an animal video.

Introduction

Here is a video demo to illustrate the motivation, the network, the datasets, and the test results on an animal video.

We have released the inference code and a pretrained model, which can be found in section inference code. We have also published dataset BG-20k, please follow the guidance in section Bg-20k to access. Since the paper is currently under review, the dataset AM-2k, training code and the rest pretrained models will be made public after review.

GFM

The architecture of our proposed end-to-end method GFM is illustrated below. We adopt three kinds of Representation of Semantic and Transition Area (RoSTa) -TT, -FT, -BT within our method.

We trained GFM with three backbones, -(d) (DenseNet-121), -(r) (ResNet-34), and -(r2b) (ResNet-34 with 2 extra blocks). The trained model for each backbone can be downloaded via the link listed below.

AM-2k

Our proposed AM-2k contains 2,000 high-resolution natural animal images from 20 categories along with manually labeled alpha mattes. Some examples are shown as below, more can be viewed in the video demo.

BG-20k

Our proposed BG-20k contains 20,000 high-resolution background images excluded salient objects, which can be used to help generate high quality synthetic data. Some examples are shown as below, more can be viewed in the video demo.

The BG-20k dataset is publish now!!

You can request it by filling out this agreement and sending it to jili8515@uni.sydney.edu.au from your academic email address. Please note the dataset can be only used for research purpose.

Results Demo

We test GFM on our AM-2k test dataset and show the results as below. More results on AM-2k test set can be found here.

Installation

Requirements:

• Python 3.6.5+ with Numpy and scikit-image
• Pytorch (version 1.4.0)
• Torchvision (version 0.5.0)

1. Clone this repository

   git clone https://github.com/JizhiziLi/animal-matting.git

2. Go into the repository

   cd animal-matting

3. Create conda environment and activate

   conda create -n animalmatting python=3.6.5

   conda activate animalmatting

4. Install dependencies, install pytorch and torchvision separately if you need

   pip install -r requirements.txt

   conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch

Our code has been tested with Python 3.6.5, Pytorch 1.4.0, Torchvision 0.5.0, CUDA 10.1 on Ubuntu 18.04.

Inference Code - How to Test on Your Images

Here we provide the procedure of testing on sample images by our pretrained models:

1. Download pretrained models as shown in section GFM, unzip to folder models/

2. Save your high-resolution sample images in folder samples/original/.

3. Setup parameters in scripts/deploy_samples.sh and run it

   chmod +x scripts/*

   ./scripts/deploy_samples.sh

4. The results of alpha matte and transparent color image will be saved in folder samples/result_alpha/. and samples/result_color/.

We show some sample images from the internet, the predicted alpha mattes, and their transparent results as below. (We adopt arch='e2e_resnet34_2b_gfm_tt' and use hybrid testing strategy.)

Statement

This project is for research purpose only, please contact us for the licence of commercial use. For any other questions please contact jili8515@uni.sydney.edu.au.