December 2, 2018

820 words 4 mins read

junyanz/interactive-deep-colorization

junyanz/interactive-deep-colorization

Deep learning software for colorizing black and white images with a few clicks.

repo name junyanz/interactive-deep-colorization
repo link https://github.com/junyanz/interactive-deep-colorization
homepage https://richzhang.github.io/ideepcolor/
language Python
size (curr.) 12625 kB
stars (curr.) 2104
created 2017-05-08
license MIT License

Interactive Deep Colorization

Project Page | Paper | Demo Video | SIGGRAPH Talk

9/3/2018 Update The code now supports a backend PyTorch model (with PyTorch 0.5.0+). Please find the Local Hints Network training code in the colorization-pytorch repository.

Real-Time User-Guided Image Colorization with Learned Deep Priors.
Richard Zhang*, Jun-Yan Zhu*, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros.
In ACM Transactions on Graphics (SIGGRAPH 2017).
(*indicates equal contribution)

We first describe the system (0) Prerequisities and steps for (1) Getting started. We then describe the interactive colorization demo (2) Interactive Colorization (Local Hints Network). There are two demos: (a) a “barebones” version in iPython notebook and (b) the full GUI we used in our paper. We then provide an example of the (3) Global Hints Network.

(0) Prerequisites

  • Linux or OSX
  • Caffe or PyTorch
  • CPU or NVIDIA GPU + CUDA CuDNN.

(1) Getting Started

  • Clone this repo:
git clone https://github.com/junyanz/interactive-deep-colorization ideepcolor
cd ideepcolor
  • Download the reference model
bash ./models/fetch_models.sh

(2) Interactive Colorization (Local Hints Network)

We provide a “barebones” demo in iPython notebook, which does not require QT. We also provide our full GUI demo.

(2a) Barebones Interactive Colorization Demo

If you need to convert the Notebook to an older version, use jupyter nbconvert --to notebook --nbformat 3 ./DemoInteractiveColorization.ipynb.

(2b) Full Demo GUI

  • Install Qt4 and QDarkStyle. (See Installation)

  • Run the UI: python ideepcolor.py --gpu [GPU_ID] --backend [CAFFE OR PYTORCH]. Arguments are described below:

--win_size    [512] GUI window size
--gpu         [0] GPU number
--image_file  ['./test_imgs/mortar_pestle.jpg'] path to the image file
--backend     ['caffe'] either use 'caffe' or 'pytorch'; 'caffe' is the official model from siggraph 2017, and 'pytorch' is the same weights converted
  • User interactions
  • Adding points: Left-click somewhere on the input pad
  • Moving points: Left-click and hold on a point on the input pad, drag to desired location, and let go
  • Changing colors: For currently selected point, choose a recommended color (middle-left) or choose a color on the ab color gamut (top-left)
  • Removing points: Right-click on a point on the input pad
  • Changing patch size: Mouse wheel changes the patch size from 1x1 to 9x9
  • Load image: Click the load image button and choose desired image
  • Restart: Click on the restart button. All points on the pad will be removed.
  • Save result: Click on the save button. This will save the resulting colorization in a directory where the image_file was, along with the user input ab values.
  • Quit: Click on the quit button.

(3) Global Hints Network

We include an example usage of our Global Hints Network, applied to global histogram transfer. We show its usage in an iPython notebook.

Installation

  • Install Caffe or PyTorch. The Caffe model is official. PyTorch is a reimplementation.

    • Install Caffe: see the Caffe installation and Ubuntu installation document. Please compile the Caffe with the python layer support (set WITH_PYTHON_LAYER=1 in the Makefile.config) and build Caffe python library by make pycaffe.

    You also need to add pycaffe to your PYTHONPATH. Use vi ~/.bashrc to edit the environment variables.

    PYTHONPATH=/path/to/caffe/python:$PYTHONPATH
    LD_LIBRARY_PATH=/path/to/caffe/build/lib:$LD_LIBRARY_PATH
    
  • Install scikit-image, scikit-learn, opencv, Qt4, and QDarkStyle pacakges:

# ./install/install_deps.sh
sudo pip install scikit-image
sudo pip install scikit-learn
sudo apt-get install python-opencv
sudo apt-get install python-qt4
sudo pip install qdarkstyle

For Conda users, type the following command lines (this may work for full Anaconda but not Miniconda):

# ./install/install_conda.sh
conda install -c anaconda protobuf  ## photobuf
conda install -c anaconda scikit-learn=0.19.1 ## scikit-learn
conda install -c anaconda scikit-image=0.13.0  ## scikit-image
conda install -c menpo opencv=2.4.11   ## opencv
conda install pyqt=4.11 ## qt4
conda install -c auto qdarkstyle  ## qdarkstyle

Training

Please find a PyTorch reimplementation of the Local Hints Network training code in the colorization-pytorch repository.

Citation

If you use this code for your research, please cite our paper:

@article{zhang2017real,
  title={Real-Time User-Guided Image Colorization with Learned Deep Priors},
  author={Zhang, Richard and Zhu, Jun-Yan and Isola, Phillip and Geng, Xinyang and Lin, Angela S and Yu, Tianhe and Efros, Alexei A},
  journal={ACM Transactions on Graphics (TOG)},
  volume={9},
  number={4},
  year={2017},
  publisher={ACM}
}

Cat Paper Collection

One of the authors objects to the inclusion of this list, due to an allergy. Another author objects on the basis that cats are silly creatures and this is a serious, scientific paper. However, if you love cats, and love reading cool graphics, vision, and learning papers, please check out the Cat Paper Collection: [Github] [Webpage]

comments powered by Disqus