August 1, 2019

2544 words 12 mins read

onnx/models

onnx/models

A collection of pre-trained, state-of-the-art models in the ONNX format

repo name onnx/models
repo link https://github.com/onnx/models
homepage
language Jupyter Notebook
size (curr.) 165588 kB
stars (curr.) 2030
created 2017-10-06
license MIT License

ONNX Model Zoo

Open Neural Network Exchange (ONNX) is an open standard format for representing machine learning models. ONNX is supported by a community of partners who have implemented it in many frameworks and tools.

The ONNX Model Zoo is a collection of pre-trained, state-of-the-art models in the ONNX format contributed by community members like you. Accompanying each model are Jupyter notebooks for model training and running inference with the trained model. The notebooks are written in Python and include links to the training dataset as well as references to the original paper that describes the model architecture.

We are standardizing on Git LFS (Large File Storage) to store our ONNX model files. To download an ONNX model, navigate to the appropriate Github page and click the Download button on the top right.

Models

Read the Usage section below for more details on the file formats in the ONNX Model Zoo (.onnx, .pb, .npz), downloading multiple ONNX models through Git LFS command line, and starter Python code for validating your ONNX model using test data.

Vision

Language

Other

Image Classification

This collection of models take images as input, then classifies the major objects in the images into 1000 object categories such as keyboard, mouse, pencil, and many animals.

Model Class Reference Description
MobileNet Sandler et al. Light-weight deep neural network best suited for mobile and embedded vision applications. Top-5 error from paper - ~10%
ResNet He et al. A CNN model (up to 152 layers). Uses shortcut connections to achieve higher accuracy when classifying images. Top-5 error from paper - ~3.6%
SqueezeNet Iandola et al. A light-weight CNN model providing AlexNet level accuracy with 50x fewer parameters. Top-5 error from paper - ~20%
VGG Simonyan et al. Deep CNN model(up to 19 layers). Similar to AlexNet but uses multiple smaller kernel-sized filters that provides more accuracy when classifying images. Top-5 error from paper - ~8%
AlexNet Krizhevsky et al. A Deep CNN model (up to 8 layers) where the input is an image and the output is a vector of 1000 numbers. Top-5 error from paper - ~15%
GoogleNet Szegedy et al. Deep CNN model(up to 22 layers). Comparatively smaller and faster than VGG and more accurate in detailing than AlexNet. Top-5 error from paper - ~6.7%
CaffeNet Krizhevsky et al. Deep CNN variation of AlexNet for Image Classification in Caffe where the max pooling precedes the local response normalization (LRN) so that the LRN takes less compute and memory.
RCNN_ILSVRC13 Girshick et al. Pure Caffe implementation of R-CNN for image classification. This model uses localization of regions to classify and extract features from images.
DenseNet-121 Huang et al. Model that has every layer connected to every other layer and passes on its own feature providing strong gradient flow and more diversified features.
Inception_V1 Szegedy et al. This model is same as GoogLeNet, implemented through Caffe2 that has improved utilization of the computing resources inside the network and helps with the vanishing gradient problem. Top-5 error from paper - ~6.7%
Inception_V2 Szegedy et al. Deep CNN model for Image Classification as an adaptation to Inception v1 with batch normalization. This model has reduced computational cost and improved image resolution compared to Inception v1. Top-5 error from paper ~4.82%
ShuffleNet_V1 Zhang et al. Extremely computation efficient CNN model that is designed specifically for mobile devices. This model greatly reduces the computational cost and provides a ~13x speedup over AlexNet on ARM-based mobile devices. Compared to MobileNet, ShuffleNet achieves superior performance by a significant margin due to it’s efficient structure. Top-1 error from paper - ~32.6%
ShuffleNet_V2 Zhang et al. Extremely computation efficient CNN model that is designed specifically for mobile devices. This network architecture design considers direct metric such as speed, instead of indirect metric like FLOP. Top-1 error from paper - ~30.6%
ZFNet-512 Zeiler et al. Deep CNN model (up to 8 layers) that increased the number of features that the network is capable of detecting that helps to pick image features at a finer level of resolution. Top-5 error from paper - ~14.3%

Domain-based Image Classification

This subset of models classify images for specific domains and datasets.

Model Class Reference Description
MNIST-Handwritten Digit Recognition Convolutional Neural Network with MNIST Deep CNN model for handwritten digit identification

Object Detection & Image Segmentation

Object detection models detect the presence of multiple objects in an image and segment out areas of the image where the objects are detected. Semantic segmentation models partition an input image by labeling each pixel into a set of pre-defined categories.

Model Class Reference Description
Tiny YOLOv2 Redmon et al. A real-time CNN for object detection that detects 20 different classes. A smaller version of the more complex full YOLOv2 network.
SSD Liu et al. Single Stage Detector: real-time CNN for object detection that detects 80 different classes.
Faster-RCNN Ren et al. Increases efficiency from R-CNN by connecting a RPN with a CNN to create a single, unified network for object detection that detects 80 different classes.
Mask-RCNN He et al. A real-time neural network for object instance segmentation that detects 80 different classes. Extends Faster R-CNN as each of the 300 elected ROIs go through 3 parallel branches of the network: label prediction, bounding box prediction and mask prediction.
YOLO v2 Redmon et al. A CNN model for real-time object detection system that can detect over 9000 object categories. It uses a single network evaluation, enabling it to be more than 1000x faster than R-CNN and 100x faster than Faster R-CNN.
YOLO v2-coco Redmon et al. A CNN model for real-time object detection system that can detect over 9000 object categories. It uses a single network evaluation, enabling it to be more than 1000x faster than R-CNN and 100x faster than Faster R-CNN. This model is trained with COCO dataset and contains 80 classes.
YOLO v3 Redmon et al. A deep CNN model for real-time object detection that detects 80 different classes. A little bigger than YOLOv2 but still very fast. As accurate as SSD but 3 times faster.
Tiny YOLOv3 Redmon et al. A smaller version of YOLOv3 model.
DUC Wang et al. Deep CNN based pixel-wise semantic segmentation model with >80% mIOU (mean Intersection Over Union). Trained on cityscapes dataset, which can be effectively implemented in self driving vehicle systems.
FCN Long et al. Deep CNN based segmentation model trained end-to-end, pixel-to-pixel that produces efficient inference and learning. Built off of AlexNet, VGG net, GoogLeNet classification methods. contribute

Body, Face & Gesture Analysis

Face detection models identify and/or recognize human faces and emotions in given images. Body and Gesture Analysis models identify gender and age in given image.

Model Class Reference Description
ArcFace Deng et al. A CNN based model for face recognition which learns discriminative features of faces and produces embeddings for input face images.
CNN Cascade Li et al. The model operates at multiple resolutions, quickly rejecting the background regions in the fast low resolution stages in an image and carefully evaluates a small number of challenging candidates in the last high resolution stage. contribute
Emotion FerPlus Barsoum et al. Deep CNN for emotion recognition trained on images of faces.
Age and Gender Classification using Convolutional Neural Networks Levi et al. This model accurately classifies gender and age even the amount of learning data is limited.contribute

Image Manipulation

Image manipulation models use neural networks to transform input images to modified output images. Some popular models in this category involve style transfer or enhancing images by increasing resolution.

Model Class Reference Description
Unpaired Image to Image Translation using Cycle consistent Adversarial Network Zhu et al. The model uses learning to translate an image from a source domain X to a target domain Y in the absence of paired examples. contribute
Super Resolution with sub-pixel CNN Shi et al. A deep CNN that uses sub-pixel convolution layers to upscale the input image.
Fast Neural Style Transfer Johnson et al. This method uses a loss network pretrained for image classification to define perceptual loss functions that measure perceptual differences in content and style between images. The loss network remains fixed during the training process.

Speech & Audio Processing

This class of models uses audio data to train models that can identify voice, generate music, or even read text out loud.

Model Class Reference Description
Speech recognition with deep recurrent neural networks Graves et al. A RNN model for sequential data for speech recognition. Labels problems where the input-output alignment is unknowncontribute
Deep voice: Real time neural text to speech Arik et al. A DNN model that performs end-to-end neural speech synthesis. Requires fewer parameters and it is faster than other systems. contribute
Sound Generative models WaveNet: A Generative Model for Raw Audio A CNN model that generates raw audio waveforms. Has predictive distribution for each audio sample. Generates realistic music fragments. contribute

Machine Comprehension

This subset of natural language processing models that answer questions about a given context paragraph.

Model Class Reference Description
Bidirectional Attention Flow Seo et al. A model that answers a query about a given context paragraph.
BERT-Squad Devlin et al. This model answers questions based on the context of the given input paragraph.
GPT-2 Radford et al. A large transformer-based language model that given a sequence of words within some text, predicts the next word.

Machine Translation

This class of natural language processing models learns how to translate input text to another language.

Model Class Reference Description
Neural Machine Translation by jointly learning to align and translate Bahdanau et al. Aims to build a single neural network that can be jointly tuned to maximize the translation performance. contribute
Google’s Neural Machine Translation System Wu et al. This model helps to improve issues faced by the Neural Machine Translation (NMT) systems like parallelism that helps accelerate the final translation speed.contribute

Language Modelling

This subset of natural language processing models learns representations of language from large corpuses of text.

Model Class Reference Description
Deep Neural Network Language Models Arisoy et al. A DNN acoustic model. Used in many natural language technologies. Represents a probability distribution over all possible word strings in a language. contribute

Visual Question Answering & Dialog

This subset of natural language processing models uses input images to answer questions about those images.

Model Class Reference Description
VQA: Visual Question Answering Agrawal et al. A model that takes an image and a free-form, open-ended natural language question about the image and outputs a natural-language answer. contribute
Yin and Yang: Balancing and Answering Binary Visual Questions Zhang et al. Addresses VQA by converting the question to a tuple that concisely summarizes the visual concept to be detected in the image. Next, if the concept can be found in the image, it provides a “yes” or “no” answer. Its performance matches the traditional VQA approach on unbalanced dataset, and outperforms it on the balanced dataset. contribute
Making the V in VQA Matter Goyal et al. Balances the VQA dataset by collecting complementary images such that every question is associated with a pair of similar images that result in two different answers to the question, providing a unique interpretable model that provides a counter-example based explanation. contribute
Visual Dialog Das et al. An AI agent that holds a meaningful dialog with humans in natural, conversational language about visual content. Curates a large-scale Visual Dialog dataset (VisDial). contribute

Other interesting models

There are many interesting deep learning models that do not fit into the categories described above. The ONNX team would like to highly encourage users and researchers to contribute their models to the growing model zoo.

Model Class Reference Description
Text to Image Generative Adversarial Text to image Synthesis Effectively bridges the advances in text and image modeling, translating visual concepts from characters to pixels. Generates plausible images of birds and flowers from detailed text descriptions. contribute
Time Series Forecasting Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks The model extracts short-term local dependency patterns among variables and to discover long-term patterns for time series trends. It helps to predict solar plant energy output, electricity consumption, and traffic jam situations. contribute
Recommender systems DropoutNet: Addressing Cold Start in Recommender Systems A collaborative filtering method that makes predictions about an individual’s preference based on preference information from other users.contribute
Collaborative filtering Neural Collaborative Filtering A DNN model based on the interaction between user and item features using matrix factorization. contribute
Autoencoders A Hierarchical Neural Autoencoder for Paragraphs and Documents An LSTM (long-short term memory) auto-encoder to preserve and reconstruct multi-sentence paragraphs.contribute

Usage

Every ONNX backend should support running the models out of the box. After downloading and extracting the tarball of each model, you will find:

  • A protobuf file model.onnx that represents the serialized ONNX model.
  • Test data (in the form of serialized protobuf TensorProto files or serialized NumPy archives).

Usage - Test data starter code

The test data files can be used to validate ONNX models from the Model Zoo. We have provided the following interface examples for you to get started. Please replace onnx_backend in your code with the appropriate framework of your choice that provides ONNX inferencing support, and likewise replace backend.run_model with the framework’s model evaluation logic.

There are two different formats for the test data files:

  • Serialized protobuf TensorProtos (.pb), stored in folders with the naming convention test_data_set_*.
import numpy as np
import onnx
import os
import glob
import onnx_backend as backend

from onnx import numpy_helper

model = onnx.load('model.onnx')
test_data_dir = 'test_data_set_0'

# Load inputs
inputs = []
inputs_num = len(glob.glob(os.path.join(test_data_dir, 'input_*.pb')))
for i in range(inputs_num):
    input_file = os.path.join(test_data_dir, 'input_{}.pb'.format(i))
    tensor = onnx.TensorProto()
    with open(input_file, 'rb') as f:
        tensor.ParseFromString(f.read())
    inputs.append(numpy_helper.to_array(tensor))

# Load reference outputs
ref_outputs = []
ref_outputs_num = len(glob.glob(os.path.join(test_data_dir, 'output_*.pb')))
for i in range(ref_outputs_num):
    output_file = os.path.join(test_data_dir, 'output_{}.pb'.format(i))
    tensor = onnx.TensorProto()
    with open(output_file, 'rb') as f:
        tensor.ParseFromString(f.read())
    ref_outputs.append(numpy_helper.to_array(tensor))

# Run the model on the backend
outputs = list(backend.run_model(model, inputs))

# Compare the results with reference outputs.
for ref_o, o in zip(ref_outputs, outputs):
    np.testing.assert_almost_equal(ref_o, o)
  • Serialized Numpy archives, stored in files with the naming convention test_data_*.npz. Each file contains one set of test inputs and outputs.
import numpy as np
import onnx
import onnx_backend as backend

# Load the model and sample inputs and outputs
model = onnx.load(model_pb_path)
sample = np.load(npz_path, encoding='bytes')
inputs = list(sample['inputs'])
outputs = list(sample['outputs'])

# Run the model with an onnx backend and verify the results
np.testing.assert_almost_equal(outputs, backend.run_model(model, inputs))

Usage - Git LFS

On default, cloning this repository will not download any ONNX models. Install Git LFS with pip install git lfs.

To download a specific model: git lfs pull --include="[MODELNAME].onnx"

To download all models: git lfs pull --include="*" --exclude=""

Usage - Model visualization

You can see visualizations of each model’s network architecture by using Netron.

Contributions

Do you want to contribute a model? To get started, pick any model presented above with the contribute link under the Description column. The links point to a page containing guidelines for making a contribution.

License

MIT License

comments powered by Disqus