JavaScript/WebGL real-time face tracking and expression detection library. Build your own emoticons animated in real time in the browser! SVG and THREE.js integration demos are provided.

NOTICE: Apple©’s lawyers threatened us to file a complain on the 21th of August 2019 for infringing their intellectual property. So we have removed our main demo webojis.com and we have replaced the 3D animated fox by a raccoon.

Indeed, Apple© owns the intellectual property of 3D animated foxes (but not on raccoons yet). Thank you for your understanding.

JavaScript/WebGL library to detect and reproduce facial expressions

Build your own animated emoticon embedded in web applications with this library. The video is processed client-side only. No devices are needed except a standard webcam.

By default a webcam feedback image is overlaid by the face detection frame. The face detection will adapt to all lighting conditions, but the evaluation of expressions can be noisy if the lighting is too high, too weak, or if there is backlighting. The webcam feedback image is useful to see the quality of the input video feed.

The computing power of your GPU is important. If your GPU is powerful, many detections per second will be processed and the result will be smooth and accurate. An old or low quality mobile device it may be slower. But it should work well on medium or high end mobile devices.

Table of contents

• Features
• Architecture
• Demonstrations
  • Run locally
  • Using the ES6 module
• Integration
  • With a bundler
  • With JavaScript frontend frameworks
  • Native
• Hosting
• About the tech
  • Under the hood
  • Compatibility
  • Future improvements
• Documentation
• Need more ?
• License
• See also
• References

Features

• face detection and tracking,
• recognize 11 facial expressions,
• face rotation along the 3 axis,
• robust for all lighting conditions,
• mobile friendly,
• example provided using SVG and THREE.js.

Architecture

• /assets/: assets, both for 3D and 2D demonstrations (3D meshes, images),
• /demos/: the most interesting: the demos !,
• /dist/: heart of the library:
  • jeelizFaceTransfer.js: main minified script. It gets the webcam video feed, exploit the neural network to detect the face and the expressions and stabilize the result,
  • jeelizFaceTransferNNC.json: neural network model loaded by the main script,
• /doc/: some additionnal documentation,
• /helpers/: The outputs of the main script are very raw. It is convenient to use these helpers to animate a 3D model with the THREE.js helper or a SVG file with the SVG helper. All demos use these helpers,
• /libs/: some javascript libs,
• /meshConverter/: only for the THREE.js use. Tool to build the 3D model file including morphs from separate .OBJ files.

Demonstrations

We have built a demo application based on this library, available on webojis.com. We have add recording capabilities using RecordRTC JavaScript library to save separately video from the <canvas> and audio from the microphone. Then we send them to the server and we encode the video using FFMPEG.

All the following demos are included in this repository, in the /demos path. You can try them:

• THREE.JS based demos:

  • Cute raccoon: source code, live demo
  • Cute raccoon from a MP4 video: source code, live demo

• SVG based demos:

  • Cool Cartman: source code, live demo

• Native demos:

  • Cute raccoon: source code and instruction to build and run it This demo works as a native application using Apache Cordova. It has been tested successfully on iOS. The base code is the same than the THREE.js raccoon demo.

• Third party demos:

  • Piano Genie: play the piano with your face expressions! This is pretty addictive ;). live demo - ace-piano-genie.glitch.me, code and further details

  • Browse Hands Free, by Oz Ramos: The demo improves web accessibility for disabled people. You can control the mouse cursor with your head (turn your head to move the cursor, smile to click). live demo - browsehandsfree.com - click on the camera icon to start, further details. The code is available on github here: handsfreejs/handsfree.

  • Vauta, by Nono: You can anime a .VRM 3D character avatar in the browser. This web application is only in Japanese. live demo - vauta.netlify.com, tweet and video

  • Drowsiness detection, by Abhilash26 aka Dinodroid: Be sure to don’t fall asleep when driving thanks to this webapp! You can try it here: dont-drive-drowsy.glitch.me, view the source code or a demo video

  • Expressions reader, by Abhilash26 aka Dinodroid: detects 5 high level expressions (happiness, fear, anger, surprise, sadness) from the morph coefficients given by this lib, and display them as smileys. You can try it here: emotion-reader.glitch.me or browse the source code

If you have made an application or a fun demonstration using this library, we would love to check it out and add a link here ! Just contact us on Twitter @StartupJeeliz or LinkedIn.

Run locally

You just have to serve the content of this directory using a HTTPS server. Webcam access can be not authorized depending on the web browser the application is hosted by an unsecured HTTP server. You can use Docker for example to run a HTTPS server:

1. Run docker-compose

docker-compose up

2. Open a browser and go to localhost:8888

If you have not bought a webcam yet, a screenshot video of the Cartman Demo is available here:

Using the ES6 module

/dist/jeelizFaceTransferES6.js is exactly the same than /dist/jeelizFaceTransfer.js except that it works with ES6, so you can import it directly using:

import 'dist/jeelizFaceTransferES6.js'

or using require:

const faceTransfer =require('./lib/jeelizFaceTransferES6.js')
//...

The demos have not been ported to ES6 yet. You are welcome to submit a pull request :).

Integration

With a bundler

If you use this library with a bundler (typically Webpack or Parcel), first you should use the ES6 version.

Then, with the standard library, we load the neural network model (specified by NNCpath provided as initialization parameter) using AJAX for the following reasons:

• If the user does not accept to share its webcam, or if WebGL is not enabled, we don’t have to load the neural network model,
• We suppose that the library is deployed using a static HTTPS server.

With a bundler, it is a bit more complicated. It is easier to load the neural network model using a classical import or require call and to provide it using the NNC init parameter:

const faceTransferAPI = require('./lib/jeelizFaceTransferES6.js')
const neuralNetworkModel = require('./dist/jeelizFaceTransferNNC.json')

faceTransferAPI.init({
  NNC:  neuralNetworkModel, //instead of NNCpath
  //... other init parameters
});

With JavaScript frontend frameworks

We don’t cover here the integration with mainstream JavaScript frontend frameworks (React, Vue, Angular). If you submit Pull Request adding the boilerplate or a demo integrated with specific frameworks, you are welcome and they will be accepted of course. We can provide this kind of integration as a specific development service ( please contact us here ). But it is not so hard to do it by yourself. Here is a bunch of submitted issues dealing with React integration. Most of them are for Jeeliz FaceFilter, but the problem is similar:

• Angular integration: Jeff Winder has integrated this library with Angular / Electron. His amazing work is published on Github here: JeffWinder/jeelizWeboji-angular-electron-example,
• React integration: #74 and #122
• is it possible to use this library in react native project
• Having difficulty using JeelizThreejsHelper in ReactApp

You can also take a look at these Github code repositories:

• ikebastuz/jeelizTest: React demo of a CSS3D FaceFilter. It is based on Create React App
• CloffWrangler/facevoice: Another demo based on [Create React App]
• nickydev100/FFMpeg-Angular-Face-Filter: Angular boilerplate

Native

It is possible to execute a JavaScript application using this library into a Webview for a native app integration. But with IOS the camera access is disabled inside webviews. You have to implement a hack to stream the camera video into the webview using websockets.

His hack has been implemented into this repository:

• Apache Cordova IOS demo (it should also work on Android)
• Youtube video of the demo
• Github submitted issue
• Linkedin post detailing pros and cons

But it is still a dirty hack introducing a bottleneck. It still run pretty well on a high end device (tested on Iphone XR), but it is better to stick on a full web environment.

Hosting

This library requires the user’s webcam feed through MediaStream API. Your application should then be hosted with a HTTPS server (the certificate can be self-signed). It won’t work at all with unsecure HTTP, even locally with some web browsers.

Be careful to enable gzip HTTP/HTTPS compression for JSON and JS files. Indeed, the neuron network JSON in, /dist/ is quite heavy, but very well compressed with GZIP. You can check the gzip compression of your server here.

The neuron network JSON file is loaded using an ajax XMLHttpRequest after the user has accepted to share its camera. We proceed this way to avoid to load this quite heavy file if the user refuses to share its webcam or if there is no webcam available. The loading will be faster if you systematically preload the JSON file using a service worker or a simple raw XMLHttpRequest just after the loading of the HTML page. Then the file will be in the browser cache and will be fast to request.

About the tech

Under the hood

The heart of the lib is JEEFACETRANSFERAPI. It is implemented by /dist/jeelizFaceTransfer.js script. It relies on Jeeliz WebGL Deep Learning technology to detect and track the user’s face using a deep learning network, and to simultaneously evaluate the expression factors. The accuracy is adaptative: the best is the hardware, the more detections are processed per second. All is done client-side.

The documentation of JEEFACETRANSFERAPI is included in this repository as a PDF file, /doc/jeefacetransferAPI.pdf. In the main scripts of the demonstration, we never call these methods directly, but always through the helpers. Here is the indices of the morphs returned by this API:

• 0: smileRight → closed mouth smile right
• 1: smileLeft → closed mouth smile left
• 2: eyeBrowLeftDown → eyebrow left frowned
• 3: eyeBrowRightDown → eyebrow right frowned
• 4: eyeBrowLeftUp → eyebrow left up (surprised)
• 5: eyeBrowRightUp → eyebrow right up (surprised)
• 6: mouthOpen → mouth open
• 7: mouthRound → mouth round
• 8: eyeRightClose → close right eye
• 9: eyeLeftClose → close left eye
• 10: mouthNasty → mouth nasty (upper lip raised)

Compatibility

• If WebGL2 is available, it uses WebGL2 and no specific extension is required,
• If WebGL2 is not available but WebGL1, we require either OES_TEXTURE_FLOAT extension or OES_TEXTURE_HALF_FLOAT extension,
• If WebGL2 is not available, and if WebGL1 is not available or neither OES_TEXTURE_FLOAT or OES_HALF_TEXTURE_FLOAT are implemented, the user is not compatible.

In all cases, you need to have WebRTC implemented in the web browser, otherwise this library will not be able to get the webcam video feed. The compatibility tables are on, caniuse.com: WebGL1, WebGL2, WebRTC.

If a compatibility error is triggered, please post an issue on this repository. If this is a camera access error, please first retry after closing all applications which could use your device (Skype, Messenger, other browser tabs and windows, …). Please include:

• a screenshot of webglreport.com - WebGL1 (about your WebGL1 implementation),
• a screenshot of webglreport.com - WebGL2 (about your WebGL2 implementation),
• the log from the web console,
• the steps to reproduce the bug, and screenshots.

This library works quite everywhere, and it works very well with a high end device like an Iphone X. But if your device is too cheap or too old, it will perform too few evaluations per second and the application will be slow.

Future improvements

We are currently working hard on this project. New neural networks are training and we are confident about improving this library. Here are our ways to improve:

• Better emotion detection with a better neural network (improving the structure, the face generator, …),
• Better tracking stabilization,
• Add a calibration estimation to take into account the variations of the coefficients between different faces.

You can subscribe to the Jeeliz Youtube channel or to the @StartupJeeliz Twitter account to be kept informed of our cutting edge developments.

Documentation

Documentation

• JEEFACETRANSFERAPI: All the helpers rely on this API to get the facial morph coefficients. With this documentation you can interface this library with your own 3D or 2D engine. Click here to read the PDF of the specs,
• README.md about using the meshConverter: Used in the THREE.JS Raccoon demo

Articles and tutorials

We list articles and tutorials about using this library:

• Create your own animated emoticon for the web
• Integrate the animated emoticon on your website

Need more?

@Jeeliz we have fully developed this library so we can still improve it or fit it to your needs. In particular:

• adapt it to your own 3D/2D engine,
• train a new neural network model adapted for a specific case (take account of new expressions for example),
• use your specific mesh format,
• integrate it better to your workflow,
• manage the video/audio capture and encoding of the weboji.

If you are interested, please contact-us here.

License

Apache 2.0. This application is free for both commercial and non-commercial use.

We appreciate attribution by including the Jeeliz logo and a link to the Jeeliz website in your application or desktop website. Of course we do not expect a large link to Jeeliz over your face filter, but if you can put the link in the credits/about/help/footer section it would be great.

See also

Jeeliz main face detection and tracking library is called Jeeliz FaceFilter API. It handles multi-face detection, and for each tracked face it provides the rotation angles and the mouth opening factor. It is perfect to build your own Snapchat/MSQRD like face filters running in the browser. It comes with dozen of integration demo, including a face swap.

If you want to detect if the user is looking at the screen or not, Jeeliz Glance Tracker is what you are looking for. It can be useful to play a video only if the user is watching it (and to pause it otherwise). This library needs fewer resources and the neural network file is much lighter.

If are interested by glasses virtual try-on (sunglasses, spectacles, ski masks), you can take a look at Jeeliz VTO widget. It includes a high quality and lightweight 3D engine which implements the following features: deferred shading, PBR, raytraced shadows, normal mapping, … It also reconstructs the lighting environment around the user (ambient and directional lighting). The glasses are hosted in a database on our servers. If you want to add some models, please contact-us.

References

• Jeeliz official website
• Three.JS official website with documentation, demos, examples…
• Webgl Academy: tutorials about WebGL and THREE.JS