Combining HTML5 Web Applications with OpenCV

The Web Dev Zone is brought to you by Stormpath—offering a pre-built Identity API for developers. Easily build powerful user management, authentication, and authorization into your web and mobile applications. Download this Forrester report on the new landscape of Customer Identity and Access Management.

OpenCV is a popular library for image processing in C++. It promises very high performance and  even real-time video processing algorithms.  While it is a native library, available for Linux, Windows and OSX, it also features Java bindings that make it available to JVM applications.

Featuring hundreds of optimized computer vision algorithms, OpenCV would be an interesting addition to many HTML5 web applications. However, for web developers, there is still a gap from the web browser to the server. Fortunately, this is the sweet spot for Vaadin.

Vaadin is a server-side Java web framework. And Java we need here. While web is mostly considered as JavaScript-driven, browser-based environment, in this case Java has its benefits - Vaadin nicely fills the gap between HTML5/JavaScript user interfaces and powerful native Java libraries.

Setting up computer vision for the Java server

The first thing to do is to find a server  to run OpenCV. Like mentioned, it is a native C++ library, but fortunately available for many operating systems - and provided with excellent getting started documentation.

I started from compiling the latest 3.0 (beta) for my target platform with Java bindings. The results are essentially a native .so library file and a jar file for JNI wrappers. You can find the build instructions here.

You need to install the  native library into your Java library path. This varies depending on your platform and JDK version, but on my Linux machine it was something like:

 

sudo cp ./lib/libopencv_java300.so \

 

    /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/

 

If you are like me, using Maven to manage library dependencies in Java development, it makes sense to install the jar into a local repository with the correct version. This way it is easy to integrate the library into your Java applications:

 

mvn install:install-file \

 

-Dfile=./bin/opencv-300.jar \

 

-DgroupId=org.opencv \  

 

-DartifactId=opencv \

 

-Dversion=3.0.0 \

 

-Dpackaging=jar

 

After the build process, you have everything you need to start developing the application itself.

Web application with face recognition

So,  now we are ready to create something. I got my first idea for a ‘computer vision web application’ from a mobile Android application that uses a face recognition algorithm to locate a face in a picture and make some fun of it: The trollator. Yes, this should be fun and easy enough also as a web application. :)

Now, while you can study the full web application source code in GitHub,  let me explain some parts of it.

In Vaadin, all starts with the user interface: the com.vaadin.ui.UI class. You extend this class to create your own user interface, which the user will see when accessing the application using a web browser.

In this application it is actually quite simple and it has only two parts: 1) a webcam component to take a picture and 2) an image component to show the processed result. The image is processed automatically when it has been taken. Better check the structure in the code.

Remember that loading the native OpenCV library has to be done early. Or at least before you use it. I used a static block that is executed when a class is loaded. You can see it in the beginning of the TrollatorUI class:

 

static {

 

    System.loadLibrary(Core.NATIVE_LIBRARY_NAME);

 

 }

 

Now it is available through the Java API and you can call the OpenCV methods in your application.

For the actual face recognition, I used the method based on Haar Cascade Classifiers. It is a very generic method for recognizing objects in pictures, and while you can create your own classifiers and train OpenCV to recognize any objects, OpenCV also conveniently ships with pre-trained classifiers for human faces.

In this faceDetect method the classifier is loaded from the file haarcascade_frontalface_default.xml and it is used to detect faces from the uploaded picture from a webcam. When matches are found, an overlay image is drawn on top of it.

OpenCV uses its own model for images called Mat and to map that to Java world, some conversion routines are needed. You can see that in toBufferedImage. Furthermore, to present the generated in-memory image to the user, I created a simple DynamicImage extension to Vaadin’s Image  component, to load the image data from java.awt.image.BufferedImage instead of normal file. Also note that in order to run the application, the JVM must be set to “headless” mode using -Djava.awt.headless=true system parameter.

Going further

Sorry, that’s it. :) I never got further than this with my computer vision apps, but I learned something, and hopefully this also write-up gives you an idea of how to get started.

It was pretty simple after all. All you need to do is: 1.) build OpenCV Java library and install it to local Maven repository and 2.) clone the sample project  from GitHub. And you have a full setup for the most amazing computer vision web applications of your own.

The Web Dev Zone is brought to you by Stormpath—offering a pre-built, streamlined User Management API for building web and mobile applications. Plan On Building User Management? Buy It Instead.Download Our White Paper To Learn More.