---

The code that accompanies this article can be downloaded here.

---

Couple of months back we investigated parts of TensorFlow’s ecosystem beyond standard library. To be more precise, we investigated TensorFlow.js and how you can build and train models in the browser and/or in the Node.js. However, we didn’t manage one important topic – integration. What we want to cover in this article is what happens when you have a neural network built and trained using TensorFlow and Python, and you have to integrate it in one Angular application.

This article should emphasize, how processes of building any machine learning model differs from the developing application that utilizes this model. However, it also reveals mutual points as well and provide further evidence how data scientists and software developers should work close together if they want to create a great product. Let’s describe the application we want to build.

MNIST TensorFlow/Angular Application

What we want to do is create an application in which user can write down a number in some sort of the canvas on the web page and application will recognize which number is written. For that we will use MNIST data set. This dataset is ‘Hello World‘ example in the world of Machine Learning. It contains 60000 images of handwritten digits. Images are centered and all the same size – 28×28. This means that additional processing on these images should be minimal. Another cool thing about this data set is that it is available as a part TensorFlow library.

Since we are working with images, we need to use Convolutional Neural Networks for this solution. You can find more on the theory of this type of neural networks here. In a nutshell, they are using several layers to detect features on the images and then they use traditional feed-forward neural networks to classify images based on those features. Detection of the features of the image is done by convolutional layers.

These layers first detect out low-level features, like edges and curves, and after that  they detect higher level features, like a face, or a hand, or in our case, hand-written digit. After that is done Convolutional Neural Networks use additional layers to remove linearity from the image, something that could cause overfitting. When linearity is removed, additional layers for compressing the image and flattening the data are used. Finally, this information is passed into a neural network, called Fully-Connected Layer in the world of Convolutional Neural Networks.

Once this model is created we want to save it and use it inside of well known JavaScript framework – Angular. As you can see, the goal of this whole blog post is to go through the several technologies and more importantly through the several parts of TensorFlow ecosystem. In this journey we will use TensorFlow, TensorFlow.js, Python, TypeScript and Angular. Sounds exciting doesn’t it.

Prerequisites

Let’s first install all necessities for this blog post. For building and training Neural Network we have to install Python 3 our local machine. In this example, to be more specific, we are using Python 3.7. The easiest way to do that is to use Anaconda distribution. It comes with Jupyter Notebook IDE as well. The implementation itself is done using TensorFlow 2.0 library. The complete guide on how to install and use Tensorflow 2.0 can be found here. Apart from that we need to install Tensoflow.js for the Python end. Soon it will be revealed why do we have to do this, but for now, make sure that you have run the command:

pip install tensorflowjs

The second section of this article requires to install Node.js. Node.js provides a non-blocking event-driven system for our applications, and it is one of the fastest growing technologies at the moment. Its asynchronous mechanism gives us an ability to handle complex web scenarios in an easier manner. Apart from that, it has its own packaging ecosystem npm. Though this ecosystem one is able to install other libraries that will be used in this example. First one should be TensorFlow.js. That can be done like this:

npm install -g @tensorflow/tfjs

Another package that we will definitely need is http-server:

npm install -g http-server

For the web application, we use Angular. The most common way to manipulate with this framework is to use Angular Command Line Interface – Angular-CLI. One of the benefits of this tool is that, once we initialize our application with it, we can use TypeScript and it will be automatically translated to JavaScript. Installing this interface is done through npm, of course, by running the command:

npm install -g angular-cli

Next thing we want to do is to initialize our Angular application. This is done by calling Angular-CLI command:

ng new application_name

This command will create a folder structure that will be used by our application. To run this application, position shell in the just created root folder of your application (cd application_name), and call command:

ng serve

If you followed the previous steps once you go to your browser and open localhost:4200 you will be able to see something like this:

That is cool, now we have running Angular application. Later in this article we will modify this code that is generated to serve our purpose. For now we have a starting point.

Building Convolution Neural Network

The first step in this process is creating a neural network using Python and TensorFlow high-level API – Keras. For this we are using Python and Jupyter Notebook IDE. Let’s start with importing necessary modules:

As you can see, Keras modules like Sequential, Dense, Conv2D, MaxPoolling and Flatten are used. Sequential class creates placeholder in which we can enter layers of neurons. Since we are building Convolutional Neural Network, we follow explanation provided in this article, and add convolutional layers first, that is why Conv2D class is used. Then we follow it by adding MaxPolling and Flatten layers. Finally we will add a couple of Dense layers to create feed-forward neural network in the end. This is how that looks like:

In the end we compiled the neural network, meaning we connected all those layers that we have put into Sequential class and now neural network is ready for training. For that, we need to load MNIST data set and normalize the data:

Here we utilize mnist module that we imported from tensorflow.keras.datasets in the beginning. We are able to load train and test data. Train data is used during the training of the neural network, while test data is used to evaluate the model and give us it’s accuracy. Also, they are split into input data – images and output data – labels. When we are talking about the images in computer science, we consider them to be matrices of values. Every image is containing three matrices (RGB) with values for each color. In this particular case, images are black and white, so we have only one matrix.

Once this is done we can run the training process:

Output of this process looks something like this:

And then evaluate the model:

We got the accuracy of 99.04%. This means that in 99% of the cases, neural network will be able to detect digit written in the image. That is awesome! We can even check that by running single predictions:

The output looks something like this:

Finally, we have to save our model in a file:

Note that saved model is in the file type .h5. We need to convert this file into .json file if we want to use it in any JavaScript framework.

Converting Model

Now, we can slowly move from the Python realm into JavaScript universe. We need to convert the model from .h5 and for this we needed to install TensorFlow.js as a Python module. To be more specific, we need to use tensorflow_converter tool to make model that is usable inside of Angular application. We need to position into directory where model.h5 file is located and run command:

tensorflowjs_converter --input_format keras ./model.h5  ./trained_model

Once this process is done, you will see several files in the newly created trained_model folder:

In order to load this inside of Angular application, we need to run server that serves this file. To do that, we need to run already installed http-server. So, we position into trained_model directory and we run command:

http-server -p 3000 --cors

This will do the trick. We now have running server on the port 3000 and we can finally load the model inside of the Angular application.

Angular Application

In the folder where Angular application is created make sure you have run the commands:

npm install -g @tensorflow/tfjs
npm install
ng serve

In order to simplify this implementation, we just modify app.component, so we don’t need to have some additional setup of the application itself. Let’s start from the HTML file:

It is pretty simple. We have a toolbar, dynamically loaded title, canvas, button and “predicted section” for presenting the results. Essentially, user writes down something on the canvas and we display the number that is written down in the “predicted section”. Also, user can clear the canvas by pressing the Clear button. Here is how that looks like:

Ok, now let’s observe TypeScript file – app.component.ts:

That is a lot of code, so let’s note some important parts. First it is important to notice that TensorFlow.js is imported. Apart from that note that OnInit and AfterViewInit are implemented. That is why we need to have methods ngOnInit and ngAfterViewInit. In the first one we initialize whole application by loading the model. This is done using loadLayersModel function from TensorFlow.js:

During this initialization, the model summary is printed out in the console. This is a nice way to see the structure of the model:

In the ngAfterViewInit method, we initialize the canvas, so user can write down the digits. Notice that we connected that HTML element with the captureEvents method. In this method, we capture two types of events. First one is mousedown. When this event on canvas is detected, we need to handle drawing. That is done with the help of drawOnCanvas method. Second event that we are detecting is mouseup. That is more interesting event for us. 

Here we get the image that is drawn on the canvas and convert it into tensor. This is performed using getImage method:

After image is retrieved and transformed into proper type we can call predict function from the model. It will return probabilities for each digit. Finally, we filter those results and update the dynamic parts:

So, we get really cool results in the end. Check it out:

Conclusion

In this article, we covered quite a lot of ground. First we have built the neural network using Python and saved it. Then we learned how we can convert that model into something that can be used by JavaScript framework. Then we have built simple Angular application that utilized this mode. Basically, we can see how everything fits together if your data scientists and web developers work together. Now, this was rather illustrative example, but imagine this with some real-world data that can bring a lot of benefit to your users.

Thank you for reading!

---

Read more posts from the author at Rubik’s Code.

---