The code that accompanies this article can be downloaded here.
Couple of months back we investigated parts of TensorFlow’s ecosystem beyond standard library. To be more precise, we investigated TensorFlow.js and how you can build and train models in the browser and/or in the Node.js. However, we didn’t manage one important topic – integration. What we want to cover in this article is what happens when you have a neural network built and trained using TensorFlow and Python, and you have to integrate it in one Angular application.
This article should emphasize, how processes of building any machine learning model differs from the developing application that utilizes this model. However, it also reveals mutual points as well and provide further evidence how data scientists and software developers should work close together if they want to create a great product. Let’s describe the application we want to build.
MNIST TensorFlow/Angular Application
What we want to do is create an application in which user can write down a number in some sort of the canvas on the web page and application will recognize which number is written. For that we will use MNIST data set. This dataset is ‘Hello World‘ example in the world of Machine Learning. It contains 60000 images of handwritten digits. Images are centered and all the same size – 28×28. This means that additional processing on these images should be minimal. Another cool thing about this data set is that it is available as a part TensorFlow library.
Since we are working with images, we need to use Convolutional Neural Networks for this solution. You can find more on the theory of this type of neural networks here. In a nutshell, they are using several layers to detect features on the images and then they use traditional feed-forward neural networks to classify images based on those features. Detection of the features of the image is done by convolutional layers.
These layers first detect out low-level features, like edges and curves, and after that they detect higher level features, like a face, or a hand, or in our case, hand-written digit. After that is done Convolutional Neural Networks use additional layers to remove linearity from the image, something that could cause overfitting. When linearity is removed, additional layers for compressing the image and flattening the data are used. Finally, this information is passed into a neural network, called Fully-Connected Layer in the world of Convolutional Neural Networks.
Let’s first install all necessities for this blog post. For building and training Neural Network we have to install Python 3 our local machine. In this example, to be more specific, we are using Python 3.7. The easiest way to do that is to use Anaconda distribution. It comes with Jupyter Notebook IDE as well. The implementation itself is done using TensorFlow 2.0 library. The complete guide on how to install and use Tensorflow 2.0 can be found here. Apart from that we need to install Tensoflow.js for the Python end. Soon it will be revealed why do we have to do this, but for now, make sure that you have run the command:
pip install tensorflowjs
The second section of this article requires to install Node.js. Node.js provides a non-blocking event-driven system for our applications, and it is one of the fastest growing technologies at the moment. Its asynchronous mechanism gives us an ability to handle complex web scenarios in an easier manner. Apart from that, it has its own packaging ecosystem npm. Though this ecosystem one is able to install other libraries that will be used in this example. First one should be TensorFlow.js. That can be done like this:
npm install -g @tensorflow/tfjs
Another package that we will definitely need is http-server:
npm install -g http-server
npm install -g angular-cli
Next thing we want to do is to initialize our Angular application. This is done by calling Angular-CLI command:
ng new application_name
This command will create a folder structure that will be used by our application. To run this application, position shell in the just created root folder of your application (cd application_name), and call command:
If you followed the previous steps once you go to your browser and open localhost:4200 you will be able to see something like this:
That is cool, now we have running Angular application. Later in this article we will modify this code that is generated to serve our purpose. For now we have a starting point.
Building Convolution Neural Network
The first step in this process is creating a neural network using Python and TensorFlow high-level API – Keras. For this we are using Python and Jupyter Notebook IDE. Let’s start with importing necessary modules:
As you can see, Keras modules like Sequential, Dense, Conv2D, MaxPoolling and Flatten are used. Sequential class creates placeholder in which we can enter layers of neurons. Since we are building Convolutional Neural Network, we follow explanation provided in this article, and add convolutional layers first, that is why Conv2D class is used. Then we follow it by adding MaxPolling and Flatten layers. Finally we will add a couple of Dense layers to create feed-forward neural network in the end. This is how that looks like:
In the end we compiled the neural network, meaning we connected all those layers that we have put into Sequential class and now neural network is ready for training. For that, we need to load MNIST data set and normalize the data:
Here we utilize mnist module that we imported from tensorflow.keras.datasets in the beginning. We are able to load train and test data. Train data is used during the training of the neural network, while test data is used to evaluate the model and give us it’s accuracy. Also, they are split into input data – images and output data – labels. When we are talking about the images in computer science, we consider them to be matrices of values. Every image is containing three matrices (RGB) with values for each color. In this particular case, images are black and white, so we have only one matrix.
Once this is done we can run the training process:
Output of this process looks something like this:
And then evaluate the model:
We got the accuracy of 99.04%. This means that in 99% of the cases, neural network will be able to detect digit written in the image. That is awesome! We can even check that by running single predictions:
The output looks something like this:
Finally, we have to save our model in a file:
tensorflowjs_converter --input_format keras ./model.h5 ./trained_model
Once this process is done, you will see several files in the newly created trained_model folder:
In order to load this inside of Angular application, we need to run server that serves this file. To do that, we need to run already installed http-server. So, we position into trained_model directory and we run command:
http-server -p 3000 --cors
This will do the trick. We now have running server on the port 3000 and we can finally load the model inside of the Angular application.
In the folder where Angular application is created make sure you have run the commands:
npm install -g @tensorflow/tfjs npm install ng serve
In order to simplify this implementation, we just modify app.component, so we don’t need to have some additional setup of the application itself. Let’s start from the HTML file:
It is pretty simple. We have a toolbar, dynamically loaded title, canvas, button and “predicted section” for presenting the results. Essentially, user writes down something on the canvas and we display the number that is written down in the “predicted section”. Also, user can clear the canvas by pressing the Clear button. Here is how that looks like:
Ok, now let’s observe TypeScript file – app.component.ts:
That is a lot of code, so let’s note some important parts. First it is important to notice that TensorFlow.js is imported. Apart from that note that OnInit and AfterViewInit are implemented. That is why we need to have methods ngOnInit and ngAfterViewInit. In the first one we initialize whole application by loading the model. This is done using loadLayersModel function from TensorFlow.js:
During this initialization, the model summary is printed out in the console. This is a nice way to see the structure of the model:
In the ngAfterViewInit method, we initialize the canvas, so user can write down the digits. Notice that we connected that HTML element with the captureEvents method. In this method, we capture two types of events. First one is mousedown. When this event on canvas is detected, we need to handle drawing. That is done with the help of drawOnCanvas method. Second event that we are detecting is mouseup. That is more interesting event for us.
Here we get the image that is drawn on the canvas and convert it into tensor. This is performed using getImage method:
After image is retrieved and transformed into proper type we can call predict function from the model. It will return probabilities for each digit. Finally, we filter those results and update the dynamic parts:
So, we get really cool results in the end. Check it out:
Thank you for reading!
Read more posts from the author at Rubik’s Code.