Implementing Self-Organizing Maps with .NET Core

The code that accompanies this article can be downloaded here.

In the previous articles, we explored what Self-Organizing Maps are and how you can implement them using Python and TensorFlow. One of the most interesting things about these networks is that they utilize unsupervised learning, a different type of learning than we got a chance to see during our trip through the world of artificial neural networks. In this type of learning, neural networks don’t get the expected result during the training process. Instead, they figure out the relationship between input data on their own. Self-Organizing Maps use this approach for clustering and classification purposes and they are quite good at it.

Apart from that, we had to redefine the concepts of neurons, connection and weights. They have a different meaning in the Self-Organizing Maps world. Neurons are grouped into two collections. The first collection represents input neurons and their number corresponds to the number of features in the dataset.

The second collection represents output neurons, which are usually organized as one or two-dimensional arrays and are triggered only by certain input values. The location of each neuron is also important’ this is another thing that is different from the standard feed-forward artificial neural networks. Not only does each neuron have a location, but it is considered that neurons that lie close to each other have similar properties and actually represent a cluster.

Another interesting fact is that every input neuron is connected to every output neuron. This makes the learning process vastly different from what we are used to. These are the main steps of this process for Self-Organizing Maps:

Weight initialization
The input vector is selected from the dataset and used as an input for the network
BMU is calculated
The radius of neighbors that will be updated is calculated
Each weight of the neurons within the radius is adjusted to make them more like the input vector
Steps from 2 to 5 are repeated for each input vector of the dataset

You can check out one of the previous articles in which this process is explained in great detail.

While implementing Self-Organizing Maps with TensorFlow and Python was a lot of fun, I decided to do a similar thing with the C#. This way I am hoping to make these concepts more understandable to the .NET developers.

Prerequisites & Technologies

The solution itself is built using .NET Core and C#. It is created as a class library – SOM. Apart from that, unit tests are written using Xunit and Moq in SOMTests library. This is the list of technologies used for developing this solution:

.NET Core 2.1
C# 7.3
Xunit 2.4.0
Moq 4.10.0

Solution

The SOM library is divided into three big classes. Each of these classes controls a different aspect of the Self-Organizing Maps. If we take a look into the Self-Organizing Map structure, we can see that we need to model an input layer, an output layer and weighted connections. In addition to that, we need to coordinate the entire learning process using these entities. To sum up, these classes are a part of this solution:

Vector – Models all vectors in the system, like input neurons and weighted connections.
Neuron – Models one neuron in the output matrix.
SOMap – Models Self-Organizing Map itself.

It is important to note that this solution covers only the creation of two-dimensional Self-Organizing maps that receives a one-dimensional array as an input. So, let’s take a look at the implementations of different classes.

Vector

his class models all the vectors in the system. It covers the input layer, weighted connections and the input itself. It is modeled as a list of double values, with one extra functionality – EuclidianDistance. This function is calculating the Euclidean distance between two vectors. For testing purposes, we added the interface which looks like this:

	public interface IVector : IList<double>
	{
	double EuclidianDistance(IVector vector);
	}

view raw

IVector.cs

hosted with ❤ by GitHub

As you can see, this interface represents the integration of IList<double> interface with the additional functionality that we mentioned previously. The implementation of this interface can be found in Vector class:

	public class Vector : List<double>, IVector
	{
	public double EuclidianDistance(IVector vector)
	{
	if (vector.Count != Count)
	throw new ArgumentException("Not the same size");

	return this.Select(x => Math.Pow(x – vector[this.IndexOf(x)], 2)).Sum();
	}
	}

view raw

Vector.cs

hosted with ❤ by GitHub

In order to cover everything defined in IList<double> interface, this class inherits List<double> class and implements IVector interface. The first thing that implementation of the EuclidianDistance function does is a check of the sizes of two vectors. Vectors must have the same size, which means that they have to be in the same dimension if we want to calculate the Euclidean distance for them. We cannot do this on one two-dimensional and one three-dimensional array, and that is why this check is performed. After that, this function does the calculation itself.

Neuron

Quite obviously, this class is in charge of modeling one of the neurons from the output layer of the Self-Organizing Map. The objects of this class will represent one of the elements in the matrix. The neuron itself is described by the interface INeuron:

	public interface INeuron
	{
	int X { get; set; }
	int Y { get; set; }
	IVector Weights { get; }

	double Distance(INeuron neuron);
	void SetWeight(int index, double value);
	double GetWeight(int index);
	void UpdateWeights(IVector input, double distanceDecay, double learningRate);
	}

view raw

INeuron.cs

hosted with ❤ by GitHub

As you can see, every neuron has an X and a Y coordinate. These properties determine the position of the neuron in the matrix. Neurons that are close to each other have similar characteristics, which means that these properties are very important. Apart from that, every neuron contains Vector object for weighted connections. This object represents all the weighted connections that are attached to this neuron. Also, this interface describes a set of functions that each neuron should implement:

Distance – Calculates the distance from that neuron to another neuron in the matrix.
SetWeight – Sets a value to a weight defined by the index.
GetWeight – Retrieves a value of a weight defined by the index.
UpdateWeights – Updates weights of a neuron based on the input, learning rate and the distance decay.

The implementation of this interface is a part of the Neuron class and it looks like this:

	public class Neuron : INeuron
	{
	public int X { get; set; }
	public int Y { get; set; }
	public IVector Weights { get; }

	public Neuron(int numOfWeights)
	{
	var random = new Random();
	Weights = new Vector();

	for (int i = 0; i < numOfWeights; i ++)
	{
	Weights.Add(random.NextDouble());
	}
	}

	public double Distance(INeuron neuron)
	{
	return Math.Pow((X – neuron.X), 2) + Math.Pow((Y – neuron.Y), 2);
	}

	public void SetWeight(int index, double value)
	{
	if (index >= Weights.Count)
	throw new ArgumentException("Wrong index!");

	Weights[index] = value;
	}

	public double GetWeight(int index)
	{
	if (index >= Weights.Count)
	throw new ArgumentException("Wrong index!");

	return Weights[index];
	}

	public void UpdateWeights(IVector input, double distanceDecay, double learningRate)
	{
	if(input.Count != Weights.Count)
	throw new ArgumentException("Wrong input!");

	for(int i = 0; i < Weights.Count; i++)
	{
	Weights[i] += distanceDecay * learningRate * (input[i] – Weights[i]);
	}

	}
	}

view raw

Neuron.cs

hosted with ❤ by GitHub

In the constructor of this class, weights of the connections are initialized to random values. Distance method is implemented in a way that it returns the Euclidean distance between neurons in the matrix, based on the coordinates X and Y. The SetWeight and GetWeight check the index value within the range and after that, either set or retrieve the value of the weight. UpdateWeights method first checks if the input is in the proper dimension, and then it uses a formula defined in this article, to update weights on each connection of the neuron.

SOMap

Finally, let’s check out the implementation of the SOMap class. This class wraps other elements into a cohesive unity, utilizes them and implements the learning process on top of that. We tried to keep the exposed API similar to the one we can see in the TensorFlow implementation. Here is how that implementation looks:

	using System;
	using SOM.NeuronNamespace;
	using SOM.VectorNamespace;

	namespace SOM
	{
	public class SOMap
	{
	internal INeuron[,] _matrix;
	internal int _height;
	internal int _width;
	internal double _matrixRadius;
	internal double _numberOfIterations;
	internal double _timeConstant;
	internal double _learningRate;

	public SOMap(int width, int height, int inputDimension, int numberOfIterations, double learningRate)
	{
	_width = width;
	_height = height;
	_matrix = new INeuron[_width, _height];
	_numberOfIterations = numberOfIterations;
	_learningRate = learningRate;

	_matrixRadius = Math.Max(_width, _height) / 2;
	_timeConstant = _numberOfIterations / Math.Log(_matrixRadius);

	InitializeConnections(inputDimension);
	}

	public void Train(Vector[] input)
	{
	int iteration = 0;
	var learningRate = _learningRate;

	while (iteration < _numberOfIterations)
	{
	var currentRadius = CalculateNeighborhoodRadius(iteration);

	for (int i = 0; i < input.Length; i++)
	{
	var currentInput = input[i];
	var bmu = CalculateBMU(currentInput);

	(int xStart, int xEnd, int yStart, int yEnd) = GetRadiusIndexes(bmu, currentRadius);

	for (int x = xStart; x < xEnd; x++)
	{
	for (int y = yStart; y < yEnd; y++)
	{
	var processingNeuron = GetNeuron(x, y);
	var distance = bmu.Distance(processingNeuron);
	if (distance <= Math.Pow(currentRadius, 2.0))
	{
	var distanceDrop = GetDistanceDrop(distance, currentRadius);
	processingNeuron.UpdateWeights(currentInput, learningRate, distanceDrop);
	}
	}
	}
	}
	iteration++;
	learningRate = _learningRate * Math.Exp(–(double)iteration / _numberOfIterations);
	}
	}

	internal (int xStart, int xEnd, int yStart, int yEnd) GetRadiusIndexes(INeuron bmu, double currentRadius)
	{
	var xStart = (int)(bmu.X – currentRadius – 1);
	xStart = (xStart < 0) ? 0 : xStart;

	var xEnd = (int)(xStart + (currentRadius * 2) + 1);
	if (xEnd > _width) xEnd = _width;

	var yStart = (int)(bmu.Y – currentRadius – 1);
	yStart = (yStart < 0) ? 0 : yStart;

	var yEnd = (int)(yStart + (currentRadius * 2) + 1);
	if (yEnd > _height) yEnd = _height;

	return (xStart, xEnd, yStart, yEnd);
	}

	internal INeuron GetNeuron(int indexX, int indexY)
	{
	if (indexX > _width \|\| indexY > _height)
	throw new ArgumentException("Wrong index!");

	return _matrix[indexX, indexY];
	}

	internal double CalculateNeighborhoodRadius(double itteration)
	{
	return _matrixRadius * Math.Exp(–itteration/_timeConstant);
	}

	internal double GetDistanceDrop(double distance, double radius)
	{
	return Math.Exp(–(Math.Pow(distance, 2.0) / Math.Pow(radius, 2.0)));
	}

	internal INeuron CalculateBMU(IVector input)
	{
	INeuron bmu = _matrix[0, 0];
	double bestDist = input.EuclidianDistance(bmu.Weights);

	for (int i = 0; i < _width; i++)
	{
	for (int j = 0; j < _height; j++)
	{
	var distance = input.EuclidianDistance(_matrix[i, j].Weights);
	if( distance < bestDist)
	{
	bmu = _matrix[i, j];
	bestDist = distance;
	}
	}
	}

	return bmu;
	}

	private void InitializeConnections(int inputDimension)
	{
	for (int i = 0; i < _width; i++)
	{
	for (int j = 0; j < _height; j++)
	{
	_matrix[i, j] = new SOM.NeuronNamespace.Neuron(inputDimension) { X = i, Y = j };
	}
	}
	}
	}
	}

view raw

SOMap.cs

hosted with ❤ by GitHub

The only publicly exposed members of this class are the constructors and the Train method. The constructor is used to receive dimensions and initialize the output matrix, as well as to handle some of the initial values for learning rate and a number of iterations. The Train method is one interesting method which implements the unsupervised learning process of Self-Organizing Maps.

It is separated into several private methods that handle different aspects of this process. In essence, this method is one big loop that runs for a defined number of iterations. In each iteration, the first thing that is done is a calculation of neighborhood radius. This is done in CalculateNeighborhoodRadius method. This is driven by the number of iterations that had been run so far as well as the total number of iterations and dimensions of the output matrix.

After this number is calculated, the BMU is determined for each input vector. For this purpose, CalculateBMU method is used. Afterward, the indexes of the neurons that are within radius are calculated. For each neuron within this radius, the distance from the BMU is calculated, which is used for calculating the new value for the distance decay. Finally, this value, along with the value of the learning rate is used to update the weights on each connection of the neuron. At the end of the Train method, the learning rate is updated after each iteration.

Using the Library

The API of this library is pretty simple and straightforward. All you have to do is model your input as Vector objects, create a SOMap object and call the Train method – something along these lines:

	var inputVector = new Vector();
	inputVector.Add(2);
	inputVector.Add(2);

	var input = new Vector[10]
	{
	inputVector,
	inputVector,
	inputVector,
	inputVector,
	inputVector,
	inputVector,
	inputVector,
	inputVector,
	inputVector,
	inputVector
	};


	var som = new SOMap(2, 2, inputVector.Count, 100, 0.5);
	som.Train(input);

view raw

SOMUseCase.cs

hosted with ❤ by GitHub

First, the test input vector is created and a complete input dataset is created by repeating that value. These are just test values, used as an example. Then a 2×2 Self-Organizing Map is created using SOMap constructor. We also define that training will take 100 iterations and that the start learning rate should be 0.5. After that, the Train method is called with input we generated at the beginning.

Conclusion

The goal of this article was to make the concepts of Self-Organizing Maps understandable to .NET developers. Apart from being a highly fun experience and experiment, I hope that some developers will benefit from the concepts explored here. We were able to see how to model and implement the main parts of the Self-Organizing Maps, such as neurons and vectors, and how to implement the unsupervised learning process. If you want to explore this field even further, now you can use a library that we developed here. In the next article, we will see discover how to solve a real-world problem using Self-Organizing Maps.

Thank you for reading!

This article is a part of Artificial Neural Networks Series, which you can check out here.

Read more posts from the author at Rubik’s Code.

Trackbacks/Pingbacks

The Morning Brew - Chris Alcock » The Morning Brew #2666 - […] Implementing Self-Organizing Maps with .NET Core – Nikola Živković […]
Szumma #128 – 2018 38. hét – ./d/fuel - […] Implementing Self-Organizing Maps with .NET Core […]