The k-NN algorithm

Mathematically speaking, the k-NN algorithm is one of the most simple machine learning algorithms out there. See the following diagram for a visual overview of how it works: 

The stars in the preceding diagram represent new data points. If we built a k-NN algorithm with three neighbors, then the stars would search for the three data points that are closest to it. 

In the lower-left case, the star sees two triangles and one circle. Therefore, the algorithm would classify the star as a triangle since the number of triangles was greater than the number of circles. 

In the upper-right case, the star sees two circles and one circle. Therefore, the algorithm will classify the star as a circle since the number of circles was greater than the number of triangles. 

The real algorithm does this in a very probabilistic manner and picks the category/shape with the highest probability.