Encoding categorical data

The trick to encode categorical data is to expand categorical data into multiple columns, each having a 1 or 0 representing whether it's true or false. This of course comes with some caveats and subtle issues that must be navigated with care. For the rest of this subsection, I shall use a real categorical variable to explain further.

Consider the LandSlope variable. There are three possible values for LandSlope:

• Gtl
• Mod
• Sev

This is one possible encoding scheme (this is commonly known as one-hot encoding):

This would be a terrible encoding scheme. To understand why, we must first understand linear regression by means of ordinary least squares. Without going into too much detail, the meat of OLS-based linear regression is the following formula (which I am so in love with that I have had multiple T-shirts with the formula printed on):

Here,is an(m x n) matrix andis an (m x 1) vector. The multiplications, therefore, are not straightforward multiplications—they are matrix multiplications. When one-hot encoding is used for linear regression, the resulting input matrixwill typically be singular—in other words, the determinant of the matrix is 0. The problem with singular matrices is that they cannot be inverted.

So, instead, we have this encoding scheme:

Here, we see an application of the Go proverb make the zero value useful for being applied in a data science context. Indeed, clever encoding of categorical variables will yield slightly better results when dealing with previously unseen data.

The topic is far too wide to broach here, but if you have categorical data that can be partially ordered, then when exposed to unseen data, simply encode the unseen data to the closest ordered variable value, and the results will be slightly better than encoding to the zero value or using random encoding. We will cover more of this in the later parts of this chapter.