Let’s talk about one of the most fascinating parts of machine learning - data vectorization.
At first look, it seems like ML models work directly with the raw data we provide. But actually, it doesn't. ML algorithms need a numerical representation of data. Specifically, they require data in the form of floating-point values called feature vector.
However, many features are naturally strings or other non-numerical values. The task is to transform these non-numerical values into numerical ones. That's the main purpose of feature engineering discipline.
Let me illustrate that on vectorization of categorical data.
Categorical data consists of specific, predefined values, like car colors, animal species, days of the week, or city street names. It can be low-dimensional (few possible values) or high-dimensional (many possible values).
For low-dimensional data, we can encode it as a vocabulary. Let’s use car colors as an example with 5 categories for simplicity: white, blue, red, black, and others (for any color not in the list).
To create a vector the following steps should be done:
1. Index each value:
- 0: white
- 1: blue
- 2: red
- 3: black
- 4: others
2. Represent each category as a vector (array) of N elements, where N is the number of categories.
|Feature|White|Blue|Red|Black|Others|
|------------|---------|-------|------|--------|---------|
|White | 1 | 0 | 0 | 0 | 0 |
|Blue | 0 | 1 | 0 | 0 | 0 |
|Red | 0 | 0 | 1 | 0 | 0 |
|Black | 0 | 0 | 0 | 1 | 0 |
|Others | 0 | 0 | 0 | 0 | 1 |
3. Convert to floating point values: replace 1 with 1.0 and 0 with 0.0. For example, the vector for blue would be
(0.0, 1.0, 0.0, 0.0, 0.0)Of course, that's very basic example to illustrate the concept. Real-world cases often involve more complex transformations and math models, depending on the data and problem.
More details and vectorization strategies:
- Working with numerical data
- Working with categorical data
- Datasets, generalization, and overfitting
#aibasics