Statistical methods can and are applied to qualitative data (e.g. text). The raw data first need to be converted to a form, or represented in a way, that can be quantitatively analyzed.

It can be done by man or, increasingly, machine. This would be the case whether cluster analysis, factor analysis, deep learning or some other method is used to analyze the data.

A very simple example is cluster or factor analyzing open end codes from a consumer survey, or using them in key driver regression.

This predates AI by many years, though machine coding would now be the preferred approach in many instances.

Sparse data can be a concern, but with rapid declines in the cost of survey data collection, this is now more feasible since very large samples can be collected at reasonable cost and within a realistic time frame.

To be clear, my original motivation for this post was not to push an alternative to standard close-ended questioning, but a response to confusion about AI I frequently encounter. However, with data of sufficient quality, complex analytics which tie attitudes, behavior and demographics together, perhaps combined simultaneously with segmentation, is possible with this alternative method. AI in some form may play a part but is not absolutely essential. The basic idea goes back nearly a century.

✴️ @AI_Python_EN