Text must be represented as numerical columns when building a classification model using free text input, such as user reviews and comments. Text vectorization is the term for this method. In other words, using a series of numerical columns to represent text.
There are two main methods for doing this.
- Count Vectorization: is a text preprocessing method that creates a matrix of term frequency counts from a collection of text documents in natural language processing (NLP). This approach involves representing each document in the corpus as a row and each unique word as a column in the matrix. The matrix’s cell values show the frequency with which each word occurs in a given document. Text data may be easily formatted for use in a variety of natural language processing (NLP) activities, including text classification and clustering, by using a technique called count vectorization.
2. TF-IDF Vectorization: Term Frequency-Inverse Document Frequency, is another text preparation method frequently employed in NLP. It is a more sophisticated approach that considers a term’s significance over the whole corpus in addition to how frequently it occurs in a text. Every word in a document is given a weight by TF-IDF based on its term frequency—how often it appears in the text—and its inverse document frequency, which measures how uncommon it is throughout all the documents. This produces a matrix with values denoting the relative relevance of each phrase inside each document, with each document represented as a row and each term as a column.