MACHINE LEARNING IN PRACTICE – QUORA COMPETITION

Computers, as calculating machines, have optimized the data analysis process very effectively, thus replacing a traditional piece of paper and a pen. Nowadays, the growth of information and the ability to store huge volumes of data have put the machines in the first place when it comes to exploring hundreds petabytes of information.

Today’s data storage process is characterized by one of the biggest growth rates, and the forecasts put forward by experts indicate that this process will continue to grow and that the total size of electronic data stored in the world in 2020 will exceed 44ZB (in 2004 it was only 4.4ZB source).

In this dynamic increase in various and gigantic data volumes, one of the major problems engineers have to deal with is the way of storing and managing this data.

On the other hand, expert opinions indicate that in the future, data will be one of the most valuable resources for humans. If we think of electronic data as another precious resource, such as gold, then we think of hard-working miners looking for gold in petagrams of useless land. This directs us to the idea of extracting valuable information from petabytes of worthless data stored on various memory disks.

For simplification, the further part of the article uses phrases “document d” to refer to the level representing the vector and “w words” reflecting the vector elements. Therefore, the use of the above naming allows the following description of our model: “Every document d_i (i-1,…,M where M is the number of analysed documents) is represented by the vector of N words w_j (j=1,…,N where N is the number of unique words).”

Another issue is determining the weight of individual words. When analyzing text sets, we need to ask ourselves: “Which shared words are important for the similarity of texts?” This is a fundamental question that every analyst should be asking. We can distinguish several ways of determining word weights:

Unfortunately, the logarithmic function, just like the linear one, does not guarantee the possibility of bounding the functions above, so they approach infinity along with the increase of frequency of occurrence of words. This problem begins to be significant when a very large group of words occur in a document and take over the majority of the vector’s weight.

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

import pandas as pd

import numpy as np

df = pd.DataFrame({ 'q1' : ['How do I become a data scientist in the data scientist team',

                            'What are the questions should not ask on Quora?​',

                             'What are the important algorithms in computer science',

                            'Can I call you tonight'], 

                     'q2' : ['How can I become a data scientist​',

                            'Which question should I ask on Quora​', 

                             'What is the importance of algorithm in computer science?',

                            'Please, could you call me? PLEASE'] })

np_list = np.concatenate((df.q1, df.q2))

Generating a TF word frequency array, based on the word count of a given question:

vect = CountVectorizer()
tf = pd.DataFrame(vect.fit_transform(np_list).toarray(), columns=vect.get_feature_names())

Generating a DF vector that represents the instances of individual words within all the questions in the set. TF functionality for bit maps was applied here:

vect = CountVectorizer(binary=True)
df = vect.fit_transform(np_list).toarray().sum(axis=0)
df_result = pd.DataFrame(df.reshape(1, len(vect.get_feature_names())), columns=vect.get_feature_names())

The last table shows the use of the second vectorizer that calculates the TF-IDF weight:

vect = TfidfVectorizer()
tfidf = pd.DataFrame(vect.fit_transform(np_list).toarray(), columns=vect.get_feature_names())

The above-mentioned operations calculate the basic vector models for tested sentences. In the final stage, we wanted to compare question vectors for individual pairs. In this case, we used TF-IDF vectors, and the cosine measure described in the following paragraph was used as the measure of similarity:

from scipy import spatial
df_1 = tfidf.iloc[:4, :]
df_2 = tfidf.iloc[4:, :]
result = []
for i in range(4):
 result.append(1-spatial.distance.cosine(np.array(df_1.iloc[i,:]), 
 np.array(df_2.iloc[i,:])))

Performing the correct vectorization and then extracting similarities between the selected pairs of questions on the basis of the received vectors was of great importance in the process of building the competition classifier. An important stage was appropriate pre-processing, which was the initial phase in our workflow. It was at this stage where we had a big influence on the unique “keywords” of every vector.

The following actions had the greatest impact on the final results of the competition: