Htykuut

I've got a problem with the output of Tfidf Vectorizer and I've test many solutions given in other topics and nothing works.

I have a csv with two columns : one column test containing ... text and a column score.
And I want to be able to predict a new score based on a text I will be able to input.
I think the better solution is to use a linear regression based on tfidf analys on the text.

My code is the following :

datas = pandas.read_csv('Data/gucci-account- 

prediction.csv',delimiter=';')

datas['score'] = datas['retweets'] + datas['likes']

import re



def tokenizer(text):

    if text:

        result = re.findall('[a-z]{2,}', text.lower())

    else:

        result = 

    return result



X = datas['text'].values

y = datas['score'].values

vect = TfidfVectorizer(tokenizer=tokenizer,stop_words='english',dtype=np.float32)

X_train = vect.fit_transform(X)

lr = Ridge(alpha=1.0)

lr.fit(X_train,y)

And I have the following error : Input contains NaN, infinity or a value too large for dtype('float64').

I already verified and my dataframe ( before vectorization ) contains no nan value so I don't understand why my X matrix would contain any nan or infinite value.

Would you have a solution so it works ? Thank you