Htykuut

Task: In the code cell below write code to run both the NLTK_Tokenise and your own Tokenise function on a sample of 10 sentences from the Reuters corpus.

I've got written the following code:

import pandas as pd

sample_size=10

r_list=



for sentence in rcr.sample_raw_sents(sample_size):

    r_list.append(sentence)



my_list = r_list



????

my_list=[i.split(tokenise) for i in my_list]

r_list=[i.split(nltk.tokenize) for i in r_list]    



pd.DataFrame(list(zip(my_list,r_list)),columns=["MINE","NLTK"])  

I have also considered (from just past the "????"):

my_list = [i.split() for i in my_list]

r_list = [i.split() for i in r_list]



tok = tokenise(my_list)

cortok = nltk.tokenize(r_list)



pd.DataFrame(list(zip(tok,cortok)),columns=["MINE","NLTK"])

Now I've got 2 lists with the same corpus information, and I want to apply my functions to said lists, though I can't figure out any way that allows me to apply functions rather than strings etc. Should I just copy & paste my tokenisers as strings, I'm sure there would be a better way to do this. For the second option I doubt I'll need the 2 separate lists and can tokenise the one list and attach it to new variables.

Further progress if anyone helps:

import pandas as pd

sample_size=10

r_list=



for sentence in rcr.sample_raw_sents(sample_size):

    r_list.append(sentence)



new_list = [i.split()[0] for i in r_list]



tok = tokenise(new_list)

cortok = nltk.tokenize(new_list)



pd.DataFrame(list(zip(tok,cortok)),columns=["MINE","NLTK"])  

What I think I want to do is separate the list into different variables to then make a DataFrame with a size of 10 (sample_size). Though I have no idea how to split a list of length into different variables unless I literally go 1,2,3,4,...,10 independently.

So I've gotten even further progress, I've realised I will have to use map():

import pandas as pd

sample_size=10

r_list=



for sentence in rcr.sample_raw_sents(sample_size):

    r_list.append(sentence)



tok = map(tokenise,r_list)

cortok = map(nltk.tokenize,r_list)



pd.DataFrame(list(zip(tok,cortok)),columns=["MINE","NLTK"])

Though something is still wrong with my final line. TypeError: 'module' object is not callable. I've googled it though still not entirely sure what the problem is. pandas has already been imported?

I've now realised I had a silly error where I input nltk.tokenize rather than word_tokenize.

Make use of map():

from nltk.tokenize import word_tokenize

import pandas as pd

sample_size=10

r_list=



for sentence in rcr.sample_raw_sents(sample_size):

    r_list.append(sentence)



tok = map(tokenise,r_list)

cortok = map(word_tokenize,r_list)



pd.DataFrame(list(zip_longest(tok,cortok)),columns=["MINE", "NLTK"])