This notebook is a reviewed English version (2020) of “03.1b-Ley-de-Luhn.ipynb” on the collection nlp_pydata2018 on GitHub.

Advanced Preprocessing: Luhn Eg.

This section is an advanced section on NLP and its basic processes. To reach the goal of this notebook you must know how to filter the stopwords which are words without semantic content and the most frequent ones, also how to steamm word suffixes. In any of those cases, statistics are essential for the algorithm or for the learning mechanism to be used, and in every step we can see how the number of words is reduced trying to find the most important ones.

Let’s now look at an example of NLP very close related to the field of Information Retrieval. It is all about Luhn’s Law [2], or the problem of finding the boundaries of the most important words in a document, usually for long texts like books.

Roadmap Summary

Datasets

For all the examples, two books will be used, one in English and the other in Spanish. One if the translation of the other, and both are under Creative Common license. The first is Free Culture and the second is its Spanish translation Cultura Libre.

Transforming Dataset Texts

Generally, almost all the materials that we have are PDFs and to operate with texts in python it is best to use simple text formats like .txt. Then, how to transform PDF into TXT?

Our recommendation is to use pypdf2 that appears in the GNU / Linux repositories and it is a pure python library (see Text Extraction with pdfMiner). If you know any script suitable for this task using the ghostscript library, we recommend it over pdftotext. However it is quite difficult to find such a custom script for our needs.

The text-preproc library since v0.2 includes a helper function that wraps the pypdf2 library to extract the text.

from preprocess.utils.io import pdf
PDF = pdf('/path/to/pdf/file.pdf')
text = PDF.extractText()
with open('/path/for/new/file.txt','w') as doc:
    doc.write(text)

If you are using preprocess v0.1 use pdftotext included in the poppler-utils package.

~$ pdftotext file.pdf

Result: file.txt

Keep in mind that text extraction scripts generate .txt files with many problems: mainly rare characters. Perhaps it will be more useful for the reader to study more specialized applications and libraries for this type of word processing problem such as: Apache-Tika, among others. However, in the following sections Transforming text some tips can be found out about how to solve these problems without using large libraries to obtain good result in basic NLP subprocess.

Transforming Text

The next codes transform text to allow the application of statistical analysis to build Luhn model.

import preprocess

from preprocess.data import load_freeculture
from preprocess.demo import preProcessFlow
text = load_freeculture()
text = preProcessFlow(text).lower()
text = preprocess.del_digits(text)

from nltk.tokenize import RegexpTokenizer
tokenizer = RegexpTokenizer("\s+", gaps=True)
tokens = tokenizer.tokenize(text)
print ("Total words: ", len(tokens))

Total words:  120303

En ambos casos se devuelve el total de palabras(o tokens) divididos por el caracter espacio. Sin embargo las palabras en un texto se repiten. ¿Cómo saber las palabras únicas?

tokens_unique=set([])
tokens_unique = set(tokens)
print ("Unique words:", len(tokens_unique))

Unique words: 8233

As it can be seen in the book of Bird et al.[1] the simplest question we can make when we see these numbers are: Which is the average of words by page? Which is the must used or frequent word? What words are used only one time? Lets see some ways to calculate that, to get those goals will be needed some extra functions.

#Start a dict to keep the frequence of every word
dict = {}
for word in tokens_unique:
    dict[word]=0
#Dict with word = frequence of word.
for token in tokens:
    dict[token]+=1
#A list of tuples with more info can result in a better resource. List([(Frequence1, word1),(freq2,word2),...])
tupla = []
for word in dict:
    tupla.append([dict[word],word])

This tuple can be ordered like this: because the frequence is in the first position, the list tupla can be sortered using tupla[i][0].

tupla=sorted(tupla)
print ("The 10 must frequent words are:")
for i in range(1,11):
    print (tupla[-i][1],":",tupla[-i][0])

The 10 must frequent words are:
. : 9178
the : 6909
of : 3479
to : 3180
a : 2408
is : 2198
that : 2052
and : 1922
in : 1718
it : 1276

As can be seen the words are non useful words, or stopwords or the puntuaction sign.

import time
from nltk.corpus import stopwords
timei = time.time()
english_stops = set(stopwords.words('en'))
#This updates are agregated after some experimental observation, because
#normalization process required more analysis time
english_stops.update(['would','pm','pages','page','th', '_r', 'could','pass','jm_qxd','❚❘','one'])
print(english_stops)

{'some', 'their', 'again', 'itself', 'where', 'pages', 'should', 'while', 'so', 'into', 'me', 'any', 'same', 'with', 'did', 'herself', 'which', 'why', 'ours', 'just', 'were', 'through', 'each', 'being', 'by', 'our', 'its', 'than', 'we', 'it', 'after', 'between', 'would', 'off', 'could', 'don', 'an', 'both', 'over', 'to', 'under', 'more', 'not', 'once', 'having', 'what', 'before', 'one', 'do', 'yourself', 'out', 'this', 'about', 'my', 'his', 'yourselves', 'does', 'because', 'against', 'myself', 'they', 'up', 'here', '_r', 'for', 'nor', 'most', 'jm_qxd', 'yours', 'doing', 'if', 'those', 'all', 'pass', 'when', 'that', 'who', 'of', 'are', 'above', 'them', 'ourselves', 'during', 'be', 'on', 'down', 'themselves', 'only', 'and', 'can', 'these', 'theirs', 'such', 'how', 'but', 'in', 'have', 'your', 'very', 'no', 'hers', 'she', 'whom', 'further', 'i', 's', 'is', 'below', 't', 'the', 'until', 'am', 'few', 'now', 'her', 'will', 'too', 'a', 'from', 'has', 'himself', 'he', 'own', 'you', 'been', 'other', 'him', 'had', 'at', 'page', 'was', 'there', 'pm', '❚❘', 'as', 'then', 'or', 'th'}

tokens_afterstops=[]
for k in range(len(tokens)-1):
    if tokens[k] not in english_stops and len(tokens[k])>1:
        tokens_afterstops.append(tokens[k])
timef = time.time()-timei
print ("Stopword filtering time: ",timef)

tokens_unique1 = set(tokens_afterstops)
dict1 = {} #dict con keys = set de tokens after stops
for word in tokens_unique1:
    dict1[word]=0

tupla1 = [] #Creating list of tuples (frequence,word) without stopwords
for token in tokens_afterstops:
    dict1[token]+=1
for word in dict1:
    tupla1.append([dict1[word],word])
tupla1=sorted(tupla1)

print ("Uunique words without stopw:", len(tokens_unique1))
print ("The 10 must frequent words after stopwords filtering are:")
for i in range(1,11):
    print (tupla1[-i][1],":",tupla1[-i][0])
print ("Total words: ", len(tokens))
print ("Total text words without stopwords:", len(tokens_afterstops))
print ("Deleted words after stopwords filtering:",
len(tokens)-len(tokens_afterstops))

Stopword filtering time:  2.2512850761413574
Uunique words without stopw: 8071
The 10 must frequent words after stopwords filtering are:
copyright : 720
law : 615
free : 441
culture : 374
property : 356
content : 281
right : 246
work : 242
use : 239
internet : 238
Total words:  120303
Total text words without stopwords: 55482
Deleted words after stopwords filtering: 64821

Luhn Model in Linguistics

Luhn argues that long texts have a logaritmic function behavior or a long tal function. Where the stopwords are at the beginning and the less frequent words are in the long tail. Lets see a graph of the book Free Culture. Using the variable tupla the 2.5 section code a numpy array is built and the media() and the variance() are extrated to be used in future calculations.

%pylab inline

Populating the interactive namespace from numpy and matplotlib

xarray = []
t=numpy.linspace(0,len(tupla),num=len(tupla))
for i in range(len(tupla)):
    xarray.append(tupla[i][0])
xarray.reverse()

x = numpy.array(xarray,dtype=numpy.int16)
mux = x.mean()
sigmax = x.std()
print (mux)
print (sigmax)
print (x.shape)

plt.plot(t[10:400],x[10:400])
plt.title("Luhn Distribution Book Free Culture")
plt.show()

14.612291995627354
153.12755040987275
(8233,)

Generate a normal distribution based on Zeta.

This step is personal proposal because of the dark in the original Luhn article [2] with respect to the boundaries for cutting. Some papers consider that in the obtention of these limits there is certain empirism. Calcs are made based on a nule hypotesis. Keith web book, chapter2

sigma = x_s.std()
mu = x_s.mean()+3*sigma

pdf = (1/(sigma * numpy.sqrt(2 * numpy.pi))
* numpy.exp( - (t - mu)**2 / (2 * sigma**2) ))

perc_05 = int(mu-2*sigma)
perc_95 = int(mu+2*sigma)
print(perc_05, perc_95)
print (mu, sigma)

113 427
270.686232489232 78.46649164766025

x_s = x[(x > 5) & (x < 1000)]
print(x_s.shape)
print(x[113]/(x.max()*pdf[113]))
print(x[427]/(x.max()*pdf[427]))
factor = x[113]/(x.max()*pdf[113]) - x[427]/(x.max()*pdf[427])

(2054,)
22.27677449217962
5.143847691885105

x[113]

138

import matplotlib.pyplot as plt
t = np.arange(0,1500,1)
# lower cut-off.
plt.axvline(perc_05,label='05 perc',c='r')
# cut-off.
plt.axvline(perc_95,label='95t perc',c='m')
p=x[10:int(mu)].mean()
#plt.twiny()
#Ploting Zipf distribution
plt.plot(t[:1100],x_s[:1100]/(x.max()), c='b')
#Ploting density function
plt.plot(t[:1100],pdf[:1100]*factor, c='g')
plt.show()

Must Important Words

print (tupla1[-perc_05][1],":",tupla1[-perc_05][0])
print (tupla1[-perc_95][1],":",tupla1[-perc_95][0])
print ('All important words:', tupla1[len(tupla1)-perc_95:len(tupla1)-perc_05+1])

terms : 67
user : 24
All important words: [[24, 'user'], [24, 'write'], [25, 'ability'], [25, 'ask'], [25, 'based'], [25, 'data'], [25, 'debate'], [25, 'democracy'], [25, 'develop'], [25, 'easily'], [25, 'images'], [25, 'libraries'], [25, 'meaning'], [25, 'napster'], [25, 'person'], [25, 'progress'], [25, 'publishing'], [25, 'regulate'], [25, 'sales'], [25, 'school'], [25, 'stories'], [25, 'text'], [25, 'whose'], [26, 'began'], [26, 'david'], [26, 'effective'], [26, 'effectively'], [26, 'generally'], [26, 'gets'], [26, 'increase'], [26, 'knowledge'], [26, 'limit'], [26, 'movie'], [26, 'office'], [26, 'really'], [26, 'rule'], [26, 'songs'], [26, 'sony'], [26, 'stations'], [26, 'strategy'], [26, 'web'], [27, 'already'], [27, 'becomes'], [27, 'certain'], [27, 'consider'], [27, 'extend'], [27, 'extension'], [27, 'given'], [27, 'interests'], [27, 'kahle'], [27, 'matter'], [27, 'produce'], [27, 'publisher'], [27, 'side'], [27, 'special'], [27, 'station'], [27, 'william'], [27, 'wrote'], [28, 'artist'], [28, 'copying'], [28, 'decision'], [28, 'defend'], [28, 'far'], [28, 'issue'], [28, 'laws'], [28, 'limits'], [28, 'lost'], [28, 'past'], [28, 'song'], [28, 'true'], [29, 'armstrong'], [29, 'commerce'], [29, 'courts'], [29, 'extended'], [29, 'imagine'], [29, 'little'], [29, 'monopoly'], [29, 'notes'], [29, 'practice'], [29, 'published'], [29, 'records'], [29, 'september'], [30, 'allowed'], [30, 'asked'], [30, 'burden'], [30, 'concentration'], [30, 'five'], [30, 'gives'], [30, 'microsoft'], [30, 'patent'], [30, 'privacy'], [30, 'rather'], [30, 'sold'], [31, 'buy'], [31, 'call'], [31, 'called'], [31, 'cd'], [31, 'drug'], [31, 'gave'], [31, 'last'], [31, 'response'], [31, 'students'], [31, 'thing'], [31, 'users'], [31, 'words'], [32, 'benefit'], [32, 'brothers'], [32, 'come'], [32, 'future'], [32, 'keep'], [32, 'mp'], [32, 'review'], [32, 'using'], [33, 'commons'], [33, 'done'], [33, 'extraordinary'], [33, 'fm'], [33, 'go'], [33, 'information'], [33, 'kids'], [33, 'process'], [33, 'says'], [33, 'sell'], [33, 'single'], [33, 'sound'], [34, 'americans'], [34, 'association'], [34, 'best'], [34, 'changed'], [34, 'clause'], [34, 'cost'], [34, 'created'], [34, 'described'], [34, 'granted'], [34, 'material'], [34, 'old'], [34, 'open'], [34, 'reason'], [34, 'regulated'], [34, 'requirement'], [34, 'scope'], [34, 'source'], [34, 'three'], [35, 'enough'], [35, 'great'], [35, 'mean'], [35, 'original'], [35, 'paid'], [35, 'since'], [35, 'u_s_'], [36, 'bill'], [36, 'built'], [36, 'cases'], [36, 'derivative'], [36, 'find'], [36, 'hard'], [36, 'ideas'], [36, 'infringement'], [36, 'range'], [36, 'registration'], [36, 'therefore'], [36, 'video'], [36, 'wanted'], [36, 'ways'], [37, 'bit'], [37, 'day'], [37, 'drugs'], [37, 'four'], [37, 'going'], [37, 'innovation'], [37, 'longer'], [37, 'second'], [37, 'statute'], [37, 'taking'], [37, 'things'], [37, 'valenti'], [38, 'aim'], [38, 'always'], [38, 'assure'], [38, 'back'], [38, 'company'], [38, 'ever'], [38, 'jesse'], [38, 'let'], [38, 'noncommercial'], [38, 'quite'], [38, 'university'], [38, '❙❚❘'], [39, 'argued'], [39, 'business'], [39, 'create'], [39, 'end'], [39, 'library'], [39, 'put'], [39, 'rules'], [39, 'ten'], [40, 'architecture'], [40, 'archive'], [40, 'competition'], [40, 'context'], [40, 'formalities'], [40, 'high'], [40, 'house'], [40, 'powerful'], [41, 'anything'], [41, 'better'], [41, 'type'], [42, 'cable'], [42, 'give'], [42, 'problem'], [42, 'produced'], [42, 'society'], [42, 'want'], [43, 'america'], [43, 'either'], [43, 'however'], [43, 'lawyer'], [43, 'making'], [43, 'networks'], [43, 'thought'], [44, 'doubt'], [44, 'makes'], [44, 'patents'], [44, 'press'], [44, 'someone'], [44, 'spread'], [44, 'told'], [45, 'constitution'], [45, 'money'], [46, 'limited'], [47, 'existing'], [47, 'less'], [47, 'publishers'], [47, 'said'], [48, 'author'], [48, 'enable'], [48, 'long'], [48, 'protected'], [48, 'real'], [49, 'around'], [49, 'become'], [49, 'believe'], [49, 'claim'], [49, 'form'], [49, 'value'], [49, 'year'], [50, 'computer'], [50, 'harm'], [50, 'justice'], [50, 'possible'], [50, 'still'], [51, 'authors'], [51, 'exclusive'], [51, 'record'], [52, 'build'], [52, 'copies'], [52, 'indeed'], [52, 'john'], [52, 'license'], [52, 'network'], [52, 'number'], [52, 'obvious'], [52, 'simple'], [53, 'course'], [53, 'general'], [53, 'nothing'], [53, 'supreme'], [54, 'clear'], [54, 'kind'], [54, 'means'], [54, 'read'], [54, 'view'], [55, 'increasingly'], [55, 'protect'], [55, 'television'], [55, 'though'], [56, 'another'], [56, 'creators'], [56, 'films'], [56, 'know'], [56, 'policy'], [56, 'times'], [56, 'understand'], [56, 'uses'], [57, 'file'], [57, 'instead'], [57, 'lawyers'], [57, 'million'], [58, 'books'], [59, 'else'], [60, 'costs'], [60, 'eldred'], [60, 'freedom'], [60, 'particular'], [60, 'protection'], [60, 'simply'], [62, 'code'], [62, 'must'], [62, 'riaa'], [63, 'balance'], [63, 'never'], [63, 'say'], [64, 'effect'], [64, 'question'], [64, 'regulation'], [65, 'companies'], [65, 'pay'], [65, 'york'], [66, 'artists'], [66, 'changes'], [66, 'copyrighted'], [66, 'need'], [66, 'story'], [67, 'set'], [67, 'terms']]

This method is widely used in the generation of automatic summaries. It can be used to give weight to the words within the document, and extract the most weighted structures: sentences, paragraphs, etc. As can be seen in the example of the book “Free Culture”, this method leaves 300 significant words from 120303 words in the original text.

References

[1] [Bird2009] Steven Bird, Ewan Klain & Edward Loper,. Book Natural Language Processing with Python. 2009. p. 10 ISBN: 978-0-596-51649-9

[2] [Luhn1958] H.P. Luhn. Paper The Automatic Creation of Literature Abstract. IBM Journal, 1958.

Alphabetic Index

Token: in linguistics is an individual occurrence of a linguistic unit, generally referred as the smallest unit of processing: words, phonemes, n-grams, etc.