Python clinic day 2: Corpus processing

Na-Rae Han (naraehan@pitt.edu), 2017-07-13, Pittsburgh NEH Institute “Make Your Edition”

Preparation

Data

• This tutorial is found on https://pittsburgh-neh-institute.github.io/Institute-Materials-2017/schedule/week_1/week_1_day_4_plan.html
• Download and unzip the “C-Span Inaugural Address Corpus”, available on NLTK’s corpora page: http://www.nltk.org/nltk_data/
• Place the unzipped inaugural folder on your desktop

Jupyter tips

• Click + to create a new cell, ► to run
• Alt+ENTER to run cell, create a new cell below
• Shift+ENTER to run cell, go to next cell

More on https://www.cheatography.com/weidadeyue/cheat-sheets/jupyter-notebook/

Review

• Let’s review what we learned yesterday.

Processing a single text file, continued

Reading in a text file

• Start by opening up the 1789 Washington address, using open(filename).read().

myfile = 'C:/Users/narae/Desktop/inaugural/1789-Washington.txt'  # Use your own userid; Mac users should omit C:
wtxt = open(myfile).read()
print(wtxt[:500])

Tokenize text, compile frequency count

import nltk    # Don't forget to import nltk
%pprint    # Turn off/on pretty printing (prints too many lines)

# Build a token list
wtokens = nltk.word_tokenize(wtxt)
len(wtokens)     # Number of words in text

# Build a dictionary of word frequency count
wfreq = nltk.FreqDist(wtokens)
wfreq['the']

len(wfreq)      # Number of unique words in text

wfreq.most_common(40)     # 40 most common words

Average sentence length, frequency of long words

sentcount = wfreq['.'] + wfreq['?'] + wfreq['!']  # Assuming every sentence ends with ., ! or 
print(sentcount)

# Tokens include symbols and punctuation. First 50 tokens:
wtokens[:50]

wtokens_nosym = [t for t in wtokens if t.isalnum()]    # alpha-numeric tokens only
len(wtokens_nosym)

# Try "n't", "20th", "."
"n't".isalnum()

# First 50 tokens, alpha-numeric tokens only: 
wtokens_nosym[:50]

len(wtokens_nosym)/sentcount     # Average sentence length in number of words

[w for w in wfreq if len(w) >= 13]       # all 13+ character words

long = [w for w in wfreq if len(w) >= 13] 
# sort long alphabetically using sorted()
for w in sorted(long) :
    print(w, len(w), wfreq[w])               # long words tend to be less frequent

Processing a corpus

• NLTK can read in an entire corpus from a directory (the “root” directory).
• As it reads in a corpus, it applies word tokenization (.words()) and sentence tokenization (.sents()).

from nltk.corpus import PlaintextCorpusReader
corpus_root = 'C:/Users/Jane Eyre/Desktop/inaugural'  # Use your own userid; Mac users should omit C:
inaug = PlaintextCorpusReader(corpus_root, '.*txt')  # all files ending in 'txt' 

# .txt file names as file IDs
inaug.fileids()

# NLTK automatically tokenizes the corpus. First 50 words: 
print(inaug.words()[:50])

# You can also specify individual file ID. First 50 words from Obama 2009:
print(inaug.words('2009-Obama.txt')[:50])

# NLTK automatically segments sentences too, which are accessed through .sents()
print(inaug.sents('2009-Obama.txt')[0])   # first sentence
print(inaug.sents('2009-Obama.txt')[1])   # 2nd sentence

# How long are these speeches in terms of word and sentence count?
print('Washington 1789:', len(inaug.words('1789-Washington.txt')), len(inaug.sents('1789-Washington.txt')))
print('Obama 2009:', len(inaug.words('2009-Obama.txt')), len(inaug.sents('2009-Obama.txt')))

# for-loop through file IDs and print out word count. 
# While looping, populate fid_avsent which holds avg sent lengths.
# Break long line with \, specify tab separator. 
fid_avsent = []
for f in inaug.fileids():
    print(len(inaug.words(f)), len(inaug.sents(f)), \
          len(inaug.words(f)) / len(inaug.sents(f)), f, sep='\t')
    fid_avsent.append((len(inaug.words(f)) / len(inaug.sents(f)), f))

# Turn pretty print back on 
%pprint
sorted(fid_avsent)

# same thing, with list comprehension! 
fid_avsent2 = [(len(inaug.words(f)) / len(inaug.sents(f)), f) for f in inaug.fileids()]
sorted(fid_avsent2)

Trouble shooting

• Unfortunately, 2005 Bush file produces a Unicode encoding error.
• Let's make a new text file from http://www.presidency.ucsb.edu/inaugurals.php
• Copy text and paste in Notepad/Notepad++ (Windows) or BBEdit (Mac).
  • Windows users: make sure to choose UTF-8 encoding and not ANSI when saving.
• The text files are locked; We will need to save, halt and then re-start the Python notebook.

# Corpus size in number of words
print(len(inaug.words()))

# Building word frequency distribution for the entire corpus
inaug_freq = nltk.FreqDist(inaug.words())
inaug_freq.most_common(100)

Extra: using regular expressions for tokenization

• re is Python's regular expression module. Start by importing.
• re.findall finds all substrings that match a pattern.
• For regular expression strings, use r'...' (rawstring) prefix.

import re

sent = "You haven't seen Star Wars...?"
re.findall(r'\w+', sent)

%pprint
re.findall(r'\w+', wtxt)

What next?

Take a Python course! There are many online courses available on Coursera, EdX, udemy, DataCamp, and more.