This repository contains Ruby scripts that can assist in analyzing words in plain text files containing French text.
Here is a short description of what each script can be used for.
Classifying words
ClassifyInterestingWords.rb
Usage: ruby ClassifyInterestingWords.rb <directory1> [<directory2> ...]
Example: ruby ClassifyInterestingWords.rb corpus-example/25stanley_Montreal
Output: words.maybe words.no words.yes
Allows you to classify words based on whether they are interesting for further analysis. The script looks for text files with the .txt extension in the given directories, and asks you to classify each word in those files. It then puts each word into file words.yes, words.maybe or words.no in the current directory, depending on your choice. The script also creates backup files ending with .bak.
Press q anytime to quit the script.
The created files are used as input for the ClassifyLoanWords.rb script, which handles the files in words.yes and words.maybe and ignores the words in words.no.
ClassifyLoanWords.rb
Usage: ruby ClassifyLoanWords.rb <directory>
Example: ruby ClassifyLoanWords.rb .
Output: words.direct words.translated words.original
Allows you to classify words based on what kind of loan words they are. The script goes through the words in words.yes and words.maybe files in the given directory, and asks you to classify each word as a "direct" (unmodified) loan, "translated" (modified) loan, or an "original" (not loaned) word. It then puts each word into file words.direct, words.translated or words.original in the current directory, depending on your choice.
Press q anytime to quit the script.
The created files are used as input for the FilterUniqueWords.rb script, which handles the files in words.direct and words.translated and ignores the words in words.original.
FilterUniqueWords.rb
Usage: ruby FilterUniqueWords.rb <directory>
Example: ruby FilterUniqueWords.rb .
Output: words.direct.filtered words.translated.filtered
Combines (filters out) different forms of a word to obtain a list of unique words. The script goes through the words in words.direct and words.translated files in the given directory, and combines the different forms (e.g. "joue" and "jouent") of each word on one line (e.g. "joue, jouent"). The results are written to files with the .filtered extension.
Counting words
CountFrequencies.rb
Usage: ruby CountFrequencies.rb <directory1> [<directory2> ...]
Example: ruby CountFrequencies.rb corpus-example/25stanley_Montreal corpus-example/LNH_Montreal
Output: .freq file per each text file, all.freq for each directory
Counts word frequencies in text files in given directories and the total word frequencies in each directory. The script looks for text files with the .txt extension in the given directories, and calculates word frequencies in each text file as well as in each directory. The frequencies are written into files with the .freq extension. The script tries its best to combine different forms of a word in its calculations.
CountWords.rb
Usage: ruby CountWords.rb <directory1> [<directory2> ...]
Example: ruby CountWords.rb corpus-example/25stanley_Montreal corpus-example/LNH_Montreal
Output: Shows word counts on the terminal
Counts the number of words in text files and the total counts in directories. The script looks for text files with the .txt extension in the given directories, and calculates how many words there are in each file as well as in each directory in total. Certain words with an apostrophe in them ("d'abord", "d'affilée", "d'ailleurs", "d'autres", "d'habitude", "d'ici", "d'où", and "l'un", see lib/Helpers.rb) are recognized as single words instead of two.
Finding words in their context
FindOccurrences.rb
Usage: ruby FindOccurrences.rb <word> <surrounding word count> <directory1> [<directory2> ...]
Example: ruby FindOccurrences.rb score 10 corpus-example/25stanley_Montreal
Output: Shows word occurrences on the terminal
Shows the occurrences of a word in all its contexts in the text files. The script looks for text files with the .txt extension in the given directories, and shows the given word and the given surrounding word amount of words preceding and following it. The script is useful for checking all the contexts in which a particular word is used.