Corpus data on Ocelot

  • Written by Mauricio Gruppi on Jan 26, 2024
  • Here you will find information about the corpus data in Ocelot and how to access it using the shell.
  • Note: Once you sign in through ssh, you be in your home folder, the absolute path for which is /home/<your_username>/
  • The corpus data resides in /data/corpora
  • Run cd /data/corpora to change the working directory to the data folder.
  • E.g.: changing the working directory to /data/corpora: 
    mgouveag@ocelot:~$ cd /data/corpora 
mgouveag@ocelot:/data/corpora$
  • Your terminal is now pointed to the directory /data/corpora - you can think of it as "opening" the folder using a graphical interface
  • To list all the files and folders in the current directory, run ls -lh. E.g.: 
    mgouveag@ocelot:/data/corpora$ ls -lh
total 32K
drwxrwxr-x  4 mgouveag data 4.0K Jan 24 19:34 coca
drwxrwxr-x  3 mgouveag data 4.0K Jan 24 19:28 coha
drwxrwxr-x  2 mgouveag data 4.0K Jan 24 19:27 compressed
drwxrwxr-x 11 mgouveag data 4.0K Jan 25 10:59 congress_text
drwxr-xr-x  2 mgouveag data 4.0K Jan 24 12:55 english-us-uk
drwxrwxr-x  3 mgouveag data 4.0K Jan 24 18:45 gutenberg
drwxr-xr-x  2 mgouveag data 4.0K Jan 24 13:00 nela
drwxr-xr-x  6 mgouveag data 4.0K Jan 24 12:53 semeval-2020-task1
  • Here is a brief explanation of what this list means:
  • The leftmost sequence of characters, such as drwxrwxr-x indicates permissions of the respective file. The leading d indicates the item is a directory, r ,w, and x indicate read, write and execute permissions for the owner, the group and all users in the system (you do not need to worry about these right now).
  • The second number is the number of hard links to the file (not relevant here)
  • The third column is the owner of the file. Here, it is myself (mgouveag).
  • The fourth column is the group of the folder. We are all part of group data, which allows us to access these files.
  • The fifth column is the size of the file (folders are always 4KB in Unix).
  • The sixth column is the date and time of creation.
  • The seventh and final column is the name of the file. Note that folders are also considered files.
  • In this section, I will briefly explain each of the corpora.

  • COCA - Corpus of Contemporary American English

    • This corpus contains texts from the year 1990 through 2012 across different categories: academic, spoken, fiction, magazine, and news.
    • The folder coca/text_files holds all such text files
    • To peek a file you can use the head command. For instance, the following will print the first few lines of file w_acad_2010.txt:
    • head coca/text_files/w_acad_2010.txt

  • COHA - Corpus of Historical American English

    • Similar to COCA, except that it includes data from a longer period (from 1820-2010).
    • Files are in coha/text_files

  • NELA - News Landscape

    • A corpus of contemporary news, containing millions of news articles from hundreds of news outlets from years 2018-2023.
    • This corpus is in a database, not plain text files

  • Congressional Record (congress_text)

    • Transcripts of the congressional speeches from the 43rd through the 114th US congress.
    • Years 1873 - 2017

  • Project Gutenberg

    • Over 71,000 books ranging including fiction and non-fiction
    • Data in: gutenberg/data/raw/
    • Metadata for each book is available
    • E.g.: to take a peek into the first 100 lines of Lewis Carrol's Alice's Adventures in Wonderland:
    • head -n 100 gutenberg/data/raw/PG11_raw.txt
      • Make sure your current directory is /data/corpora
    • The metadata about all the books is in gutenberg/metadata/metadata.csv