04-Stylometry/01-Ad-Hoc-Stylometry.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Stylometry\n",
    "\n",
    "This notebook is designed to reproduce several findings from Emily Thornbury's chapter \"The Poet Alone\" in her book *Becoming a Poet in Anglo-Saxon England.* In particular, Fig. 4.5 on page 170.\n",
    "\n",
    "First, however, we're going to think about what we might do with lists of strings. After all, how else can we count features of a string unless we can somehow make a list of items out of it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "[\"þæt\", \"wearð\", \"underne\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do I know?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type([\"þæt\", \"wearð\", \"underne\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can assign these to variables too!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "first_hemistich = [\"þæt\", \"wearð\", \"underne\"]\n",
    "second_hemistich = [\"eorðbuendum\"]\n",
    "print(first_hemistich)\n",
    "print(second_hemistich)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And perform mathematical operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(first_hemistich + second_hemistich)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's assign that to `first_line`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "first_line = first_hemistich + second_hemistich\n",
    "print(first_line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get the length of a `list` using the `len` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "len(first_line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can index lists with brackets `[]`, let's get the first word of the first line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(first_line[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-danger\">\n",
    "Don't forget, Python (and many other langauges) start counting from 0.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(first_line[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get ranges using a semi-colon `:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(first_line[:2])\n",
    "print(type(first_line[:2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenge 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Concatenate the first three lines of Christ and Satan.\n",
    "- Retrieve the third element from the combined list.\n",
    "- Retrieve the fourth through sixth elements from the combined list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "first_line = ['þæt', 'wearð', 'underne', 'eorðbuendum,']\n",
    "second_line = ['þæt', 'meotod', 'hæfde', 'miht', 'and', 'strengðo']\n",
    "third_line = ['ða', 'he', 'gefestnade', 'foldan', 'sceatas.']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "---\n",
    "\n",
    "## List Comprehension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now, think of a list comprehension as a fast way to sift out items from a list, instead of writing a `for` loop that appends to a new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "[word for word in first_line if \"e\" in word]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***INSTEAD OF***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "has_e = []\n",
    "\n",
    "for word in first_line:\n",
    "    if \"e\" in word:\n",
    "        has_e.append(word)\n",
    "\n",
    "has_e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Now you know why list comprehensions are one of the best parts of Python!\n",
    "\n",
    "Especially for text analysis, these will come in handy when we want to parse and sift through text.\n",
    "\n",
    "## Challenge 2\n",
    "\n",
    "- Concatenate the first three lines of Christ and Satan.\n",
    "- Create a new list that contains only the words whose ***last*** letter is \"e\"\n",
    "- Create a new list that contains the ***first*** letter of each word.\n",
    "- Create a new list that contains only words longer than ***two*** letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Word Frequencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with open('data/christ-and-satan.txt', 'r') as f:\n",
    "    christ_and_satan = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tokens = christ_and_satan.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Looks like a decent start. But we still have verse numbering in there, as well as some punctuation. What if we just want the words?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from string import punctuation, digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "digits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does it feel like time for a list comprehension? It should.\n",
    "\n",
    "## Challenge 3\n",
    "\n",
    "Write a list comprehension to remove line numbers and punctuation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Python comes with the convenient `Counter` method from the `collections` library. It returns a `dictionary` like object that will return the frequency of a particular key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "cs_dict = Counter(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cs_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cs_dict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cs_dict.values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cs_dict.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Believe it or not, even 1000 years ago \"and\" was still used all the time :) ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Challenge 4\n",
    "\n",
    "- A common measure of lexical diversity for a given text is its Type-Token Ratio: the ratio of unique words (type) to number of all words (tokens) in the text.\n",
    "- Calculate the Type-Token Ratio for *Christ and Satan*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from datascience import *\n",
    "import numpy as np\n",
    "\n",
    "words, frequency = zip(*cs_dict.items())\n",
    "\n",
    "t = Table([\"Words\", \"Frequency\"])\n",
    "t.append_column(\"Words\", words)\n",
    "t.append_column(\"Frequency\", frequency)\n",
    "top_table = t.sort(\"Frequency\", descending=\"True\").take(np.arange(5))\n",
    "top_table.bar(column_for_categories=\"Words\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ad Hoc Stylometry\n",
    "\n",
    "We can now put together our knowledge of strings, list comprehensions, and plotting frequencies to look at frequency of alliteration letters. Remember: Alliteration is the repetition of a sound at the beginning of two or more words in the same line.\n",
    "\n",
    "Let's start by looking at the first letter of every word in the whole text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cs_tokens = christ_and_satan.lower().split()\n",
    "first_letters = [x[0] if x[0] not in ['a','e','i','o','u','y'] else 'a' for x in cs_tokens]\n",
    "first_l_dict = Counter(first_letters)\n",
    "first_l_freq = first_l_dict.most_common()\n",
    "print(first_l_freq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# plot\n",
    "letters, frequency = zip(*first_l_dict.items())\n",
    "\n",
    "t = Table([\"Letters\", \"Frequency\"])\n",
    "t.append_column(\"Letters\", letters)\n",
    "t.append_column(\"Frequency\", frequency)\n",
    "top_table = t.sort(\"Frequency\", descending=\"True\").take(np.arange(5))\n",
    "top_table.bar(column_for_categories=\"Letters\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool! But we need it within a line, and Thornbury specifically does it for each Fitt. What's a \"Fitt\"? It's a further division in poetry constituted by a group of lines. Luckily this is nicely delimited by double line breaks (`\\n\\n`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cs_fitts = christ_and_satan.split('\\n\\n')\n",
    "cs_fitts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize = (10,10))\n",
    "\n",
    "# iterate through fitts\n",
    "for i in range(len(cs_fitts)):\n",
    "    \n",
    "    # lowercase the string and get the tokens for each line back\n",
    "    fitt_tokens = [l.split() for l in cs_fitts[i].lower().split('\\n')]\n",
    "    \n",
    "    # collect letter of most freq alliteration\n",
    "    most_freq_allit = []\n",
    "    \n",
    "    # cycle through lines\n",
    "    for l in fitt_tokens:\n",
    "        \n",
    "        # get first letter of all words in line\n",
    "        first_letters = [x[0] if x[0] not in ['a','e','i','o','u','y'] else 'a' for x in l]\n",
    "        \n",
    "        # count freq of all first letters\n",
    "        allit_freq = Counter(first_letters).most_common()\n",
    "        try:\n",
    "            # append most freq letter (alliterated letter) to list for all lines\n",
    "            most_freq_allit.append(allit_freq[0][0])\n",
    "        except:\n",
    "            pass\n",
    "    \n",
    "    # use Counter to get the most common alliterations\n",
    "    allit_freq = Counter(most_freq_allit).most_common()\n",
    "\n",
    "    # need keys for x axis\n",
    "    common_keys = [x[0] for x in allit_freq]\n",
    "    \n",
    "    # need values for y axes\n",
    "    common_values = [x[1] for x in allit_freq]\n",
    "    \n",
    "    # normalize so we can compare across Fitts despite different number of words\n",
    "    normed_values = [x[1]/sum(common_values) for x in allit_freq]\n",
    "    \n",
    "    # add up to get cumulative alliteration of the four most preferred patterns\n",
    "    cumulative_values = np.cumsum(normed_values)\n",
    "\n",
    "    # add the Fitt to the plot\n",
    "    plt.xticks(range(4), ['1st','2nd','3rd','4th'], rotation='vertical')\n",
    "    plt.plot(cumulative_values[:4], color = plt.cm.bwr(i*.085), lw=3)\n",
    "plt.legend(labels=['Fitt '+str(i+1) for i in range(12)], loc=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Homework: Acrostics\n",
    "In poetry, an acrostic is a message created by taking certain letters in a pattern over lines. One 9th century German writer, Otfrid of Weissenburg, was notorius for his early use of acrostics, one instance of which is in the text below: *Salomoni episcopo Otfridus*. His message can be found by taking the first character of every other line. Print Otfrid's message!\n",
    "\n",
    "Source: http://titus.uni-frankfurt.de/texte/etcs/germ/ahd/otfrid/otfri.htm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text = '''si sálida gimúati      sálomones gúati, \n",
    "     ther bíscof ist nu édiles      kóstinzero sédales; \n",
    "     allo gúati gidúe thio sín,      thio bíscofa er thar hábetin, \n",
    "     ther ínan zi thiu giládota,      in hóubit sinaz zuívalta! \n",
    "     lékza ih therera búachi      iu sentu in suábo richi, \n",
    "     thaz ir irkíaset ubar ál,      oba siu frúma wesan scal; \n",
    "     oba ir hiar fíndet iawiht thés      thaz wírdig ist thes lésannes: \n",
    "     iz iuer húgu irwállo,      wísduames fóllo. \n",
    "     mir wárun thio iuo wízzi      ju ófto filu núzzi, \n",
    "     íueraz wísduam;      thes duan ih míhilan ruam. \n",
    "     ófto irhugg ih múates      thes mánagfalten gúates, \n",
    "     thaz ír mih lértut hárto      íues selbes wórto. \n",
    "     ni thaz míno dohti      giwérkon thaz io móhti, \n",
    "     odo in thén thingon      thio húldi so gilángon; \n",
    "     iz datun gómaheiti,      thio íues selbes gúati, \n",
    "     íueraz giráti,      nales míno dati. \n",
    "     emmizen nu ubar ál      ih druhtin férgon scal, \n",
    "     mit lón er iu iz firgélte      joh sínes selbes wórte; \n",
    "     páradyses résti      gébe iu zi gilústi; \n",
    "     ungilónot ni biléip      ther gotes wízzode kleip. \n",
    "     in hímilriches scóne      so wérde iz iu zi lóne \n",
    "     mit géltes ginúhti,      thaz ír mir datut zúhti. \n",
    "     sínt in thesemo búache,      thes gómo theheiner rúache; \n",
    "     wórtes odo gúates,      thaz lích iu iues múates: \n",
    "     chéret thaz in múate      bi thia zúhti iu zi gúate, \n",
    "     joh zellet tház ana wánc      al in íuweran thanc. \n",
    "     ofto wírdit, oba gúat      thes mannes júngoro giduat, \n",
    "     thaz es líwit thráto      ther zúhtari gúato. \n",
    "     pétrus ther rícho      lono iu es blídlicho, \n",
    "     themo zi rómu druhtin gráp      joh hús inti hóf gap; \n",
    "     óbana fon hímile      sént iu io zi gámane \n",
    "     sálida gimýato      selbo kríst ther gúato! \n",
    "     oba ih irbálden es gidár,      ni scal ih firlázan iz ouh ál, \n",
    "     nub ih ío bi iuih gerno      gináda sina férgo, \n",
    "     thaz hóh er iuo wírdi      mit sínes selbes húldi, \n",
    "     joh iu féstino in thaz múat      thaz sinaz mánagfalta gúat; \n",
    "     firlíhe iu sines ríches,      thes hohen hímilriches, \n",
    "     bi thaz ther gúato hiar io wíaf      joh émmizen zi góte riaf; \n",
    "     rihte íue pédi thara frúa      joh míh gifúage tharazúa, \n",
    "     tház wir unsih fréwen thar      thaz gotes éwiniga jár, \n",
    "     in hímile unsih blíden,      thaz wízi wir bimíden; \n",
    "     joh dúe uns thaz gimúati      thúruh thio síno guati! \n",
    "     dúe uns thaz zi gúate      blídemo múate! \n",
    "     mit héilu er gibóran ward,      ther io thia sálida thar fand, \n",
    "     uuanta es ni brístit furdir      (thes gilóube man mír), \n",
    "     nirfréwe sih mit múatu      íamer thar mit gúatu. \n",
    "     sélbo krist ther guato      firlíhe uns hiar gimúato, \n",
    "     wir íamer fro sin múates      thes éwinigen gúates!'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# HINT: remember what % does, (maybe) lookup enumerate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Otfrid was more skillful than to settle for the first letter of every other line. What happens if you extract the last letter of the last word of each line, for every other line starting on the second line?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# HINT: first remove punctuation, tab is represented by \\t\n",
    "from string import punctuation"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}