Cleanup of section 2

Author: Christopher Fonnesbeck

2. Data Wrangling with Pandas.ipynb

@@ -509,6 +509,23 @@
       "Notice that `mmsi` field that was an index on the `vessels` table is no longer an index on the merged table."
      ]
     },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "Here, we used the `merge` function to perform the merge; we could also have used the `merge` method for either of the tables:"
+     ]
+    },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "vessels.merge(segments, left_index=True, right_on='mmsi').head()"
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
+    },
     {
      "cell_type": "markdown",
      "metadata": {},
@@ -588,8 +605,8 @@
      "cell_type": "code",
      "collapsed": false,
      "input": [
-      "mb1 = pd.ExcelFile('data/microbiome/MID1.xls').parse(\"Sheet 1\", index_col=0, header=None, names=['Taxon', 'Count'])\n",
-      "mb2 = pd.ExcelFile('data/microbiome/MID2.xls').parse(\"Sheet 1\", index_col=0, header=None, names=['Taxon', 'Count'])\n",
+      "mb1 = pd.read_excel('data/microbiome/MID1.xls', 'Sheet 1', index_col=0, header=None)\n",
+      "mb2 = pd.read_excel('data/microbiome/MID2.xls', 'Sheet 1', index_col=0, header=None)\n",
       "mb1.shape, mb2.shape"
      ],
      "language": "python",
@@ -606,6 +623,13 @@
      "metadata": {},
      "outputs": []
     },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "Let's give the index and columns meaningful labels:"
+     ]
+    },
     {
      "cell_type": "code",
      "collapsed": false,
@@ -626,6 +650,16 @@
      "metadata": {},
      "outputs": []
     },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "mb1.head()"
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
+    },
     {
      "cell_type": "markdown",
      "metadata": {},
@@ -754,7 +788,7 @@
      "cell_type": "code",
      "collapsed": false,
      "input": [
-      "mb1.combine_first(mb2).shape"
+      "mb1.combine_first(mb2).head()"
      ],
      "language": "python",
      "metadata": {},
@@ -788,7 +822,7 @@
      "cell_type": "code",
      "collapsed": false,
      "input": [
-      "pd.concat({'patient1': mb1, 'patient2': mb2}, axis=1).head()"
+      "pd.concat(dict(patient1=mb1, patient2=mb2), axis=1).head()"
      ],
      "language": "python",
      "metadata": {},
@@ -909,18 +943,29 @@
      "metadata": {},
      "outputs": []
     },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "cdystonia2.index.is_unique"
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
+    },
     {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
-      "If we want to transform this data so that repeated measurements are in columns, we can `unstack` according to `obs`."
+      "If we want to transform this data so that repeated measurements are in columns, we can `unstack` the `twstrs` measurements according to `obs`."
      ]
     },
     {
      "cell_type": "code",
      "collapsed": false,
      "input": [
-      "cdystonia2.unstack('obs')"
+      "twstrs_wide = cdystonia2['twstrs'].unstack('obs')\n",
+      "cdystonia.ix[:,:-1].merge(twstrs_wide, right_index=True, left_on='patient').head()"
      ],
      "language": "python",
      "metadata": {},
@@ -950,17 +995,7 @@
      "cell_type": "code",
      "collapsed": false,
      "input": [
-      "cdystonia.pivot_table?"
-     ],
-     "language": "python",
-     "metadata": {},
-     "outputs": []
-    },
-    {
-     "cell_type": "code",
-     "collapsed": false,
-     "input": [
-      "cdystonia.pivot('patient', 'obs', 'twstrs').head()"
+      "cdystonia.pivot(index='patient', columns='obs', values='twstrs').head()"
      ],
      "language": "python",
      "metadata": {},
@@ -970,7 +1005,7 @@
      "cell_type": "markdown",
      "metadata": {},
      "source": [
-      "If we omit the `value` argument, we get a `DataFrame` with hierarchical columns, just as when we applied `unstack` to the hierarchically-indexed table:"
+      "If we omit the `values` argument, we get a `DataFrame` with hierarchical columns, just as when we applied `unstack` to the hierarchically-indexed table:"
      ]
     },
     {
@@ -1329,7 +1364,7 @@
      "source": [
       "### Permutation and sampling\n",
       "\n",
-      "For some data analysis tasks, such as simulation, we need to be able to randomly reorder our data, or draw random values from it. Calling NumPy's `permutation` function with the length of the sequence you want to permute generates an array with a random array of integers, which can be used to re-order the sequence."
+      "For some data analysis tasks, such as simulation, we need to be able to randomly reorder our data, or draw random values from it. Calling NumPy's `permutation` function with the length of the sequence you want to permute generates an array with a permuted sequence of integers, which can be used to re-order the sequence."
      ]
     },
     {
@@ -1376,7 +1411,7 @@
      "source": [
       "## Exercise\n",
       "\n",
-      "Use this permutation approach to draw a random sample (without replacement) of 5 ships from the `vessels` DataFrame."
+      "Its easy to see how this permutation approach allows us to draw a random sample **without replacement**. How would you sample **with replacement**? Generate a random sample of 5 ships from the `vessels` DataFrame using this scheme."
      ]
     },
     {
@@ -1395,9 +1430,9 @@
       "\n",
       "One of the most powerful features of Pandas is its **GroupBy** functionality. On occasion we may want to perform operations on *groups* of observations within a dataset. For exmaple:\n",
       "\n",
-      "* aggregation, such as computing the sum of mean of each group, which involves applying a function to each group and returning the aggregated results\n",
-      "* slicing the DataFrame into groups and then doing something with the resulting slices (*e.g.* ploting)\n",
-      "* group-wise transformation, such as standardization/normalization"
+      "* **aggregation**, such as computing the sum of mean of each group, which involves applying a function to each group and returning the aggregated results\n",
+      "* **slicing** the DataFrame into groups and then doing something with the resulting slices (*e.g.* plotting)\n",
+      "* group-wise **transformation**, such as standardization/normalization"
      ]
     },
     {
@@ -1456,7 +1491,9 @@
       "\n",
       "For example, we may want to aggregate our data with with some function.\n",
       "\n",
-      "![split-apply-combine](http://f.cl.ly/items/0s0Z252j0X0c3k3P1M47/Screen%20Shot%202013-06-02%20at%203.04.04%20PM.png)"
+      "![split-apply-combine](http://f.cl.ly/items/0s0Z252j0X0c3k3P1M47/Screen%20Shot%202013-06-02%20at%203.04.04%20PM.png)\n",
+      "\n",
+      "*(taken from \"Python for Data Analysis\")*"
      ]
     },
     {
@@ -1516,6 +1553,7 @@
      "cell_type": "code",
      "collapsed": false,
      "input": [
+      "# The median of the `twstrs` variable\n",
       "cdystonia_grouped['twstrs'].quantile(0.5)"
      ],
      "language": "python",
@@ -1678,7 +1716,7 @@
      "collapsed": false,
      "input": [
       "def top(df, column, n=5):\n",
-      "    return df.sort_index(by=column)[-n:]"
+      "    return df.sort_index(by=column, ascending=False)[:n]"
      ],
      "language": "python",
      "metadata": {},
@@ -1781,16 +1819,6 @@
      "metadata": {},
      "outputs": []
     },
-    {
-     "cell_type": "code",
-     "collapsed": false,
-     "input": [
-      "mb_class.index.is_unique"
-     ],
-     "language": "python",
-     "metadata": {},
-     "outputs": []
-    },
     {
      "cell_type": "markdown",
      "metadata": {},
@@ -1802,7 +1830,7 @@
      "cell_type": "code",
      "collapsed": false,
      "input": [
-      "mb_class.groupby(level=0).sum().tail(10)"
+      "mb_class.groupby(level=0).sum().head(10)"
      ],
      "language": "python",
      "metadata": {},