3 - Benchmarking Algorithms.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".container{width:75%;}\n",
       "</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<style>\n",
    ".container{width:75%;}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benchmarking optimization algorithms in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is time to see how different algorithms perform on a given set of funtions. In order to compare them we have build the OptBenchmark class, a Dashboard that will help us manage the data collection process. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                        fun: -0.0\n",
      " lowest_optimization_result:       fun: -0.0\n",
      " hess_inv: array([[1, 0],\n",
      "       [0, 1]])\n",
      "      jac: array([ 0.,  0.])\n",
      "  message: 'Optimization terminated successfully.'\n",
      "     nfev: 4\n",
      "      nit: 0\n",
      "     njev: 1\n",
      "   status: 0\n",
      "  success: True\n",
      "        x: array([ 47.39172714,  41.82602464])\n",
      "                    message: ['requested number of basinhopping iterations completed successfully']\n",
      "      minimization_failures: 0\n",
      "                       nfev: 404\n",
      "                        nit: 100\n",
      "                       njev: 101\n",
      "                          x: array([ 47.39172714,  41.82602464])\n"
     ]
    }
   ],
   "source": [
    "benchmark = OptBenchmark()\n",
    "benchmark.widget"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<img src=\"benchmark_data/widget.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, this class is meant to be used as an interactive dashboard. It has many options about how to perform the benchmark."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. The main menu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following image, we have the following widgets located at the left of the Dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"benchmark_data/status.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Version**: Textbox when you can write a name for the algorithm currently tested. This allows to distinguis between two different set of paramters for the same algorithm. \n",
    "- **Run button**: Starts the benchmark.\n",
    "- **Progress bars**: Show the current state of the benchmarking calculations. \n",
    "- **Max time**: Sets the mast time a single function optimization can last. It is only taken into account is Time is an active end condition. \n",
    "- **Rounds**: How many rounds the algorithm will be tested against every objective function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. End condition options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These widgets allows us to set when we want the optimization process to finish. We can set up to 4 complementary conditions for ending an optimization process. Clicking on a toggle button will activate one of the following options:\n",
    "\n",
    "- **Find best**: The optimization process will stop once the minimum value of the objective function has been found. \n",
    "- **Max epochs**: Maximum number of iterations an algorithm can perform before finishing the optimization process.\n",
    "- **Time**: Maximum time to be spend when trying to optimize an objective function.\n",
    "- **Samples**: Maximum number of function evaluations allowed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"benchmark_data/end_codition.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Found options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This menu is intended for specifiying how a successful optimization will be defined. The widgets in this submenu allow us to specify the sensibility for triggering the \"Find best\" condition, and therefore for considering an objective function solved.\n",
    "\n",
    "There are two complementary ways of defining an and condition:\n",
    "\n",
    "- **Distance**: The euclidean distance between the position of the best solution found and the position of the global minimum is less than a threshold.\n",
    "- **Error**: The difference between the best value found and the global minimum is less than a threshold.\n",
    "\n",
    "This threshold n is set using the slider widget. If the log10 error checkbox is clicked the threshold value will be 10^n. If the checkbox is not active the trheshold value will be n."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"benchmark_data/found.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Save options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to automatically save the results of a given benchmarking process as it is being performed using the save options widgets.\n",
    "\n",
    "It is also possible to specify the name of the file where the data will be stored. This file will be created inside a data/ folder that must be located in the same directory directory as the notebook.\n",
    "\n",
    "The togglebuttons menu is used for selecting how often the results of the benchmark will be saved. \n",
    "\n",
    "- **Round**: Save the results every time that all the algorithms have finished all the objective functions once. \n",
    "- **Function**: Save after every algorithm finishes one objective function. \n",
    "- **Algorithm**: Save when an algorithm finishes all the objective functions.\n",
    "\n",
    "The slider labeled \"Every\" selects how many times the condition selected on the togglemenu has to be met before saving the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"benchmark_data/save.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, this is how we access the benchmarking results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>id_val</th>\n",
       "      <th>run</th>\n",
       "      <th>time</th>\n",
       "      <th>reads</th>\n",
       "      <th>value</th>\n",
       "      <th>x</th>\n",
       "      <th>found</th>\n",
       "      <th>dist</th>\n",
       "      <th>error</th>\n",
       "      <th>function</th>\n",
       "      <th>algo</th>\n",
       "      <th>id</th>\n",
       "      <th>epoch</th>\n",
       "      <th>version</th>\n",
       "      <th>full_id</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id_val</th>\n",
       "      <th>run</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"10\" valign=\"top\">matyas_bh</th>\n",
       "      <th>0</th>\n",
       "      <td>matyas_bh</td>\n",
       "      <td>0</td>\n",
       "      <td>0.008182</td>\n",
       "      <td>62.0</td>\n",
       "      <td>5.138246e-16</td>\n",
       "      <td>[-1.0781327840561628e-07, -8.349620681837033e-08]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.363647e-07</td>\n",
       "      <td>5.138315e-16</td>\n",
       "      <td>matyas</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>matyas_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>matyas_bh</td>\n",
       "      <td>1</td>\n",
       "      <td>0.005991</td>\n",
       "      <td>58.0</td>\n",
       "      <td>3.753407e-16</td>\n",
       "      <td>[-9.598174804600901e-08, -9.758724177008626e-08]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.368787e-07</td>\n",
       "      <td>3.753398e-16</td>\n",
       "      <td>matyas</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>matyas_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>matyas_bh</td>\n",
       "      <td>2</td>\n",
       "      <td>0.008007</td>\n",
       "      <td>82.0</td>\n",
       "      <td>3.349327e-13</td>\n",
       "      <td>[1.297370838707247e-06, 1.7815516410070865e-07]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.309546e-06</td>\n",
       "      <td>3.349327e-13</td>\n",
       "      <td>matyas</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>matyas_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>matyas_bh</td>\n",
       "      <td>3</td>\n",
       "      <td>0.005951</td>\n",
       "      <td>54.0</td>\n",
       "      <td>1.018301e-13</td>\n",
       "      <td>[2.591796298620497e-07, -3.785900221653371e-07]</td>\n",
       "      <td>1</td>\n",
       "      <td>4.588077e-07</td>\n",
       "      <td>1.018301e-13</td>\n",
       "      <td>matyas</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>matyas_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>matyas_bh</td>\n",
       "      <td>4</td>\n",
       "      <td>0.008369</td>\n",
       "      <td>82.0</td>\n",
       "      <td>8.005503e-13</td>\n",
       "      <td>[-1.3233630792589197e-06, 4.5771165845296486e-07]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.400282e-06</td>\n",
       "      <td>8.005503e-13</td>\n",
       "      <td>matyas</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>matyas_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>matyas_bh</td>\n",
       "      <td>5</td>\n",
       "      <td>0.006056</td>\n",
       "      <td>58.0</td>\n",
       "      <td>8.006043e-13</td>\n",
       "      <td>[6.632430225028685e-07, -1.1239134699581044e-06]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.305018e-06</td>\n",
       "      <td>8.006043e-13</td>\n",
       "      <td>matyas</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>matyas_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>matyas_bh</td>\n",
       "      <td>6</td>\n",
       "      <td>0.007150</td>\n",
       "      <td>70.0</td>\n",
       "      <td>1.849580e-14</td>\n",
       "      <td>[-6.804233569934239e-07, -6.795621167506732e-07]</td>\n",
       "      <td>1</td>\n",
       "      <td>9.616551e-07</td>\n",
       "      <td>1.849579e-14</td>\n",
       "      <td>matyas</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>matyas_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>matyas_bh</td>\n",
       "      <td>7</td>\n",
       "      <td>0.007109</td>\n",
       "      <td>66.0</td>\n",
       "      <td>3.398700e-11</td>\n",
       "      <td>[4.590942048465774e-06, -7.058285431460476e-06]</td>\n",
       "      <td>1</td>\n",
       "      <td>8.419985e-06</td>\n",
       "      <td>3.398700e-11</td>\n",
       "      <td>matyas</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>matyas_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>matyas_bh</td>\n",
       "      <td>8</td>\n",
       "      <td>0.007133</td>\n",
       "      <td>74.0</td>\n",
       "      <td>2.927527e-13</td>\n",
       "      <td>[-1.2189360017010196e-06, -1.7323770660858774e...</td>\n",
       "      <td>1</td>\n",
       "      <td>1.231185e-06</td>\n",
       "      <td>2.927527e-13</td>\n",
       "      <td>matyas</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>matyas_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>matyas_bh</td>\n",
       "      <td>9</td>\n",
       "      <td>0.006685</td>\n",
       "      <td>66.0</td>\n",
       "      <td>3.763831e-15</td>\n",
       "      <td>[-3.0621034229126685e-07, -3.0726690395917536e...</td>\n",
       "      <td>1</td>\n",
       "      <td>4.337946e-07</td>\n",
       "      <td>3.763832e-15</td>\n",
       "      <td>matyas</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>matyas_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"10\" valign=\"top\">shaffer2_bh</th>\n",
       "      <th>0</th>\n",
       "      <td>shaffer2_bh</td>\n",
       "      <td>0</td>\n",
       "      <td>0.771252</td>\n",
       "      <td>12833.0</td>\n",
       "      <td>4.741642e-01</td>\n",
       "      <td>[58.30268550332871, 1.4512290916753523e-05]</td>\n",
       "      <td>0</td>\n",
       "      <td>5.830269e+01</td>\n",
       "      <td>4.741642e-01</td>\n",
       "      <td>shaffer2</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer2_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>shaffer2_bh</td>\n",
       "      <td>1</td>\n",
       "      <td>0.962435</td>\n",
       "      <td>16037.0</td>\n",
       "      <td>4.324969e-01</td>\n",
       "      <td>[41.492078635935286, 1.0527103917473452e-05]</td>\n",
       "      <td>0</td>\n",
       "      <td>4.149208e+01</td>\n",
       "      <td>4.324969e-01</td>\n",
       "      <td>shaffer2</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer2_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>shaffer2_bh</td>\n",
       "      <td>2</td>\n",
       "      <td>0.741794</td>\n",
       "      <td>12021.0</td>\n",
       "      <td>4.761784e-01</td>\n",
       "      <td>[-3.301929503868298e-06, 59.84492889690253]</td>\n",
       "      <td>0</td>\n",
       "      <td>5.984493e+01</td>\n",
       "      <td>4.761784e-01</td>\n",
       "      <td>shaffer2</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer2_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>shaffer2_bh</td>\n",
       "      <td>3</td>\n",
       "      <td>1.051098</td>\n",
       "      <td>17843.0</td>\n",
       "      <td>3.630946e-01</td>\n",
       "      <td>[-30.183797114883564, 0.00034492292641253035]</td>\n",
       "      <td>0</td>\n",
       "      <td>3.018380e+01</td>\n",
       "      <td>3.630946e-01</td>\n",
       "      <td>shaffer2</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer2_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>shaffer2_bh</td>\n",
       "      <td>4</td>\n",
       "      <td>0.952352</td>\n",
       "      <td>16530.0</td>\n",
       "      <td>2.422162e-01</td>\n",
       "      <td>[1.5978020490559108e-05, 19.81662742323059]</td>\n",
       "      <td>0</td>\n",
       "      <td>1.981663e+01</td>\n",
       "      <td>2.422162e-01</td>\n",
       "      <td>shaffer2</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer2_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>shaffer2_bh</td>\n",
       "      <td>5</td>\n",
       "      <td>1.040035</td>\n",
       "      <td>17701.0</td>\n",
       "      <td>2.875390e-01</td>\n",
       "      <td>[23.109963756800788, 1.5319062802942453e-06]</td>\n",
       "      <td>0</td>\n",
       "      <td>2.310996e+01</td>\n",
       "      <td>2.875390e-01</td>\n",
       "      <td>shaffer2</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer2_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>shaffer2_bh</td>\n",
       "      <td>6</td>\n",
       "      <td>0.599706</td>\n",
       "      <td>10101.0</td>\n",
       "      <td>4.883051e-01</td>\n",
       "      <td>[-74.42195759146219, -0.00016215578590627363]</td>\n",
       "      <td>0</td>\n",
       "      <td>7.442196e+01</td>\n",
       "      <td>4.883051e-01</td>\n",
       "      <td>shaffer2</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer2_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>shaffer2_bh</td>\n",
       "      <td>7</td>\n",
       "      <td>1.008703</td>\n",
       "      <td>16512.0</td>\n",
       "      <td>3.054629e-01</td>\n",
       "      <td>[6.0244955475754426e-06, 24.559834635572734]</td>\n",
       "      <td>0</td>\n",
       "      <td>2.455983e+01</td>\n",
       "      <td>3.054629e-01</td>\n",
       "      <td>shaffer2</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer2_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>shaffer2_bh</td>\n",
       "      <td>8</td>\n",
       "      <td>0.820075</td>\n",
       "      <td>13939.0</td>\n",
       "      <td>1.604756e-09</td>\n",
       "      <td>[-0.0010503235254215146, 0.0007079650176725939]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.266647e-03</td>\n",
       "      <td>1.604756e-09</td>\n",
       "      <td>shaffer2</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>86</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer2_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>shaffer2_bh</td>\n",
       "      <td>9</td>\n",
       "      <td>0.521098</td>\n",
       "      <td>8368.0</td>\n",
       "      <td>2.547386e-08</td>\n",
       "      <td>[0.0028455581917616645, -0.004158378164620517]</td>\n",
       "      <td>1</td>\n",
       "      <td>5.038781e-03</td>\n",
       "      <td>2.547386e-08</td>\n",
       "      <td>shaffer2</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>75</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer2_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"10\" valign=\"top\">ackley_bh</th>\n",
       "      <th>0</th>\n",
       "      <td>ackley_bh</td>\n",
       "      <td>0</td>\n",
       "      <td>0.112759</td>\n",
       "      <td>1241.0</td>\n",
       "      <td>1.433697e-08</td>\n",
       "      <td>[-4.954063120405926e-09, 1.0727653522963028e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>5.068882e-09</td>\n",
       "      <td>1.433697e-08</td>\n",
       "      <td>ackley</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>bh</td>\n",
       "      <td>ackley_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ackley_bh</td>\n",
       "      <td>1</td>\n",
       "      <td>0.041530</td>\n",
       "      <td>645.0</td>\n",
       "      <td>7.815203e-09</td>\n",
       "      <td>[1.863644864362959e-09, -2.039976951845041e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>2.763092e-09</td>\n",
       "      <td>7.815203e-09</td>\n",
       "      <td>ackley</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>bh</td>\n",
       "      <td>ackley_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ackley_bh</td>\n",
       "      <td>2</td>\n",
       "      <td>0.061931</td>\n",
       "      <td>949.0</td>\n",
       "      <td>2.746191e-09</td>\n",
       "      <td>[4.864087366626489e-10, -8.402984685290087e-10]</td>\n",
       "      <td>1</td>\n",
       "      <td>9.709248e-10</td>\n",
       "      <td>2.746191e-09</td>\n",
       "      <td>ackley</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>bh</td>\n",
       "      <td>ackley_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ackley_bh</td>\n",
       "      <td>3</td>\n",
       "      <td>0.102293</td>\n",
       "      <td>814.0</td>\n",
       "      <td>2.460633e-08</td>\n",
       "      <td>[-4.515094838708027e-09, -7.436253953469113e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>8.699653e-09</td>\n",
       "      <td>2.460633e-08</td>\n",
       "      <td>ackley</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>ackley_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ackley_bh</td>\n",
       "      <td>4</td>\n",
       "      <td>0.041564</td>\n",
       "      <td>666.0</td>\n",
       "      <td>2.879579e-08</td>\n",
       "      <td>[-7.610519870656951e-09, -6.7623678100991716e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.018085e-08</td>\n",
       "      <td>2.879579e-08</td>\n",
       "      <td>ackley</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>ackley_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ackley_bh</td>\n",
       "      <td>5</td>\n",
       "      <td>0.070353</td>\n",
       "      <td>602.0</td>\n",
       "      <td>2.202005e-08</td>\n",
       "      <td>[-7.3135966613803755e-09, -2.6686341669412977e...</td>\n",
       "      <td>1</td>\n",
       "      <td>7.785262e-09</td>\n",
       "      <td>2.202005e-08</td>\n",
       "      <td>ackley</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>ackley_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ackley_bh</td>\n",
       "      <td>6</td>\n",
       "      <td>0.050363</td>\n",
       "      <td>775.0</td>\n",
       "      <td>2.920471e-08</td>\n",
       "      <td>[-7.418626095529172e-09, -7.181805650833529e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.032542e-08</td>\n",
       "      <td>2.920471e-08</td>\n",
       "      <td>ackley</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>bh</td>\n",
       "      <td>ackley_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ackley_bh</td>\n",
       "      <td>7</td>\n",
       "      <td>0.036326</td>\n",
       "      <td>562.0</td>\n",
       "      <td>1.153601e-08</td>\n",
       "      <td>[-1.8423461877202908e-10, -4.074432743762903e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>4.078596e-09</td>\n",
       "      <td>1.153601e-08</td>\n",
       "      <td>ackley</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>ackley_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ackley_bh</td>\n",
       "      <td>8</td>\n",
       "      <td>0.039346</td>\n",
       "      <td>579.0</td>\n",
       "      <td>2.961988e-08</td>\n",
       "      <td>[-7.445366600508773e-09, -7.364348946481484e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.047221e-08</td>\n",
       "      <td>2.961988e-08</td>\n",
       "      <td>ackley</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>bh</td>\n",
       "      <td>ackley_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ackley_bh</td>\n",
       "      <td>9</td>\n",
       "      <td>0.062274</td>\n",
       "      <td>1029.0</td>\n",
       "      <td>9.754501e-09</td>\n",
       "      <td>[1.9685520525171863e-09, -2.831710771537884e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>3.448736e-09</td>\n",
       "      <td>9.754501e-09</td>\n",
       "      <td>ackley</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>bh</td>\n",
       "      <td>ackley_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"10\" valign=\"top\">sphere_bh</th>\n",
       "      <th>0</th>\n",
       "      <td>sphere_bh</td>\n",
       "      <td>0</td>\n",
       "      <td>0.006031</td>\n",
       "      <td>66.0</td>\n",
       "      <td>1.110313e-16</td>\n",
       "      <td>[-7.450590271625731e-09, -7.450600802928666e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.053673e-08</td>\n",
       "      <td>1.110245e-16</td>\n",
       "      <td>sphere</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>sphere_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>sphere_bh</td>\n",
       "      <td>1</td>\n",
       "      <td>0.009492</td>\n",
       "      <td>62.0</td>\n",
       "      <td>2.592949e-17</td>\n",
       "      <td>[4.4637365470556745e-09, 2.449556030237332e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>5.091686e-09</td>\n",
       "      <td>2.592698e-17</td>\n",
       "      <td>sphere</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>sphere_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sphere_bh</td>\n",
       "      <td>2</td>\n",
       "      <td>0.005932</td>\n",
       "      <td>66.0</td>\n",
       "      <td>3.643864e-17</td>\n",
       "      <td>[5.940528624570618e-09, -1.0689918950035349e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>6.035944e-09</td>\n",
       "      <td>3.643810e-17</td>\n",
       "      <td>sphere</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>sphere_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sphere_bh</td>\n",
       "      <td>3</td>\n",
       "      <td>0.006381</td>\n",
       "      <td>62.0</td>\n",
       "      <td>2.462564e-17</td>\n",
       "      <td>[4.241524798054286e-09, 2.575458901699079e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>4.962209e-09</td>\n",
       "      <td>2.463039e-17</td>\n",
       "      <td>sphere</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>sphere_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>sphere_bh</td>\n",
       "      <td>4</td>\n",
       "      <td>0.006092</td>\n",
       "      <td>70.0</td>\n",
       "      <td>8.478296e-17</td>\n",
       "      <td>[-6.518948091001385e-09, -6.5027132547924404e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>9.207712e-09</td>\n",
       "      <td>8.478763e-17</td>\n",
       "      <td>sphere</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>sphere_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>sphere_bh</td>\n",
       "      <td>5</td>\n",
       "      <td>0.005708</td>\n",
       "      <td>62.0</td>\n",
       "      <td>3.752685e-17</td>\n",
       "      <td>[6.41222963615462e-10, 6.091919524386924e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>6.125573e-09</td>\n",
       "      <td>3.752700e-17</td>\n",
       "      <td>sphere</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>sphere_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>sphere_bh</td>\n",
       "      <td>6</td>\n",
       "      <td>0.006584</td>\n",
       "      <td>74.0</td>\n",
       "      <td>1.108500e-16</td>\n",
       "      <td>[-7.444663910234357e-09, -7.444871683269245e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.052849e-08</td>\n",
       "      <td>1.108549e-16</td>\n",
       "      <td>sphere</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>sphere_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>sphere_bh</td>\n",
       "      <td>7</td>\n",
       "      <td>0.007276</td>\n",
       "      <td>66.0</td>\n",
       "      <td>1.110261e-16</td>\n",
       "      <td>[-7.450579590242111e-09, -7.45058459573249e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>1.053671e-08</td>\n",
       "      <td>1.110287e-16</td>\n",
       "      <td>sphere</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>sphere_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>sphere_bh</td>\n",
       "      <td>8</td>\n",
       "      <td>0.006619</td>\n",
       "      <td>66.0</td>\n",
       "      <td>4.087050e-17</td>\n",
       "      <td>[5.979187533977637e-09, -2.2614922712271834e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>6.392576e-09</td>\n",
       "      <td>4.086821e-17</td>\n",
       "      <td>sphere</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>sphere_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>sphere_bh</td>\n",
       "      <td>9</td>\n",
       "      <td>0.005936</td>\n",
       "      <td>58.0</td>\n",
       "      <td>9.823404e-17</td>\n",
       "      <td>[-6.876730351823568e-09, -7.137237689747167e-09]</td>\n",
       "      <td>1</td>\n",
       "      <td>9.911084e-09</td>\n",
       "      <td>9.823412e-17</td>\n",
       "      <td>sphere</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>bh</td>\n",
       "      <td>sphere_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"10\" valign=\"top\">shaffer4_bh</th>\n",
       "      <th>0</th>\n",
       "      <td>shaffer4_bh</td>\n",
       "      <td>0</td>\n",
       "      <td>0.830229</td>\n",
       "      <td>13465.0</td>\n",
       "      <td>4.605263e-01</td>\n",
       "      <td>[-2.5330953670655876e-06, -35.998705127189126]</td>\n",
       "      <td>0</td>\n",
       "      <td>3.725184e+01</td>\n",
       "      <td>1.679473e-01</td>\n",
       "      <td>shaffer4</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer4_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>shaffer4_bh</td>\n",
       "      <td>1</td>\n",
       "      <td>1.011895</td>\n",
       "      <td>14421.0</td>\n",
       "      <td>4.015736e-01</td>\n",
       "      <td>[-0.00016919551876847937, 21.306332943534386]</td>\n",
       "      <td>0</td>\n",
       "      <td>2.005320e+01</td>\n",
       "      <td>1.089946e-01</td>\n",
       "      <td>shaffer4</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer4_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>shaffer4_bh</td>\n",
       "      <td>2</td>\n",
       "      <td>0.304489</td>\n",
       "      <td>4757.0</td>\n",
       "      <td>4.870699e-01</td>\n",
       "      <td>[54.87589890607741, 0.3844966237177956]</td>\n",
       "      <td>0</td>\n",
       "      <td>5.488277e+01</td>\n",
       "      <td>1.944909e-01</td>\n",
       "      <td>shaffer4</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer4_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>shaffer4_bh</td>\n",
       "      <td>3</td>\n",
       "      <td>0.929411</td>\n",
       "      <td>15353.0</td>\n",
       "      <td>3.499296e-01</td>\n",
       "      <td>[13.32289742952959, -1.379454777284802e-06]</td>\n",
       "      <td>0</td>\n",
       "      <td>1.338170e+01</td>\n",
       "      <td>5.735064e-02</td>\n",
       "      <td>shaffer4</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer4_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>shaffer4_bh</td>\n",
       "      <td>4</td>\n",
       "      <td>0.613335</td>\n",
       "      <td>9817.0</td>\n",
       "      <td>4.896876e-01</td>\n",
       "      <td>[-59.092132580596285, -7.075552760174266e-05]</td>\n",
       "      <td>0</td>\n",
       "      <td>5.910542e+01</td>\n",
       "      <td>1.971086e-01</td>\n",
       "      <td>shaffer4</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer4_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>shaffer4_bh</td>\n",
       "      <td>5</td>\n",
       "      <td>0.746043</td>\n",
       "      <td>10701.0</td>\n",
       "      <td>4.056930e-01</td>\n",
       "      <td>[-1.0963346161440763e-06, -22.03124502160071]</td>\n",
       "      <td>0</td>\n",
       "      <td>2.328438e+01</td>\n",
       "      <td>1.131140e-01</td>\n",
       "      <td>shaffer4</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer4_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>shaffer4_bh</td>\n",
       "      <td>6</td>\n",
       "      <td>0.457608</td>\n",
       "      <td>7253.0</td>\n",
       "      <td>4.891813e-01</td>\n",
       "      <td>[-0.4706794602959995, 58.18322368283262]</td>\n",
       "      <td>0</td>\n",
       "      <td>5.693204e+01</td>\n",
       "      <td>1.966023e-01</td>\n",
       "      <td>shaffer4</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer4_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>shaffer4_bh</td>\n",
       "      <td>7</td>\n",
       "      <td>0.904107</td>\n",
       "      <td>14697.0</td>\n",
       "      <td>4.080417e-01</td>\n",
       "      <td>[22.454961953060746, -3.5628694232000132e-06]</td>\n",
       "      <td>0</td>\n",
       "      <td>2.248990e+01</td>\n",
       "      <td>1.154627e-01</td>\n",
       "      <td>shaffer4</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer4_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>shaffer4_bh</td>\n",
       "      <td>8</td>\n",
       "      <td>0.861504</td>\n",
       "      <td>13181.0</td>\n",
       "      <td>4.816823e-01</td>\n",
       "      <td>[48.68605057605636, 6.0523098756563206e-05]</td>\n",
       "      <td>0</td>\n",
       "      <td>4.870217e+01</td>\n",
       "      <td>1.891033e-01</td>\n",
       "      <td>shaffer4</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer4_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>shaffer4_bh</td>\n",
       "      <td>9</td>\n",
       "      <td>0.978297</td>\n",
       "      <td>15277.0</td>\n",
       "      <td>4.336789e-01</td>\n",
       "      <td>[0.0009714580568198287, -27.771581507619164]</td>\n",
       "      <td>0</td>\n",
       "      <td>2.902471e+01</td>\n",
       "      <td>1.410999e-01</td>\n",
       "      <td>shaffer4</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>shaffer4_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"10\" valign=\"top\">easom_bh</th>\n",
       "      <th>0</th>\n",
       "      <td>easom_bh</td>\n",
       "      <td>0</td>\n",
       "      <td>0.046175</td>\n",
       "      <td>505.0</td>\n",
       "      <td>5.783178e-21</td>\n",
       "      <td>[40.65190852158313, 84.10352188550745]</td>\n",
       "      <td>0</td>\n",
       "      <td>8.922924e+01</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>easom</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>easom_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>easom_bh</td>\n",
       "      <td>1</td>\n",
       "      <td>0.046600</td>\n",
       "      <td>505.0</td>\n",
       "      <td>5.004659e-21</td>\n",
       "      <td>[68.94239232430984, -19.71583335908359]</td>\n",
       "      <td>0</td>\n",
       "      <td>6.965779e+01</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>easom</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>easom_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>easom_bh</td>\n",
       "      <td>2</td>\n",
       "      <td>0.051067</td>\n",
       "      <td>505.0</td>\n",
       "      <td>1.236350e-21</td>\n",
       "      <td>[-66.55401448196136, 47.195305153936744]</td>\n",
       "      <td>0</td>\n",
       "      <td>8.245124e+01</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>easom</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>easom_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>easom_bh</td>\n",
       "      <td>3</td>\n",
       "      <td>0.050147</td>\n",
       "      <td>505.0</td>\n",
       "      <td>4.646269e-21</td>\n",
       "      <td>[5.967240705576149, 82.11566060047139]</td>\n",
       "      <td>0</td>\n",
       "      <td>7.902460e+01</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>easom</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>easom_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>easom_bh</td>\n",
       "      <td>4</td>\n",
       "      <td>0.045000</td>\n",
       "      <td>505.0</td>\n",
       "      <td>4.736119e-21</td>\n",
       "      <td>[49.2901589714586, -85.38599098977517]</td>\n",
       "      <td>0</td>\n",
       "      <td>9.983398e+01</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>easom</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>easom_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>easom_bh</td>\n",
       "      <td>5</td>\n",
       "      <td>0.046571</td>\n",
       "      <td>505.0</td>\n",
       "      <td>7.308608e-21</td>\n",
       "      <td>[84.00193753867482, 44.470734004526165]</td>\n",
       "      <td>0</td>\n",
       "      <td>9.081020e+01</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>easom</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>easom_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>easom_bh</td>\n",
       "      <td>6</td>\n",
       "      <td>0.044977</td>\n",
       "      <td>505.0</td>\n",
       "      <td>4.070331e-21</td>\n",
       "      <td>[-3.758121479824666, 26.883940390126753]</td>\n",
       "      <td>0</td>\n",
       "      <td>2.472459e+01</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>easom</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>easom_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>easom_bh</td>\n",
       "      <td>7</td>\n",
       "      <td>0.045514</td>\n",
       "      <td>505.0</td>\n",
       "      <td>5.209061e-21</td>\n",
       "      <td>[28.36036116161709, -28.274930789873775]</td>\n",
       "      <td>0</td>\n",
       "      <td>4.028628e+01</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>easom</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>easom_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>easom_bh</td>\n",
       "      <td>8</td>\n",
       "      <td>0.048220</td>\n",
       "      <td>505.0</td>\n",
       "      <td>7.016806e-21</td>\n",
       "      <td>[11.210395062272148, -51.20192503796223]</td>\n",
       "      <td>0</td>\n",
       "      <td>5.493927e+01</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>easom</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>easom_bh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>easom_bh</td>\n",
       "      <td>9</td>\n",
       "      <td>0.051791</td>\n",
       "      <td>505.0</td>\n",
       "      <td>6.460530e-22</td>\n",
       "      <td>[47.600073359139174, 42.085921768137254]</td>\n",
       "      <td>0</td>\n",
       "      <td>5.910345e+01</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>easom</td>\n",
       "      <td>basin_hopping</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>bh</td>\n",
       "      <td>easom_bh</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      id_val  run      time    reads         value  \\\n",
       "id_val      run                                                      \n",
       "matyas_bh   0      matyas_bh    0  0.008182     62.0  5.138246e-16   \n",
       "            1      matyas_bh    1  0.005991     58.0  3.753407e-16   \n",
       "            2      matyas_bh    2  0.008007     82.0  3.349327e-13   \n",
       "            3      matyas_bh    3  0.005951     54.0  1.018301e-13   \n",
       "            4      matyas_bh    4  0.008369     82.0  8.005503e-13   \n",
       "            5      matyas_bh    5  0.006056     58.0  8.006043e-13   \n",
       "            6      matyas_bh    6  0.007150     70.0  1.849580e-14   \n",
       "            7      matyas_bh    7  0.007109     66.0  3.398700e-11   \n",
       "            8      matyas_bh    8  0.007133     74.0  2.927527e-13   \n",
       "            9      matyas_bh    9  0.006685     66.0  3.763831e-15   \n",
       "shaffer2_bh 0    shaffer2_bh    0  0.771252  12833.0  4.741642e-01   \n",
       "            1    shaffer2_bh    1  0.962435  16037.0  4.324969e-01   \n",
       "            2    shaffer2_bh    2  0.741794  12021.0  4.761784e-01   \n",
       "            3    shaffer2_bh    3  1.051098  17843.0  3.630946e-01   \n",
       "            4    shaffer2_bh    4  0.952352  16530.0  2.422162e-01   \n",
       "            5    shaffer2_bh    5  1.040035  17701.0  2.875390e-01   \n",
       "            6    shaffer2_bh    6  0.599706  10101.0  4.883051e-01   \n",
       "            7    shaffer2_bh    7  1.008703  16512.0  3.054629e-01   \n",
       "            8    shaffer2_bh    8  0.820075  13939.0  1.604756e-09   \n",
       "            9    shaffer2_bh    9  0.521098   8368.0  2.547386e-08   \n",
       "ackley_bh   0      ackley_bh    0  0.112759   1241.0  1.433697e-08   \n",
       "            1      ackley_bh    1  0.041530    645.0  7.815203e-09   \n",
       "            2      ackley_bh    2  0.061931    949.0  2.746191e-09   \n",
       "            3      ackley_bh    3  0.102293    814.0  2.460633e-08   \n",
       "            4      ackley_bh    4  0.041564    666.0  2.879579e-08   \n",
       "            5      ackley_bh    5  0.070353    602.0  2.202005e-08   \n",
       "            6      ackley_bh    6  0.050363    775.0  2.920471e-08   \n",
       "            7      ackley_bh    7  0.036326    562.0  1.153601e-08   \n",
       "            8      ackley_bh    8  0.039346    579.0  2.961988e-08   \n",
       "            9      ackley_bh    9  0.062274   1029.0  9.754501e-09   \n",
       "...                      ...  ...       ...      ...           ...   \n",
       "sphere_bh   0      sphere_bh    0  0.006031     66.0  1.110313e-16   \n",
       "            1      sphere_bh    1  0.009492     62.0  2.592949e-17   \n",
       "            2      sphere_bh    2  0.005932     66.0  3.643864e-17   \n",
       "            3      sphere_bh    3  0.006381     62.0  2.462564e-17   \n",
       "            4      sphere_bh    4  0.006092     70.0  8.478296e-17   \n",
       "            5      sphere_bh    5  0.005708     62.0  3.752685e-17   \n",
       "            6      sphere_bh    6  0.006584     74.0  1.108500e-16   \n",
       "            7      sphere_bh    7  0.007276     66.0  1.110261e-16   \n",
       "            8      sphere_bh    8  0.006619     66.0  4.087050e-17   \n",
       "            9      sphere_bh    9  0.005936     58.0  9.823404e-17   \n",
       "shaffer4_bh 0    shaffer4_bh    0  0.830229  13465.0  4.605263e-01   \n",
       "            1    shaffer4_bh    1  1.011895  14421.0  4.015736e-01   \n",
       "            2    shaffer4_bh    2  0.304489   4757.0  4.870699e-01   \n",
       "            3    shaffer4_bh    3  0.929411  15353.0  3.499296e-01   \n",
       "            4    shaffer4_bh    4  0.613335   9817.0  4.896876e-01   \n",
       "            5    shaffer4_bh    5  0.746043  10701.0  4.056930e-01   \n",
       "            6    shaffer4_bh    6  0.457608   7253.0  4.891813e-01   \n",
       "            7    shaffer4_bh    7  0.904107  14697.0  4.080417e-01   \n",
       "            8    shaffer4_bh    8  0.861504  13181.0  4.816823e-01   \n",
       "            9    shaffer4_bh    9  0.978297  15277.0  4.336789e-01   \n",
       "easom_bh    0       easom_bh    0  0.046175    505.0  5.783178e-21   \n",
       "            1       easom_bh    1  0.046600    505.0  5.004659e-21   \n",
       "            2       easom_bh    2  0.051067    505.0  1.236350e-21   \n",
       "            3       easom_bh    3  0.050147    505.0  4.646269e-21   \n",
       "            4       easom_bh    4  0.045000    505.0  4.736119e-21   \n",
       "            5       easom_bh    5  0.046571    505.0  7.308608e-21   \n",
       "            6       easom_bh    6  0.044977    505.0  4.070331e-21   \n",
       "            7       easom_bh    7  0.045514    505.0  5.209061e-21   \n",
       "            8       easom_bh    8  0.048220    505.0  7.016806e-21   \n",
       "            9       easom_bh    9  0.051791    505.0  6.460530e-22   \n",
       "\n",
       "                                                                 x  found  \\\n",
       "id_val      run                                                             \n",
       "matyas_bh   0    [-1.0781327840561628e-07, -8.349620681837033e-08]      1   \n",
       "            1     [-9.598174804600901e-08, -9.758724177008626e-08]      1   \n",
       "            2      [1.297370838707247e-06, 1.7815516410070865e-07]      1   \n",
       "            3      [2.591796298620497e-07, -3.785900221653371e-07]      1   \n",
       "            4    [-1.3233630792589197e-06, 4.5771165845296486e-07]      1   \n",
       "            5     [6.632430225028685e-07, -1.1239134699581044e-06]      1   \n",
       "            6     [-6.804233569934239e-07, -6.795621167506732e-07]      1   \n",
       "            7      [4.590942048465774e-06, -7.058285431460476e-06]      1   \n",
       "            8    [-1.2189360017010196e-06, -1.7323770660858774e...      1   \n",
       "            9    [-3.0621034229126685e-07, -3.0726690395917536e...      1   \n",
       "shaffer2_bh 0          [58.30268550332871, 1.4512290916753523e-05]      0   \n",
       "            1         [41.492078635935286, 1.0527103917473452e-05]      0   \n",
       "            2          [-3.301929503868298e-06, 59.84492889690253]      0   \n",
       "            3        [-30.183797114883564, 0.00034492292641253035]      0   \n",
       "            4          [1.5978020490559108e-05, 19.81662742323059]      0   \n",
       "            5         [23.109963756800788, 1.5319062802942453e-06]      0   \n",
       "            6        [-74.42195759146219, -0.00016215578590627363]      0   \n",
       "            7         [6.0244955475754426e-06, 24.559834635572734]      0   \n",
       "            8      [-0.0010503235254215146, 0.0007079650176725939]      1   \n",
       "            9       [0.0028455581917616645, -0.004158378164620517]      1   \n",
       "ackley_bh   0     [-4.954063120405926e-09, 1.0727653522963028e-09]      1   \n",
       "            1      [1.863644864362959e-09, -2.039976951845041e-09]      1   \n",
       "            2      [4.864087366626489e-10, -8.402984685290087e-10]      1   \n",
       "            3     [-4.515094838708027e-09, -7.436253953469113e-09]      1   \n",
       "            4    [-7.610519870656951e-09, -6.7623678100991716e-09]      1   \n",
       "            5    [-7.3135966613803755e-09, -2.6686341669412977e...      1   \n",
       "            6     [-7.418626095529172e-09, -7.181805650833529e-09]      1   \n",
       "            7    [-1.8423461877202908e-10, -4.074432743762903e-09]      1   \n",
       "            8     [-7.445366600508773e-09, -7.364348946481484e-09]      1   \n",
       "            9     [1.9685520525171863e-09, -2.831710771537884e-09]      1   \n",
       "...                                                            ...    ...   \n",
       "sphere_bh   0     [-7.450590271625731e-09, -7.450600802928666e-09]      1   \n",
       "            1      [4.4637365470556745e-09, 2.449556030237332e-09]      1   \n",
       "            2     [5.940528624570618e-09, -1.0689918950035349e-09]      1   \n",
       "            3       [4.241524798054286e-09, 2.575458901699079e-09]      1   \n",
       "            4    [-6.518948091001385e-09, -6.5027132547924404e-09]      1   \n",
       "            5        [6.41222963615462e-10, 6.091919524386924e-09]      1   \n",
       "            6     [-7.444663910234357e-09, -7.444871683269245e-09]      1   \n",
       "            7      [-7.450579590242111e-09, -7.45058459573249e-09]      1   \n",
       "            8     [5.979187533977637e-09, -2.2614922712271834e-09]      1   \n",
       "            9     [-6.876730351823568e-09, -7.137237689747167e-09]      1   \n",
       "shaffer4_bh 0       [-2.5330953670655876e-06, -35.998705127189126]      0   \n",
       "            1        [-0.00016919551876847937, 21.306332943534386]      0   \n",
       "            2              [54.87589890607741, 0.3844966237177956]      0   \n",
       "            3          [13.32289742952959, -1.379454777284802e-06]      0   \n",
       "            4        [-59.092132580596285, -7.075552760174266e-05]      0   \n",
       "            5        [-1.0963346161440763e-06, -22.03124502160071]      0   \n",
       "            6             [-0.4706794602959995, 58.18322368283262]      0   \n",
       "            7        [22.454961953060746, -3.5628694232000132e-06]      0   \n",
       "            8          [48.68605057605636, 6.0523098756563206e-05]      0   \n",
       "            9         [0.0009714580568198287, -27.771581507619164]      0   \n",
       "easom_bh    0               [40.65190852158313, 84.10352188550745]      0   \n",
       "            1              [68.94239232430984, -19.71583335908359]      0   \n",
       "            2             [-66.55401448196136, 47.195305153936744]      0   \n",
       "            3               [5.967240705576149, 82.11566060047139]      0   \n",
       "            4               [49.2901589714586, -85.38599098977517]      0   \n",
       "            5              [84.00193753867482, 44.470734004526165]      0   \n",
       "            6             [-3.758121479824666, 26.883940390126753]      0   \n",
       "            7             [28.36036116161709, -28.274930789873775]      0   \n",
       "            8             [11.210395062272148, -51.20192503796223]      0   \n",
       "            9             [47.600073359139174, 42.085921768137254]      0   \n",
       "\n",
       "                         dist         error  function           algo  id  \\\n",
       "id_val      run                                                            \n",
       "matyas_bh   0    1.363647e-07  5.138315e-16    matyas  basin_hopping   0   \n",
       "            1    1.368787e-07  3.753398e-16    matyas  basin_hopping   0   \n",
       "            2    1.309546e-06  3.349327e-13    matyas  basin_hopping   0   \n",
       "            3    4.588077e-07  1.018301e-13    matyas  basin_hopping   0   \n",
       "            4    1.400282e-06  8.005503e-13    matyas  basin_hopping   0   \n",
       "            5    1.305018e-06  8.006043e-13    matyas  basin_hopping   0   \n",
       "            6    9.616551e-07  1.849579e-14    matyas  basin_hopping   0   \n",
       "            7    8.419985e-06  3.398700e-11    matyas  basin_hopping   0   \n",
       "            8    1.231185e-06  2.927527e-13    matyas  basin_hopping   0   \n",
       "            9    4.337946e-07  3.763832e-15    matyas  basin_hopping   0   \n",
       "shaffer2_bh 0    5.830269e+01  4.741642e-01  shaffer2  basin_hopping   0   \n",
       "            1    4.149208e+01  4.324969e-01  shaffer2  basin_hopping   0   \n",
       "            2    5.984493e+01  4.761784e-01  shaffer2  basin_hopping   0   \n",
       "            3    3.018380e+01  3.630946e-01  shaffer2  basin_hopping   0   \n",
       "            4    1.981663e+01  2.422162e-01  shaffer2  basin_hopping   0   \n",
       "            5    2.310996e+01  2.875390e-01  shaffer2  basin_hopping   0   \n",
       "            6    7.442196e+01  4.883051e-01  shaffer2  basin_hopping   0   \n",
       "            7    2.455983e+01  3.054629e-01  shaffer2  basin_hopping   0   \n",
       "            8    1.266647e-03  1.604756e-09  shaffer2  basin_hopping   0   \n",
       "            9    5.038781e-03  2.547386e-08  shaffer2  basin_hopping   0   \n",
       "ackley_bh   0    5.068882e-09  1.433697e-08    ackley  basin_hopping   0   \n",
       "            1    2.763092e-09  7.815203e-09    ackley  basin_hopping   0   \n",
       "            2    9.709248e-10  2.746191e-09    ackley  basin_hopping   0   \n",
       "            3    8.699653e-09  2.460633e-08    ackley  basin_hopping   0   \n",
       "            4    1.018085e-08  2.879579e-08    ackley  basin_hopping   0   \n",
       "            5    7.785262e-09  2.202005e-08    ackley  basin_hopping   0   \n",
       "            6    1.032542e-08  2.920471e-08    ackley  basin_hopping   0   \n",
       "            7    4.078596e-09  1.153601e-08    ackley  basin_hopping   0   \n",
       "            8    1.047221e-08  2.961988e-08    ackley  basin_hopping   0   \n",
       "            9    3.448736e-09  9.754501e-09    ackley  basin_hopping   0   \n",
       "...                       ...           ...       ...            ...  ..   \n",
       "sphere_bh   0    1.053673e-08  1.110245e-16    sphere  basin_hopping   0   \n",
       "            1    5.091686e-09  2.592698e-17    sphere  basin_hopping   0   \n",
       "            2    6.035944e-09  3.643810e-17    sphere  basin_hopping   0   \n",
       "            3    4.962209e-09  2.463039e-17    sphere  basin_hopping   0   \n",
       "            4    9.207712e-09  8.478763e-17    sphere  basin_hopping   0   \n",
       "            5    6.125573e-09  3.752700e-17    sphere  basin_hopping   0   \n",
       "            6    1.052849e-08  1.108549e-16    sphere  basin_hopping   0   \n",
       "            7    1.053671e-08  1.110287e-16    sphere  basin_hopping   0   \n",
       "            8    6.392576e-09  4.086821e-17    sphere  basin_hopping   0   \n",
       "            9    9.911084e-09  9.823412e-17    sphere  basin_hopping   0   \n",
       "shaffer4_bh 0    3.725184e+01  1.679473e-01  shaffer4  basin_hopping   0   \n",
       "            1    2.005320e+01  1.089946e-01  shaffer4  basin_hopping   0   \n",
       "            2    5.488277e+01  1.944909e-01  shaffer4  basin_hopping   0   \n",
       "            3    1.338170e+01  5.735064e-02  shaffer4  basin_hopping   0   \n",
       "            4    5.910542e+01  1.971086e-01  shaffer4  basin_hopping   0   \n",
       "            5    2.328438e+01  1.131140e-01  shaffer4  basin_hopping   0   \n",
       "            6    5.693204e+01  1.966023e-01  shaffer4  basin_hopping   0   \n",
       "            7    2.248990e+01  1.154627e-01  shaffer4  basin_hopping   0   \n",
       "            8    4.870217e+01  1.891033e-01  shaffer4  basin_hopping   0   \n",
       "            9    2.902471e+01  1.410999e-01  shaffer4  basin_hopping   0   \n",
       "easom_bh    0    8.922924e+01  1.000000e+00     easom  basin_hopping   0   \n",
       "            1    6.965779e+01  1.000000e+00     easom  basin_hopping   0   \n",
       "            2    8.245124e+01  1.000000e+00     easom  basin_hopping   0   \n",
       "            3    7.902460e+01  1.000000e+00     easom  basin_hopping   0   \n",
       "            4    9.983398e+01  1.000000e+00     easom  basin_hopping   0   \n",
       "            5    9.081020e+01  1.000000e+00     easom  basin_hopping   0   \n",
       "            6    2.472459e+01  1.000000e+00     easom  basin_hopping   0   \n",
       "            7    4.028628e+01  1.000000e+00     easom  basin_hopping   0   \n",
       "            8    5.493927e+01  1.000000e+00     easom  basin_hopping   0   \n",
       "            9    5.910345e+01  1.000000e+00     easom  basin_hopping   0   \n",
       "\n",
       "                 epoch version      full_id  \n",
       "id_val      run                              \n",
       "matyas_bh   0        1      bh    matyas_bh  \n",
       "            1        1      bh    matyas_bh  \n",
       "            2        1      bh    matyas_bh  \n",
       "            3        1      bh    matyas_bh  \n",
       "            4        1      bh    matyas_bh  \n",
       "            5        1      bh    matyas_bh  \n",
       "            6        1      bh    matyas_bh  \n",
       "            7        1      bh    matyas_bh  \n",
       "            8        1      bh    matyas_bh  \n",
       "            9        1      bh    matyas_bh  \n",
       "shaffer2_bh 0      100      bh  shaffer2_bh  \n",
       "            1      100      bh  shaffer2_bh  \n",
       "            2      100      bh  shaffer2_bh  \n",
       "            3      100      bh  shaffer2_bh  \n",
       "            4      100      bh  shaffer2_bh  \n",
       "            5      100      bh  shaffer2_bh  \n",
       "            6      100      bh  shaffer2_bh  \n",
       "            7      100      bh  shaffer2_bh  \n",
       "            8       86      bh  shaffer2_bh  \n",
       "            9       75      bh  shaffer2_bh  \n",
       "ackley_bh   0       16      bh    ackley_bh  \n",
       "            1        4      bh    ackley_bh  \n",
       "            2        8      bh    ackley_bh  \n",
       "            3        1      bh    ackley_bh  \n",
       "            4        1      bh    ackley_bh  \n",
       "            5        1      bh    ackley_bh  \n",
       "            6        6      bh    ackley_bh  \n",
       "            7        1      bh    ackley_bh  \n",
       "            8        2      bh    ackley_bh  \n",
       "            9        5      bh    ackley_bh  \n",
       "...                ...     ...          ...  \n",
       "sphere_bh   0        1      bh    sphere_bh  \n",
       "            1        1      bh    sphere_bh  \n",
       "            2        1      bh    sphere_bh  \n",
       "            3        1      bh    sphere_bh  \n",
       "            4        1      bh    sphere_bh  \n",
       "            5        1      bh    sphere_bh  \n",
       "            6        1      bh    sphere_bh  \n",
       "            7        1      bh    sphere_bh  \n",
       "            8        1      bh    sphere_bh  \n",
       "            9        1      bh    sphere_bh  \n",
       "shaffer4_bh 0      100      bh  shaffer4_bh  \n",
       "            1      100      bh  shaffer4_bh  \n",
       "            2      100      bh  shaffer4_bh  \n",
       "            3      100      bh  shaffer4_bh  \n",
       "            4      100      bh  shaffer4_bh  \n",
       "            5      100      bh  shaffer4_bh  \n",
       "            6      100      bh  shaffer4_bh  \n",
       "            7      100      bh  shaffer4_bh  \n",
       "            8      100      bh  shaffer4_bh  \n",
       "            9      100      bh  shaffer4_bh  \n",
       "easom_bh    0      100      bh     easom_bh  \n",
       "            1      100      bh     easom_bh  \n",
       "            2      100      bh     easom_bh  \n",
       "            3      100      bh     easom_bh  \n",
       "            4      100      bh     easom_bh  \n",
       "            5      100      bh     easom_bh  \n",
       "            6      100      bh     easom_bh  \n",
       "            7      100      bh     easom_bh  \n",
       "            8      100      bh     easom_bh  \n",
       "            9      100      bh     easom_bh  \n",
       "\n",
       "[150 rows x 15 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "benchmark.results.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#data wrangling imports\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import time\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from IPython.core.display import clear_output\n",
    "from shaolin.core.dashboard import Dashboard\n",
    "from shaolin.dashboards.colormap import ColormapPicker\n",
    "from test_functions import TestFunctions\n",
    "from optimizer import Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class OptBenchmark(Dashboard):\n",
    "    \n",
    "    def __init__(self,\n",
    "                 time=3000,\n",
    "                 rounds=100,\n",
    "                 min_error=-6.,\n",
    "                 min_dist=-3.,\n",
    "                 width=\"145em\",\n",
    "                 height=\"85em\",\n",
    "                 sleep=0.1,\n",
    "                 **kwargs):\n",
    "\n",
    "\n",
    "        self._trans = {'differential_evolution':'de',\n",
    "                       'basin_hopping':'bh',\n",
    "                      }\n",
    "        \n",
    "        test_funcs = TestFunctions(name='test_funcs',mode='interactive')\n",
    "        optimizer = Optimizer(test_funcs.functions['eggholder'],mode='interactive',name=\"optimizer_d\",**kwargs)\n",
    "        self._target_name = optimizer.name\n",
    "        self._height = height\n",
    "        self._width = width\n",
    "        self._filtered_df  = None\n",
    "        dashboard = ['c$N=mpl_plot',\n",
    "                     [optimizer,\n",
    "                      test_funcs,\n",
    "                      [\"r$N=bench_row\",[[\"c$n=bench_col_1\",[[\"r$N=run_row\",[\"btn$d=Run&N=run_button\",\"text$d=Version&val=\"]],\n",
    "                                         \"ip$d=Run&n=run_pb&min=0&max=\"+str(int(rounds))+\"&val=0\",\n",
    "                                         \"ip$d=Function&n=func_pb&min=0&max=\"+str(len(test_funcs.functions.keys()))+\"&val=0\",\n",
    "                                         \n",
    "                                         \"(0.,3600,1,\"+str(int(time))+\")$d=Max time\",\n",
    "                                         \"(1,1000,1,\"+str(int(rounds))+\")$d=Rounds\"]],\n",
    "                                        [\"c$n=end_col\",[\"###End condition$n=end_title\",\n",
    "                                                        \"[True]$d=Find best&n=best_tog\",\n",
    "                                                        \"[False]$d=Max epochs&n=epoch_tog\",\n",
    "                                                        \"[False]$d=Time&n=time_tog\",\n",
    "                                                        \"[True]$d=Samples&n=sample_tog\"]]\n",
    "                                        ,[\"c$N=found_col\",[\"###Found condition$n=found_title\",\n",
    "                                                           [\"r$n=mini_row\",[\"[False]$d=Distance&n=dist_tog\",\n",
    "                                                            \"[True]$d=Value&n=val_tog\"]],\n",
    "                                                            \"(-20.,100.,0.1,\"+str(float(min_error))+\")$d=Min error\",\n",
    "                                                            \"(-20.,100.,0.1,\"+str(float(min_dist))+\")$d=Min distance\",\n",
    "                                                            \"True$d=Log10 error\",\n",
    "                                                           ]\n",
    "                                          ]\n",
    "                                           \n",
    "                       ,[\"r$N=filters_row\",[[\"c$n=mem_col\",[\"text$d=File name&n=save_name\",\n",
    "                                                            \"togs$d=Save&n=val_every&o=['Round','Function','Algorithm']\",\n",
    "                                                            \"(1,250,1,1)$d=Every&n=num_every\"]],\n",
    "                                                          \"selmul$d=Algorithms&N=algos_sel&o=\"+str(optimizer.algos)]]]],             \n",
    "                      'togs$n=control_togs&o=[\"Benchmark\",\"Optimizer\",\"Test functions\"]','html$N=plot'\n",
    "                      \n",
    "                     ]\n",
    "                    ]\n",
    "        Dashboard.__init__(self, dashboard, mode='interactive', **kwargs)\n",
    "        #default benchmark functions\n",
    "        self.set_default_benchmark()\n",
    "        #self.show_filters.observe(self._toggle_filters)\n",
    "        self._html_display_value = ''\n",
    "        #layout fixes\n",
    "        self.plot.widget.layout.width = \"100%\"\n",
    "        self.plot.target.layout.width = \"100%\"\n",
    "        self.version.widget.layout.width = \"12em\"\n",
    "        self.version.target.layout.width = \"12em\"\n",
    "      \n",
    "        self.control_togs.observe(self.update_layout)\n",
    "        self.dist_tog.observe(self._update_error_layout)\n",
    "        self.val_tog.observe(self._update_error_layout)\n",
    "        self.run_button.observe(self.run_benchmark)\n",
    "        self.optimizer.run_btn.visible = False\n",
    "        self._update_error_layout()\n",
    "        self.update_layout()\n",
    "        self.reset_data()\n",
    "        self.run_details = {}\n",
    "\n",
    "    def set_default_benchmark(self):\n",
    "        \"\"\"Defaults settings for the test functions\"\"\"\n",
    "        vals = list(self.test_funcs.wiki.func_sel.target.options)\n",
    "\n",
    "        vals.pop(vals.index('bukin6'))\n",
    "        self.test_funcs.wiki.func_sel.value = tuple(vals)\n",
    "        self.test_funcs.multi.ras_tog.value = False\n",
    "        self.test_funcs.multi.dv2_tog.value = False\n",
    "        self.test_funcs.multi.ras_dim.value = (30, 61)\n",
    "        self.test_funcs.multi.ras_step.value = 30\n",
    "        self.test_funcs.multi.ros_tog.value = False\n",
    "        self.test_funcs.multi.ros_dim.value = (30, 61)\n",
    "        self.test_funcs.multi.ros_step.value = 30\n",
    "        self.test_funcs.multi.update()\n",
    "        self.test_funcs.update()\n",
    "        \n",
    "    \n",
    "    def reset_pbar(self):\n",
    "        \"\"\"Reset progress bar\"\"\"\n",
    "        if self.rounds.value < self.run_pb.value:\n",
    "            self.run_pb.value = self.rounds.value \n",
    "        self.run_pb.target.max = self.rounds.value \n",
    "        self.run_pb.value = 1\n",
    "        if len(self.funcs.keys()) < self.func_pb.value:\n",
    "            self.func_pb.value = len(self.funcs.keys())\n",
    "        self.run_pb.target.max = len(self.funcs.keys())\n",
    "        self.run_pb.value = 1\n",
    "        \n",
    "    \n",
    "            \n",
    "    def _update_error_layout(self,_=None):\n",
    "        \"\"\"Visual updating of the layout\"\"\"\n",
    "        if self.dist_tog.value:\n",
    "            self.min_distance.visible = True\n",
    "        else:\n",
    "            self.min_distance.visible = False\n",
    "        if self.val_tog.value:\n",
    "            self.min_error.visible = True\n",
    "        else:\n",
    "            self.min_error.visible = False\n",
    "    \n",
    "    def compute_error(self):\n",
    "        \"\"\"Compute error values for the benchmark functions\"\"\"\n",
    "        fo_val = False\n",
    "        fo_dist = False\n",
    "        error = np.nan\n",
    "        dist = np.nan\n",
    "        error = self.optimizer.best[1]-self.optimizer.pot_func.benchmark[1]\n",
    "         \n",
    "        if self.val_tog.value:\n",
    "            if self.log10_error.value:\n",
    "                fo_val = error<10**self.min_error.value\n",
    "            else:\n",
    "                fo_val = error<self.min_error.value\n",
    "        if not self.optimizer.best[0] is None:\n",
    "            dist = np.linalg.norm(self.optimizer.pot_func.benchmark[0]-self.optimizer.best[0])\n",
    "        else:\n",
    "            dist = 1e10 \n",
    "        dist = max(dist,1e-10)\n",
    "        if self.dist_tog.value:\n",
    "            if self.log10_error.value:\n",
    "                fo_dist = dist<10**self.min_distance.value\n",
    "            else:\n",
    "                fo_dist = dist<self.min_distance.value        \n",
    "        found = fo_val or fo_dist\n",
    "        assert(not np.isnan(error))\n",
    "        return error,dist,found\n",
    "    \n",
    "    \n",
    "    def end_condition(self):\n",
    "        \"\"\"Callback that returns true when the finishing conditions are met\"\"\"\n",
    "        end = False\n",
    "        if self.best_tog.value:\n",
    "            end = self.compute_error()[2]#best found\n",
    "        #max epoch reached\n",
    "        if self.epoch_tog.value:\n",
    "            end = end or self.optimizer.epoch >= self.optimizer.max_epoch.value\n",
    "        #max reads reached\n",
    "        if self.sample_tog.value:\n",
    "            end = end or self.optimizer.pot_func.n_reads > self.optimizer.max_reads.value\n",
    "        #max time exceded\n",
    "        if self.time_tog.value:\n",
    "            end = end or time.time()-self._run_start > self.max_time.value\n",
    "        return end\n",
    "    \n",
    "    def reset_data(self):\n",
    "        \"\"\"Restarts the data collection process\"\"\"\n",
    "        funcs = list(self.funcs.keys())\n",
    "        ids = []\n",
    "        for suf in self._trans.values():\n",
    "            for f in funcs:\n",
    "                n = f+'_'+suf\n",
    "                ids.append(n+'_'+self.version.value if self.version.value !='' else n)\n",
    "        self._id_vals = ids\n",
    "        runs = np.arange(self.rounds.value).tolist()\n",
    "        res_ix = pd.MultiIndex.from_product([ids,runs])\n",
    "        res_ix.levels[0].name = 'id_val'\n",
    "        res_ix.levels[1].name = 'run'\n",
    "        self.results = pd.DataFrame(index=res_ix,columns=['time','reads','value','x','found','dist','error','function','algo','id','epoch'])\n",
    "        self.results = self.results.reset_index()\n",
    "        self.results = self.results.fillna(0)\n",
    "        self.results.index = res_ix\n",
    "\n",
    "    def update_layout(self,_=None):\n",
    "        \"\"\"Updating of GUI\"\"\"\n",
    "        if self.control_togs.value == 'Optimizer':\n",
    "            self.test_funcs.visible = False\n",
    "            self.optimizer_d.visible = True\n",
    "            self.bench_row.visible = False\n",
    "        elif self.control_togs.value == 'Test functions':\n",
    "            self.test_funcs.visible = True\n",
    "            self.optimizer_d.visible = False\n",
    "            self.bench_row.visible = False\n",
    "        elif self.control_togs.value == 'Benchmark':\n",
    "            self.test_funcs.visible = False\n",
    "            self.optimizer_d.visible = False\n",
    "            self.bench_row.visible = True\n",
    "    \n",
    "    @property\n",
    "    def funcs(self):\n",
    "        \"\"\"Selected test functions\"\"\"\n",
    "        return self.test_funcs.functions\n",
    "    \n",
    "    @property\n",
    "    def optimizer(self):\n",
    "        \"\"\"Selected optimizer dashboard\"\"\"\n",
    "        return getattr(self,self._target_name)\n",
    "    \n",
    "    \n",
    "    def run_benchmark(self,_=None):\n",
    "        \"\"\"Test the selected solvers\"\"\"\n",
    "        self.reset_data()\n",
    "        self.reset_pbar()\n",
    "        start= time.time()\n",
    "        for k,(algo,suf) in enumerate(self._trans.items()):\n",
    "            if algo in self.algos_sel.value:\n",
    "                self.func_pb.value = 1\n",
    "                for i in range(self.rounds.value):                   \n",
    "                    for j,(_name,func) in enumerate(self.funcs.items()):\n",
    "                        name = _name+'_'+suf\n",
    "                        full_name = name if self.version.value==''else name+'_'+self.version.value\n",
    "                        version = suf if self.version.value==''else suf+'_'+self.version.value\n",
    "                        \n",
    "                        clear_output(True)\n",
    "                        print(\"Calculating {} round {}\".format(_name,i))\n",
    "                        x,val,reads,elapsed_time = self.run_one(name=algo,func=func)\n",
    "                        self.results.loc[(full_name,i),\"epoch\"] = self.optimizer.epoch\n",
    "                        self.results.loc[(full_name,i),\"version\"] = str(version)\n",
    "                        #full_id is func_version\n",
    "                        self.results.loc[(full_name,i),\"full_id\"] = str(full_name)\n",
    "                        self.results.loc[(full_name,i),\"function\"] = str(_name)\n",
    "                        self.results.loc[(full_name,i),\"algo\"] = str(algo)\n",
    "                        self.results.loc[(full_name,i),\"x\"] = str(x.tolist())\n",
    "                        #noise for avoiding singular matrices when plotting distributions\n",
    "                        self.results.loc[(full_name,i),\"value\"] = (max(min(float(val) if not np.isinf(val) or np.isnan(val) else 1e20,1e20),-1e20))+np.random.random()*1e-20\n",
    "                        self.results.loc[(full_name,i),\"time\"] = float(elapsed_time)\n",
    "                        self.results.loc[(full_name,i),\"reads\"] = int(reads)+np.random.random()*1e-20\n",
    "                        error,dist,found = self.compute_error()\n",
    "                        self.results.loc[(full_name,i),\"error\"] =max(min(float(error),1e20),-1e20)+np.random.random()*1e-20\n",
    "                        self.results.loc[(full_name,i),\"dist\"] = max(min(float(dist),1e20),-1e20)+np.random.random()*1e-20\n",
    "                        self.results.loc[(full_name,i),\"found\"] = int(found)\n",
    "                        self.func_pb.value = j+1\n",
    "                        if self.val_every.value == 'Function' and np.mod(self.num_every.value,j) == 0:\n",
    "                            self.save_data()\n",
    "                    if self.val_every.value == 'Round':\n",
    "                        self.save_data()\n",
    "                    self.run_pb.value = i+1\n",
    "                    subset = self.results['full_id'] == full_name\n",
    "                    self._html_display_value = self.results[subset].describe().to_html()\n",
    "                    self.update_html(name=full_name)\n",
    "                    #time.sleep(0.15)\n",
    "                    \n",
    "        \n",
    "            if self.val_every.value == 'Algorithm' and np.mod(self.num_every.value,k) == 0:\n",
    "                self.save_data()\n",
    "        self.save_data()\n",
    "        self.update_html()\n",
    "        clear_output(True) \n",
    "        \n",
    "    \n",
    "    def save_data(self,path='data/'):\n",
    "        \"\"\"Saves data to pickle\"\"\"\n",
    "        self.results.dropna().to_pickle('{}{}.pck'.format(path,self.save_name.value))\n",
    "        \n",
    "    def update_html(self,_=None, name=''):\n",
    "        \"\"\"Updates the summary results of the optimization\"\"\"\n",
    "        self.plot.value =  name+'<div style=\"width:100%;height:100%\">'+self._html_display_value+'</div>'\n",
    "        \n",
    "    def run_one(self,func,name):\n",
    "        \"\"\"Solve one function with one algorithm\"\"\"\n",
    "        self.optimizer.pot_func = func\n",
    "        self.optimizer.reset()\n",
    "        self._run_start = time.time()\n",
    "        self.optimizer.run(name=name,end_callback=self.end_condition)\n",
    "        x,val,reads = self.optimizer.best[0],self.optimizer.best[1],self.optimizer.pot_func.n_reads\n",
    "        elapsed_time = float(time.time()-self._run_start)\n",
    "        return  x,val,reads,elapsed_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "widgets": {
   "state": {
    "2fc6107afef94ef4874d980fe75f3db4": {
     "views": [
      {
       "cell_index": 3
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}