{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Smart Selectors" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's start by defining a task." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from metricx import Metric, Task\n", "\n", "task = Task(\n", " name=\"task\",\n", " metrics=[\n", " Metric(name=\"score\", is_higher_better=True),\n", " ],\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now suppose we have three models whose performance on the task is distributed as follows." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "models = {\n", " \"model-A\": lambda: np.random.normal(loc=0.0, scale=3.0),\n", " \"model-B\": lambda: np.random.normal(loc=10.0, scale=1.0),\n", " \"model-C\": lambda: np.random.normal(loc=1.0, scale=1.0),\n", " \"model-D\": lambda: np.random.normal(loc=-1.0, scale=0.5),\n", "}\n", "for model, func in models.items():\n", " task.report(model, {\"score\": func()})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The selector provides a method to help you choose the next model to evaluate on this task. It allows you to customize the policy used to select models. The default policy is to start by obtaining 3 samples for each model and then transitioning to randomly choosing between several heuristic policies ranging from standard errors to power analysis." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAACcCAYAAAA3ZnylAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxiUlEQVR4nO3deXyU9Z3A8c93jlyQhJzckIhBIBIDBUQuBRStxyK61rJUoaz1KF2V0oO2YmClUC3rdu2ybbG2shUobUGrrawKiggqSDRFkMihAcKZO4GcM/PbP2YSEsgxSWYySeb7fr3mNTPP83ue5zsPw/Dld4oxBqWUUkop1bEsgQ5AKaWUUioYaRKmlFJKKRUAmoQppZRSSgWAJmFKKaWUUgGgSZhSSimlVABoEqaUUkopFQCahCmlVDNE5AYRyQ10HEqp7keTMKVUtyUiOSJyY6DjUEqpxmgSppRSSikVAJqEKaW6JRH5AzAIeE1EzovID0TkzyJyRkRKRGSHiKTWK3+riHwmImUiclJEvtfEeR/1lBvQUZ9FKdU9aRKmlOqWjDH3AceBO4wxPY0xzwBbgBQgEfgYWFfvkBeAh4wxkcDVwNuXnlNEngTmAdcbY7SfmFKqXWyBDkAppTqKMeZ3ta9FZClQJCLRxpgSoAYYISL/MMYUAUX1DhUReRYYB0z1lFdKqXbRmjClVFAQEauI/ExEjopIKZDj2RXveb4buBU4JiLvish19Q7vBTwIrNQETCnlK5qEKaW6M1Pv9b8AM4EbgWggybNdAIwxHxljZuJuqnwF+FO9Y4uA24Hfi8hE/4aslAoWmoQppbqzs8AVnteRQBVQAEQAK2oLiUiIiMzxNE3WAKWAq/6JjDHbgTnAZhEZ1wGxK6W6OU3ClFLd2UrgCREpBmKBY8BJ4DPgw0vK3gfkeJoqH8adcDVgjHkLmI97xOVoP8atlAoCYoxpuZRSSimllPIprQlTSimllAoATcKUUkoppQJAkzCllFJKqQDQJEwppZRSKgA0CVNKKaWUCoAut2xRfHy8SUpKCnQYSimllFItyszMzDfGJDS2r8slYUlJSezduzfQYSillFJKtUhEjjW1T5sjlVJKKaUCQJMwpZRSSqkA0CRMKXUZ5/kL5C5cSN5zzwU6FKWU6ra6XJ8wpZT/lbz6V8q2/B9lQPTMmYQMHhzokJRSHaimpobc3FwqKysDHUqXERYWxoABA7Db7V4fo0mYUuoyZa9vwRobi7OwkNIt/0f8ww8FOiSlVAfKzc0lMjKSpKQkRCTQ4XR6xhgKCgrIzc0lOTnZ6+O0OVIp1YBxOqn47DOibr2VkKQkKvZ/GuiQlFIdrLKykri4OE3AvCQixMXFtbrmUGvClFINVH/xBaa8nLCrU3EWF1OuU8IoFZQ0AWudttwvrQlTSjVQdfgwAGHDhxM2fBiOM2dwlpQEOCqllGq7pKQk8vPz21zG4XCQkJDA4sWLfRqXJmFKqQaqjx8HIGTQoLoO+dXHTwQyJKWUCqi33nqLoUOH8uc//xljjM/Oq0mYUqqB6pxj2Hr3xhIejn3QIPe2401O+KyUUn6Rk5PDsGHDmDdvHkOHDmXOnDls3bqViRMnkpKSwp49eygsLOTOO+8kLS2N8ePHs2/fPgAKCgqYMWMGqampPPDAAw0Sp5deeolx48aRnp7OQw89hNPpbDGWDRs28NhjjzFo0CA++OADn31GTcKUUg1UHz9OiCf5Chk4EIAaT+2YUkp1pCNHjrBo0SKys7PJzs5m/fr17Ny5k1WrVrFixQoyMjIYNWoU+/btY8WKFdx///0ALFu2jEmTJnHgwAFmzZrFcc9v2MGDB9m4cSO7du0iKysLq9XKunXrmo2hsrKSrVu3cscddzB79mw2bNjgs8+nHfOVUg3UnDpFj/HjAbCEh2NNiKc6NzfAUSmlAmXZawf47FSpT885ol8UGXektlguOTmZkSNHApCamsr06dMREUaOHElOTg7Hjh1j06ZNAEybNo2CggJKS0vZsWMHmzdvBuC2224jJiYGgG3btpGZmcnYsWMBqKioIDExsdkY/va3vzF16lTCw8O5++67eeqpp/jFL36B1Wpt8+evpUmYUqqOcblw5OVh6927bps9sTeOvLwARqWUClahoaF1ry0WS917i8WCw+Fo1cSo4J7Pa+7cuaxcubLJMqtXr+b5558H4PXXX2fDhg3s3LmTpKQkwN3U+fbbb3PTTTe18tNcTpMwpVQdZ2EhOJ3YEhPqttkSE6k5fTqAUSmlAsmbGqtAmTx5MuvWrWPJkiVs376d+Ph4oqKimDJlCuvXr+eJJ55gy5YtFBUVATB9+nRmzpzJwoULSUxMpLCwkLKyMgbXWxVkwYIFLFiwAIDS0lLee+89Tpw4UZcA/v73v2fDhg0+ScK0T5hSqk7N2bOAO/GqZUtMxHHuXKBCUkqpJi1dupTMzEzS0tJYvHgxa9euBSAjI4MdO3aQmprK5s2bGeTp5zpixAiWL1/OjBkzSEtL46abbuJ0M//JfPnll5k2bVqDGrmZM2fy2muvUVVV1e74xZdDLTvCmDFjzF6dPFIpvyh75x1yH/k2SX/aSHhaGgB5q1eT/8v/5qp9/8ASEhLgCJVSHeHgwYMMHz480GF0OY3dNxHJNMaMaay832rCROR3InJORPY3sV9E5DkROSIi+0RktL9iUUp5x3HO3ferfk2Y3dM/zKn9wpRSyqf82Rz5InBLM/u/CqR4Hg8Cv/JjLEopLzjOnQMRbHFxddtqE7IabZJUSimf8lsSZozZARQ2U2Qm8L/G7UOgl4j09Vc8SqmWOc6dxRoXh9QbcVSbhNXWkimllPKNQHbM7w/UXwsl17NNKRUgNefONRgZCfWTMK0JU0opX+oSoyNF5EER2Ssie/O0X4pSfuM4l4c9sXeDbdZevcBux3HubGCCUkqpbiqQSdhJYGC99wM82y5jjFljjBljjBmTkJDQWBGllA84zp5t0CkfQCwWbAnxWhOmlFI+Fsgk7FXgfs8oyfFAiTFGZ4RUKkCM04mzqAhbfNxl+2zxCTjy8gMQlVJKtV9SUhL5+c3/hjVV5oYbbuCqq64iPT2d4cOHs2bNGp/F5bcZ80VkA3ADEC8iuUAGYAcwxvwaeB24FTgClAPf9FcsSqmWOYuLwRisMbGX7bPFxOjoSKVU0Fq3bh1jxoyhsLCQIUOGMG/ePEJ8MG+iP0dHzjbG9DXG2I0xA4wxLxhjfu1JwPCMilxgjBlijBlpjNEZWJUKIKdnWQ9rbMxl+6yxsXX7lVKqI+Tk5DBs2DDmzZvH0KFDmTNnDlu3bmXixImkpKSwZ88eCgsLufPOO0lLS2P8+PHs27cPcK/vOGPGDFJTU3nggQeoPzH9Sy+9xLhx40hPT+ehhx7C6XR6HdP58+fp0aOHTxbvhi7SMV8p5X+OQveMMrbYy2vCrLExOAsL6WorbCilurYjR46waNEisrOzyc7OZv369ezcuZNVq1axYsUKMjIyGDVqFPv27WPFihXcf//9ACxbtoxJkyZx4MABZs2axfHjxwH3jPYbN25k165dZGVlYbVaWbduXYtxzJkzh7S0NK666iqWLFnisyRMF/BWSgHgLPTUhMVcXhNmi4nBVFfjulCOtWePjg5NKRVIWxbDmU99e84+I+GrP2uxWHJyMiNHjgQgNTWV6dOnIyKMHDmSnJwcjh07xqZNmwCYNm0aBQUFlJaWsmPHDjZv3gzAbbfdRoznd23btm1kZmYyduxYACoqKki8ZDBSY2qbI/Py8pgwYQK33HJLg0W/20qTMKUUAM4id01YY33Carc5iwo1CVNKdZj6C2dbLJa69xaLBYfDgb3exNLeMMYwd+5cVq5c2WSZ1atX8/zzzwPw+uuvN9iXkJDA6NGj2b17tyZhSinfqWuOjOl12b7afmLOoiIYOPCy/UqpbsyLGqtAmTx5MuvWrWPJkiVs376d+Ph4oqKimDJlCuvXr+eJJ55gy5YtFHn6tE6fPp2ZM2eycOFCEhMTKSwspKysrEFCtWDBAhYsWNDo9crLy/nkk0/4wQ9+4JP4NQlTSgHgLCrGEhmJNDLip7afWG2ippRSncHSpUuZP38+aWlpREREsHbtWgAyMjKYPXs2qampTJgwgUGDBgEwYsQIli9fzowZM3C5XNjtdlavXt1irdacOXMIDw+nqqqKefPm8ZWvfMUn8UtX62g7ZswYs3evDqRUytdOfncRFQf2c+Ubb1y2r/r4cY7OuJm+K1bQ665ZAYhOKdWRDh48yPDhwwMdRpfT2H0TkUxjzJjGyuvoSKUUAI6iQmyN9AcD9xQVcLHfmFJKqfbTJEwpBbhHRzY2MhLA0qMHYrdrc6RSSvmQJmFKKQCchYWNTtQKICKeCVuLOzYopZTqxjQJU0phjMFRXNzoRK21rLGxOLUmTCmlfEaTMKUUrvPnoaam0TnCatlieuHQPmFKKeUzmoQppepquJpqjgT3hK21s+orpZRqP03ClFL1JmptJgnTRbyVUl1UUlIS+fn5bS6zZcsWxowZw4gRIxg1ahSLFi3ySVyahCml6pIrazN9wmyxMbjOn8dVXd1RYSmlVMDt37+f73znO7z00kt89tln7N27lyuvvNIn59YkTCl1MQlrpk/YxfUjtTZMKeV/OTk5DBs2jHnz5jF06FDmzJnD1q1bmThxIikpKezZs4fCwkLuvPNO0tLSGD9+PPv27QOgoKCAGTNmkJqaygMPPED9ielfeuklxo0bR3p6Og899BBOp7PZOJ555hl+8pOfMGzYMACsViuPPPKITz6jJmFKqYvNkc32CfOsH6kjJJVSHeTIkSMsWrSI7OxssrOzWb9+PTt37mTVqlWsWLGCjIwMRo0axb59+1ixYgX3338/AMuWLWPSpEkcOHCAWbNmcfz4ccA9o/3GjRvZtWsXWVlZWK1W1q1b12wM+/fv99kyRZfyeu1IEQkHBhljPvdLJEqpgHEWFiFhYVgiIposY6u/iLdSKmg8vedpsguzfXrOYbHD+OG4H7ZYLjk5mZEjRwKQmprK9OnTERFGjhxJTk4Ox44dY9OmTQBMmzaNgoICSktL2bFjB5s3bwbgtttuI8bzn8ht27aRmZnJ2LFjAaioqCAxMdGnn601vKoJE5E7gCzg/zzv00XkVT/GpZTqQM7CwiZny69lrVvEW5MwpVTHCA0NrXttsVjq3lssFhwOR6vPZ4xh7ty5ZGVlkZWVxeeff87SpUsblFm9ejXp6emkp6dz6tQpUlNTyczMbNfnaIq3NWFLgXHAdgBjTJaIJPslIqVUh3MUFzU7MhLqrR+pzZFKBRVvaqwCZfLkyaxbt44lS5awfft24uPjiYqKYsqUKaxfv54nnniCLVu2UOSpwZ8+fTozZ85k4cKFJCYmUlhYSFlZGYMHD64754IFC1iwYEHd++9///vcddddTJo0iaFDh+JyuVizZg0PP/xwu+P3NgmrMcaUiEj9baapwkqprsVZWNTsyEgAa1QUWCw6YatSqtNYunQp8+fPJy0tjYiICNauXQtARkYGs2fPJjU1lQkTJjBo0CAARowYwfLly5kxYwYulwu73c7q1asbJGGXSktL4xe/+AWzZ8+mvLwcEeH222/3SfxSf8RAk4VEXgC2AYuBu4FHAbsxpv1pYCuNGTPG7N27t6Mvq1S3dmT6jYR/ZTT9n3mm2XKHrptA5IwZ9F22tGMCU11bTSU4KiG8V6AjUa108OBBhg8fHugwupzG7puIZBpjxjRW3tvRkf8GpAJVwHqgBHi87WEqpToTR1ERtmamp6il60cqr2VtgFUp8HQSvPwIOKoCHZFSnU6LzZEiYgX+boyZCvzE/yEppTqSq7ISU17eYsd8cM+or82RqkVH34ZXHoHBE6HP1bD712APh9ufDXRkSnUqLSZhxhiniLhEJNoYU9IRQSmlOs7F2fJbTsKssbFUHT3q75BUV+Z0wOs/gLgrYc6fISQCLDb44L/hmtkwcGygI1Sq0/C2OfI88KmIvCAiz9U+/BmYUqpjXJyo1ZvmyBhtjlTN27cRCg7D9CfdCRjA1B9DeCzs1JowperzdnTkZs9DKdXNOAtbXjeyli02FmdxMcbpRKxWf4emuhpj4P3noE8aDL/j4vaQHjDuW/Du05B/GOJTAhejUp2IVzVhxpi1wAYg0/NY79mmlOrinJ4+Xt70CbP2igFjcJZozwTViJOZkJcNY/8VGk5pBGO/BdYQ2Pu7wMSmVCfk7Yz5NwCHgdXA/wCHRGSK/8JSSnWU1jVH6oStqhmfvAS2cEi96/J9PRMgZQbs3wyu5hdMVsrXkpKSyM/Pb1OZF198kYSEBEaNGkVKSgo333wz77//vk/i8rZP2H8AM4wx1xtjpgA3A//pkwiUUgHlLCoGqxVLZGSLZWvXj3RoEqYu5XTAZ6+4myHDohovc/XdcP4MHNvVoaEp1V733nsvn3zyCYcPH2bx4sXcddddHDx4sN3n9TYJs9dfuNsYcwiwt/vqSqmAq103Uiwt/xxcrAnT9SPVJU7shooiGN7MTOJDb4GQnu7aMKVakJOTw7Bhw5g3bx5Dhw5lzpw5bN26lYkTJ5KSksKePXsoLCzkzjvvJC0tjfHjx7Nv3z4ACgoKmDFjBqmpqTzwwAPUn5j+pZdeYty4caSnp/PQQw/hdLauZnbq1Kk8+OCDrFmzpt2f0dskbK+I/FZEbvA8ngd02nqlugFHUWGL60bWsnomdHUWaxKmLnFoi7vP15BpTZcJiYArboDDb7k78SvVgiNHjrBo0SKys7PJzs5m/fr17Ny5k1WrVrFixQoyMjIYNWoU+/btY8WKFdx///0ALFu2jEmTJnHgwAFmzZrF8ePHAfeM9hs3bmTXrl1kZWVhtVpZt25dq+MaPXo02dnZ7f583o6OfARYgHu5IoD3cPcNU0p1cd6sG1nLFtML0OZI1YjPt0DSJAhtoVk7ZQZk/w3OHYTeIzomNtUuZ1asoOpg+xOO+kKHD6PPj3/cYrnk5GRGjhwJQGpqKtOnT0dEGDlyJDk5ORw7doxNmzYBMG3aNAoKCigtLWXHjh1s3uyucb3tttuI8fxHc9u2bWRmZjJ2rHu+uoqKChITE1sdvzdLPnrD25owG/Bfxpi7jDF3Ac8BLY5PF5FbRORzETkiIosb2T9PRPJEJMvzeKB14Sul2stZWOjVRK0AEhKCJTJSmyNVQ/mHoeAIXHVry2VTbnI/H37TvzGpbiE0NLTutcViqXtvsVhwOBytPp8xhrlz55KVlUVWVhaff/45S5cubVBm9erVpKenk56ezqlTpxo9zyeffOKTtTW9rQnbBtyIe9JWgHDgTWBCUwd4ljtaDdwE5AIficirxpjPLim60RjznVZFrZTyGaeX60bWssbohK3qEof+z/089OaWy0b1g94j3U2Skx73a1jKN7ypsQqUyZMns27dOpYsWcL27duJj48nKiqKKVOmsH79ep544gm2bNlCkWdlkOnTpzNz5kwWLlxIYmIihYWFlJWVMXjw4LpzLliwgAULFjR5zXfffZc1a9bwzjvvtDt+b5OwMGNMbQKGMea8iES0cMw44Igx5gsAEfkjMBO4NAlTSgWIcThwlpR4NUdYLV0/Ul3mi+0QfxX0GuRd+ZSbYNd/QWUJhEX7NTTVvS1dupT58+eTlpZGREQEa9e6pzDNyMhg9uzZpKamMmHCBAYNcn83R4wYwfLly5kxYwYulwu73c7q1asbJGGN2bhxIzt37qS8vJzk5GQ2bdrUoTVhF0RktDHmYwARGQNUtHBMf+BEvfe5wLWNlLvbM+fYIWChMeZEI2WUUn7gLC4GvFs3spY1NpaaJqroVRByVMOxDyD9X7w/JuUm9xJGX7wLI/7Jf7GpLi0pKYn9+/fXvX/xxRcb3ffKK69cdmxcXBxvvtl4k/e9997Lvffee9n2nJycRsvPmzePefPmeR13a3jbJ+wx4M8i8p6IvAf8EfBFE+JrQJIxJg14C2h0Fn4ReVBE9orI3ry8PB9cVikFrZuotZauH6kaOPUx1FyA5FbM391/DNgjIOc9/8WlVBfgbRKWDIzCPUryLeBzoKWhASeBgfXeD/Bsq2OMKTDGVHne/hb4SmMnMsasMcaMMcaMSUhI8DJkpVRL6taNbEWfMFtsLI7iYp+NDlJd3JfvAeIeGektWwgMvNZzrFLBy9skbIkxphToBUzFPT3Fr1o45iMgRUSSRSQE+Drwav0CItK33tt/Ato//axSymvOwgKgtc2RcVBTg6u01F9hqa7ky3ehz0iI8D6RByB5MuQdhPPauqGCl7dJWO10srcBzxtj/g6ENHeAMcaBu8nyDdzJ1Z+MMQdE5N9FpLYTwKMickBE/oF7DrJ5rf0ASqm2c3ia922tqGG2xce7j21hHTYVBGoq4MSe1jVF1kryHKNNkp2W1na3Tlvul7cd80+KyG9wTzfxtIiE4kUCZ4x5HXj9km1P1nv9I+BH3oerlPIlR34+Yrdj7dXL62NqEzZHXh6hQ4b4KTLVJZzYA84qSL6+9cf2S3cvYZTzHlzdyILfKqDCwsIoKCggLi4OEQl0OJ2eMYaCggLCwsJadZy3SdjXgFuAVcaYYk8z4vdbGaNSqpNxnMvDmhDfqh9ZW2JtEqY1YUHvyx0gVhh8XeuPtdph0HXaL6yTGjBgALm5uehgOO+FhYUxYMCAVh3jVRJmjCkHNtd7fxo43aorKaU6HUdeHrb41g12qWuObOWPszFG/0fd3eS8B/1Ht7xUUVOSp8CRt6D0NET1bbm86jB2u53k5ORAh9HteVsTppTqhhz5+dgHDmy5YD2WyEgkNNSrJOzMhTNsyN7A9hPbOV563D2ILiqJ6wdcz71X3UvfnvoPb5dVVQYnM2HiY20/R/Jk93POTki7xzdxKdWFeNsxXynVDTny8rAlxLfqGBHBlpDQbMd8p8vJmn1ruHXzrfzvgf+lT48+3J96P/eNuI/48Hh+f+D33PHKHfzmH7/B6XI2eR7ViR3/EFwOSJrc9nP0SXPPmJ+zw3dxKdWFaE2YUkHKVFe7141sZXMkuDvnN1UTVlJVwsLtC/nozEfcnHQzi76y6LIar1PnT/Fs5rP8d9Z/88m5T3h6ytNEh+ryNV3KF9vBGgKDxrf9HBYrDJ7o7lumVBDSmjClgpSjwD1HWGump6jVVBKWX5HP/Dfmk3Uui6cmPsXPp/y80SbHfj37ser6VTx53ZPsPrOb+W/Mp6iyqPUfQgXOlzvcE67aw9t3nqTJUJQDxbpinQo+moQpFaRqmxPblITFx1/WHFlaXcq33vwWJ8pOsHr6au688s4WO+LfM/QeVk9fzbHSY3zrzW9RUlXS6lhUAJQXwplP2zY1xaXq+oXpKEkVfDQJUypItWWi1lq2xARcJSW4qtyrjtU4a/juO98lpzSHX077Jdf1837Kggn9JvDc1Oc4WnKU7737PRwuR6vjUR0s5z3AtG2S1kslpkJ4rDZJqqCkSZhSQcpxrjYJa13HfPcxF+cKM8bw1IdPsfvMbpZet5Rr+17b6vNN6D+BJ8c/yYenP+TnH/281cerDvblDrD3cE9P0V4WC1xxPRx9B3SGdhVkNAlTKkjVnD4NVmvdvF+tYevdBwDHmdO89sVrvHzkZR5Me5CZV85sczyzUmYxd8Rc1mevZ/PhzS0foALnyx0weIJ7wlVfuGIqnD8Dedm+OZ9SXYQmYUoFqZpTp7D37o3YWj9I2t6/HwCnj+zjpx/+lDG9x/Dta77d7pgWfmUh4/uOZ+XulRwtPtru8yk/KD0F+Yd80xRZa8hU9/PRt313TqW6AE3ClApSNadOYe/fv03H2vu5k7A3PvwDNouNlZNXYrVY2x2T1WJl5eSVRNgj+N6736PSUdnucyofq11m6AofdMqv1WsQxKW4mySVCiKahCkVpGpOnqxLplrLEhpKVXQ4rtNnWTphKX169PFZXPHh8ayYtIIjxUdYtXeVz86rfOTo2xAeA71H+va8Q6a6Z853VPn2vEp1YpqEKRWETE0NjnPn6poVW2vP6T0cj6gktSaRmwbf5OPoYGL/icxLncfGzzey7dg2n59ftZHL5V7r8cob3R3qfWnINHBUwIndvj2vUp2YJmFKBaGas2fB5WpTTVhxZTE/2vkjyuN7MOBCqB+ic3t01KOMiBtBxgcZnL1w1m/XUa1w6mMoL4CUGb4/d9IksNi0X5gKKpqEKRWEak6eAmh1nzBjDEs/WEphZSHXjLwR56nTGJfLHyFit9p5evLTVDur+cnOn+Ay/rmOaoXDbwLirgnztdBIGDAOjmjNpwoemoQpFYRqTnmSsFbWhP3l8F/Ydnwbj49+nN5DrnY3azazkHd7JUUnsXjcYnaf2c2LB17023WUlw69AQPGQkSsf84/9GY4s0+XMFJBQ5MwpYJQzcmTANj6Xr6uY1O+KP6CZ/Y8w4R+E7hvxH11CVztufxl1pWzuGnwTfzy419yoOCAX6+lmlF2Fk5nwVA/NEXWGn6H+zn77/67hlKdiCZhSgWh6pwcbP36YgkJ8ap8lbOK7+/4PuG2cJZPXI5FLIQMHuw51zF/hoqIkHFdBnHhcfxwxw8pryn36/VUE7Jfcz8P/ar/rhE3BBJHwMHX/HcNpToRTcKUCkJVR44QeuWVXpdf9dEqDhUdYvmk5SREuJcsChk4EOx2qo8e8VeYdaJDo1k5eSXHS4/z9EdP+/16qhH7N0P8VdA71b/XGXY7HH8fLvivmVupzkKTMKWCjHE6qf7iC0KvTPGq/NZjW/nj539k7oi5TBlwcZZ0sdsJTUqi6rD/kzCAsX3GMv/q+Ww+vJm3jr3VIddUHqWn4Nj7cPXdIOLfaw2/A4xLmyRVUNAkTKkgU3PiBKa6mtAhQ1ose+r8KZ58/0mujruax0Y/dtn+0JQrqTrSMUkYwIL0BaTGpbL0/aWcuXCmw64b9A68Ahi4+i7/X6vPSIgdAvs2+v9aSgWYJmFKBZnapCk0pfnmyApHBY+/8zjGGJ65/hnsjSzWHHLlldScPImrvGP6admtdp6e8jQOl4OF7yykyqmzq3eI/X+BPmkQ713tabuIwKhvwLFdkN9xCb5SgaBJmFJBpjYJC7mi6ZowYwwZuzLILszmZ5N/xsDIgY2WCx3iTuSqjn7h+0CbMDhqMCsmrWB/wX6Wvb8MY0yHXTsonfkUTmbCNV/vuGum/wuIFT75Q8ddU6kA0CRMqSBTdegwtn59sfbs0WSZ3376W7bkbOHR0Y9y/cCmF2qurU2rOnTI53EaY3C5DA6ni2qHi8oaJ9UOF8YYpg+ezrfTv81rX7ym84f520cvgC0MrpndcdeM7OOeM+wfG8BZ03HXVaqD2QIdgFKqY1V8+inhVze9+PJfDv2F5z55jluTb+Vfr/7XZs8VkpSEJSqK8k8+ptfd7v5CFdVO8s9XUVReTVF5DcXl1RRduPi6sLyGC1UOyqsdVNS4qKh2UF7tpLLGSXm1O9FyGkNzFVx2q2C3Dsba5xqezXyW/9l2lhjXdfQMtdEzzE5kmI3IUBs9Q21EhtnpFWEnvmcocT1DiO8ZQlyPUKLD7Vgsfu5k3tWVF8K+P7k75PtrgtamjJ4Ln78On/0VRv5zx15bqQ6iSZhSQcRRUEDNiRPEfL3xpqU3c97kqQ+fYmL/iSyfuBypNxLOGEPe+SpOFVdytvTiY2TfFCK37mRe3LucKamktNLR5PWjwmzE9AghMsxGhN1GdLidvlFhRIRYCQ+xEm63Emq3YBVBRLCIYBGwWAQRMAaqHS5qnO5HpeN77Cx7moJeG+hliSW0+hpKKmo4WVROWaWD81XuBK8xVosQ2yOEuB4h9IkOo290GH2jwy8+9wqjX3Q44SHW9t30ruzDX0HNBbhuQcdfO2UGJAyD9/4DUu/y/YLhSnUCmoQpFUTKP/oIgPDRoy7b99rR11iy60lSolO5NXExa98/wfHCck4UlnOiqILconIqaxqu32izCN+IHMS9n2dydWg1143qT2JUGAmRocREhBATYaeX5zk63I7N6vt/SMtrXuBbb36Lzwp/xU+n/pRbr7i1wX6H00VxRQ0F56spOF9F/gX3c8H5agouVJFXVs2Z0gr2nywh/3z1ZefvFWGnT1QYA2MjGBgTwaDYcAbGRjAoNoIBMRHdN0mrKILdv3HP2+XvucEaY7HApO/Cyw/CwVch9c6Oj0EpP9MkTKkgUvreLujRg92hvcn9IIcTRRWcKCznwIXXKArbhOPCEPZm383eD9zLA0WG2RgYE8GQhB7cMDSBgbER9O8VTp/oMHpHhRHXI4SqAwPJuedlMgZXE33b1R3+mSLsEfz6pl/z6NuP8sP3fkhRVRFzhs+p22+zWojvGUp8z1AgstlzVdY4OVtayemSSk6XVHCquJIzJZWcKq7geEE5Ow/nU1HTsGYtITKUgTHhDIqNcCdqtclaXAR9osKwdtUmz7d/CtVlcMOPAhfD1XfDrv+Ct5bA0FvAHha4WJTyA+lqI4vGjBlj9u7dG+gwlOqUnC7DmdJKjheUc6KonFxPLdbxwnJy88/zH5uWcDBmMD+9di4AoXYHUQNeozJsN/3s13Jn/++RFBtdl0hER1w+LcWljMPBoQkTiZx6A/2eDtxs9lXOKn7w7g94+8TbzBwykx9f+2Mi7BE+vYYxhvzz1Zwo8tQQFpZ7agvd9/h0SQWuej+pdqvQr1c4A2MiGBgbzoCY2iTNXZsW1yOkQZNvp5GbCS/cCGMfgFt/HthYvtwBa+9w14rdmBHYWJRqAxHJNMaMaWyf1oQp1YUYYygqr6nXTOhOAGpfnyquoMZ5MQuwCPSNDmdgbDhfC8knrrKU5Hv+iU23XUex8xDP7VvJlyVf8vA1D/Nw2sNYLa1vWhObjcgbb6TsjTdwVVRgCQ/35Uf2Wqg1lGdveJZf/eNXrNm3hk/zP2X5xOWMTGh6EEJriQgJkaEkRIYyelDMZftrnC5OFVfUJWXHC8vJLXInwm8eOEvBhYbNneF2KwNja5O0CAbE1CZq7iQtKqzlJNjnKorhL9+EqAEw9Scdf/1LJU+BUffBzv+EK66HK24IdERK+YzWhCnVSRhjKK1wcKqkwt0EVlLB6eKLTWO1z5f2y4rrEcIAT+3KoHrNYQNjw+nXKxy7px9W7qOPcWH3biL/voHfHHqRzYc3069HPzImZDCh34R2xX5hzx6O3z+XPsuWEXPv19p1Ll/48PSH/Pi9H5NXkcesK2fxb6P+rW7Ny0C6UOUgt6iiYQLtqVXLLargfFXDQQ3R4Xb3n6NnwEBvzwCC3lHuwQN9osJ82yetpgJeuhtO7IZvboGB43x37vaovgBrpkLZGfjm69Cn45u9lWqr5mrC/JqEicgtwH8BVuC3xpifXbI/FPhf4CtAAXCvMSanuXNqEqa6miqHk4Lz1eSVVZF/3v1wv3ZvyztfRX5ZFWdKKy8byWe1CL0jQ92j93qF088zcs/d98hdg9IjtOUK7fJPP+XYPV/jwG3D+Wn6l2DgGyO+wSPXPOKTJjtjDDn3fA1HQQFD/v43LBG+bQZsiws1F/jNP37DHz77AyLC7VfczpzhcxgaM7RTNgEaYyipqGmQmNUmaqc9iXljI0+jw90DB/pEh9EnKoz4SPcUHHE96z33DCE2IqT5gRFlZ2DjNyB3L9z92843LUTxcXjhZvdozXtehCHTAh2RUl4JSBImIlbgEHATkAt8BMw2xnxWr8y3gTRjzMMi8nVgljHm3ubOq0mY6mjGGCpqnFyoclJe7eBClZOyyhqKK2ooKa+hpKKG4opqij2vax/FnnmxmpqyITLMRkKku8N4Qs9QekeF0a+Xp4Yj2v06oWdom0cUVjgqyDqXxSfZ27n6yQ1YK2tY/HAEXx15N99M/SZ9e/Ztz225TPnevRy7734ib7mZ/k8/jYSE+PT8bXWi9ARrP1vLX4/8lUpnJcnRydw46EbG9R3HNQnXEG4LTPNpW5RXOzhT4h4scMYzgOBsveczJZUUXKjG6Wr8dz0mwk5cz1Bie4TQK9xOVLidOLuDCSV/Y/yJ57EaB59d+wyVKbcTFe6eQqRnqI2IEFvnGGBQdAzW3wt5ByH9GzDp8Y5ZSkmpdghUEnYdsNQYc7Pn/Y8AjDEr65V5w1PmAxGxAWeABNNMUJqEdS/GGFzG3aHc6TI4jbn42mVwXfLeWTuLer39Dpd7W22ZKqeLqhoX1Z6Z1qsc7glAax9Vjob7qjzbyqvcCdYFz+ShFzxzTF2odjQ7cSi4a6x6hduJ9kzF0CvcPTVDdLiduB4hxEe6E614T3+iuB4hhNlb34zkMi4qHZWUO8qpcFRQ6aikrLqMvIo88srzyKvI40TZCb7IO0TFiWOkfeni7l0uImos5D41n4lffYDo0Og2/mm1rOCFFzj381WEpaURN38+4aNGYYuLRWyB735aXFnMm8fe5I2cN9h7di8u48ImNgZGDWRI9BCSopNICE8gPjyehIgEeth7EG4LJ9wWToQtgjBbGBbp/HNVuVyG0soa8j1TchTUTslRVk55aT7VpQXYy3JJrDjK0OoDjHX+gwipYpczlSWOb/KF6dfoeUNtFiJCrESE2AgPsdLDM7db7fsIu5WIECshNgt2q4UQm+dhbfhcf1+o1YLdZsFqEWwW97xwVov7UfvaZhEsFsEqgsUC1ppyen7wc0I+fgFxVuHqPxZzxQ3QOxWJvQLpmYCERkFID/c6lEoFWKCSsH8GbjHGPOB5fx9wrTHmO/XK7PeUyfW8P+opk9/UeYf3CDe/G57U/MW9+Ehe/dWsf54mDhAf3j5fnaujz9PUvWxw+3z1Z+KFjv4zafV3qT3nwbuYbPW6jYUkOOh/bSVhsZ6Nbf47791xJV/aOfdxOI7yiwmLxW4QS73jPR9WBN/9wdfy4vMZEaqBGhEcAg4Ep5dxCFx2K7r0P/W1fwi1H8I0eGqWabJgx/Q1FkzXvvcqKEzMzO7aoyNF5EHgQYAreoVzMumSuWLa+7dQmnzThuMbZ1osJnXlWjqtkeb3u3e2HFTttZoraaSp/XJZuUb3XPKBLv1plkteu88jddulsfPW2yiXnETq7Rfco9kuO1e9/V7xqqA0+9br41s4zgLYsGATCzYRbGIhBAsRYiPCYiPMsxys2K3YYyMIHxRDaJ+o1gbTTJgtf7DosRB1t4uKnEIqT5bgvFCFs7wGXMa92LahLlEyTTSbteZ6vmIMVOOk0rgfDtzLJzmMcb/GHb/B8xGo/xrPls5MQCxgsYLFhrGFuBfJ9iP3H7d7Cara//C7AAy4ar8D5mIyV/8OXrbNXPz9MJe8x7iwmBqsLgeCCzEuBFe9hPzieaWxP6VWJJ5KtVpm07v8mYSdBAbWez/As62xMrme5sho3B30GzDGrAHWgLs58p//os2RSnVmAkR4HkopFdSa+c+kPzs4fASkiEiyiIQAXwdevaTMq8Bcz+t/Bt5urj+YUkoppVR34beaMGOMQ0S+A7yBe4qK3xljDojIvwN7jTGvAi8AfxCRI0Ah7kRNKaWUUqrb82ufMGPM68Drl2x7st7rSuAef8aglFJKKdUZdbkZ80UkDzjWhkPjgSZHXQY5vTeN0/vSNL03TdN70zi9L03Te9O07nBvBhtjGl2yo8slYW0lInubGiIa7PTeNE7vS9P03jRN703j9L40Te9N07r7ven8Mw8qpZRSSnVDmoQppZRSSgVAMCVhawIdQCem96Zxel+apvemaXpvGqf3pWl6b5rWre9N0PQJU0oppZTqTIKpJkwppZRSqtPo9kmYiNwjIgdExCUiY+ptTxKRChHJ8jx+Hcg4O1pT98Wz70cickREPheRmwMVY2cgIktF5GS978mtgY4p0ETkFs9344iILA50PJ2FiOSIyKee70lQr60mIr8TkXMisr/etlgReUtEDnueYwIZY6A0cW+C/ndGRAaKyDsi8pnn36bHPNu79fem2ydhwH7gLmBHI/uOGmPSPY+HOziuQGv0vojICNwrF6QCtwD/I+LnVX47v/+s9z15veXi3Zfnu7Aa+CowApjt+c4ot6me70m3HVLvpRdx/37UtxjYZoxJAbZ53gejF7n83oD+zjiARcaYEcB4YIHnt6Vbf2+6fRJmjDlojPk80HF0Ns3cl5nAH40xVcaYL4EjwLiOjU51YuOAI8aYL4wx1cAfcX9nlKpjjNmBeym6+mYCaz2v1wJ3dmRMnUUT9yboGWNOG2M+9rwuAw4C/enm35tun4S1IFlEPhGRd0VkcqCD6ST6Ayfqvc/1bAtm3xGRfZ5mhG5VFd4G+v1omgHeFJFMEXkw0MF0Qr2NMac9r88AvQMZTCekvzMeIpIEjAJ2082/N90iCRORrSKyv5FHc/9DPw0MMsaMAr4LrBeRqI6JuGO08b4EnRbu06+AIUA67u/MfwQyVtWpTTLGjMbdVLtARKYEOqDOyriH5evQ/Iv0d8ZDRHoCm4DHjTGl9fd1x++NXxfw7ijGmBvbcEwVUOV5nSkiR4GhQLfpUNuW+wKcBAbWez/As63b8vY+icjzwN/8HE5nF3TfD28ZY056ns+JyMu4m24b64sarM6KSF9jzGkR6QucC3RAnYUx5mzt62D+nRERO+4EbJ0xZrNnc7f+3nSLmrC2EJGE2g7nInIFkAJ8EdioOoVXga+LSKiIJOO+L3sCHFPAeP7S15qFe0BDMPsISBGRZBEJwT2I49UAxxRwItJDRCJrXwMz0O/KpV4F5npezwX+GsBYOhX9nQEREeAF4KAx5tl6u7r196bbT9YqIrOAXwIJQDGQZYy5WUTuBv4dqAFcQIYx5rWABdrBmrovnn0/AebjHq3yuDFmS6DiDDQR+QPuJgID5AAP1eufEJQ8w+d/AViB3xljfhrYiALP8x+5lz1vbcD6YL4vIrIBuAGIB84CGcArwJ+AQcAx4GvGmKDroN7EvbmBIP+dEZFJwHvAp7j/TQb4Me5+Yd32e9PtkzCllFJKqc4oaJsjlVJKKaUCSZMwpZRSSqkA0CRMKaWUUioANAlTSimllAoATcKUUkoppQJAkzCllFJKqQDQJEwppZRSKgA0CVNKKaWUCoD/B7bv/K0i/7ZTAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from copy import deepcopy\n", "from metricx import Selector\n", "\n", "selector = Selector(task)\n", "for _ in range(20):\n", " model = selector.propose()\n", " task.report(model, {\"score\": models[model]()})\n", "task.to_figure();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Although the usage of the Selector class is optional, it's highly recommended as it can reduce the number of samples needed to be able to confidently determine the ranking of the models." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }