create_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f1cf8c9c",
   "metadata": {},
   "source": [
    "# CI/CD Automation Using - Git source example\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6c5437d",
   "metadata": {},
   "source": [
    "This example shows how to create a project YAML for CI/CD Automation from a Github source  - this process is equivalent for using tar, zip source "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "659c8398",
   "metadata": {},
   "source": [
    "After this example you will learn how to:\n",
    "* Create a new MLRun project\n",
    "* Connect a project to source\n",
    "* Build a project YAML "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23f2b136",
   "metadata": {},
   "source": [
    "Install mlrun - if does not install use ``pip install mlrun==<mlrun server version>`` or ``sh align_mlrun.sh`` (our default mlrun installer - automatically install the server version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "eb758c84",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlrun"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a96ff86b",
   "metadata": {},
   "source": [
    "MLRun provides you three option to create or loading a project:\n",
    "\n",
    "1. [get_or_create_project](https://docs.mlrun.org/en/latest/api/mlrun.projects.html?highlight=get_or_create_project#mlrun.projects.get_or_create_project) - this method allows you to load a project from MLRun DB or optionally load it from a yaml/zip/tar/git template, or create/import if doesnt exist\n",
    "2. [new_project](https://docs.mlrun.org/en/latest/api/mlrun.projects.html?highlight=new_project#mlrun.projects.new_project) - Create a new MLRun project, optionally load it from a yaml/zip/tar/git template.\n",
    "3. [load_project](https://docs.mlrun.org/en/latest/api/mlrun.projects.html?highlight=load_project#mlrun.projects.load_project) - Load an MLRun project from yaml/zip/tar/git/dir or from MLRun db\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb69b02a",
   "metadata": {},
   "source": [
    "On project creation MLRun create a light project YAML, for example: \n",
    "````\n",
    "kind: project\n",
    "metadata:\n",
    "  name: default\n",
    "  created: '2022-06-30T09:41:05.612000'\n",
    "spec:\n",
    "  functions: []\n",
    "  workflows: []\n",
    "  artifacts: []\n",
    "  desired_state: online\n",
    "status:\n",
    "  state: online\n",
    "````\n",
    "For update project YAML use **projec.save()**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a7afc337",
   "metadata": {},
   "outputs": [],
   "source": [
    "project = mlrun.new_project('new-git-proj',init_git=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f5a1f0e",
   "metadata": {},
   "source": [
    "The source define your source code this source need to include all your project files  - MLRun knows to point to the files in the source address and run the functions based on the source files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6ed5eeaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "source = 'git://github.com/GiladShapira94/load-project-example.git'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f93fe5f",
   "metadata": {},
   "source": [
    "The pull_at_runtime flag will determine if the code is loaded in runtime or added to the image during build. the first (at runtime) option is better for debugging while the secound is better for production. Note that if you choose the 2nd option you'll need to build the function before run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0d1846a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "project.set_source(source=source,pull_at_runtime=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56c80fe7",
   "metadata": {},
   "source": [
    "For functions definations use the [set_function](https://docs.mlrun.org/en/latest/api/mlrun.projects.html?highlight=set_function#mlrun.projects.MlrunProject.set_function) method.\n",
    "\n",
    "The **set_function** method allow you to set the functions attributes in the project YAML, for example: \n",
    "function source (YAML, py, ipynb, function object) , name of the fucntion, function handler, function image and function kind.\n",
    "\n",
    "````\n",
    "project.set_function(\n",
    "    name=\"training\", handler=\"training.model_training\",\n",
    "    image=\"mlrun/mlrun\", kind=\"job\", with_repo=True,\n",
    ")\n",
    "````\n",
    "> Set the with_repo=True to add the entire repo code into the destination container during build or run time. \n",
    "\n",
    "> When using with_repo=True the functions need to be deployed (function.deploy()) to build a container, unless you set project.spec.load_source_on_run=True which instructs MLRun to load the git/archive repo into the function container at run time and do not require a build (this is simpler when developing, for production it’s preferred to build the image with the code)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "441e450a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.function.RemoteRuntime at 0x7f49499ade10>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.set_function(name='nuclio',handler='multi:multi_3',kind='nuclio',image='mlrun/mlrun',with_repo=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7379d588",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f4949971210>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.set_function(name='my_job',handler='my_job.func',kind='job',image='mlrun/mlrun',with_repo=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7c18c7dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f4949971750>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.set_function(name='my_job_with_requirements',handler='my_job_with_requirements.log_df',kind='job',image='mlrun/mlrun',requirements=['pandas'],with_repo=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55f028b8",
   "metadata": {},
   "source": [
    "**Important Note -** Remote functions as Serving need a function YAML as a source or function object\n",
    "Before you creating fucntion YAML you need to the create a fucntion object you can do it with [code_to_function()](https://docs.mlrun.org/en/latest/api/mlrun.html?highlight=code_to_function#mlrun.code_to_function), [new_function()](https://docs.mlrun.org/en/latest/api/mlrun.run.html?highlight=new_function#mlrun.run.new_function).\n",
    "\n",
    "After you creating a function object you can use the [export()](https://docs.mlrun.org/en/latest/api/mlrun.runtimes.html?highlight=export#mlrun.runtimes.BaseRuntime.export) method, For Example:\n",
    "````\n",
    "<function object>.export('./model_training.yaml')\n",
    "````\n",
    "\n",
    "````\n",
    "project.set_function(\n",
    "    func=\"training.yaml\",name='training',with_repo=True,kind='serving')\n",
    "````"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d852f5a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: mlrun&#45;flow Pages: 1 -->\n",
       "<svg width=\"497pt\" height=\"44pt\"\n",
       " viewBox=\"0.00 0.00 496.87 44.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 40)\">\n",
       "<title>mlrun&#45;flow</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-40 492.8747,-40 492.8747,4 -4,4\"/>\n",
       "<!-- _start -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>_start</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"38.5476,-.0493 40.698,-.1479 42.8263,-.2953 44.9236,-.4913 46.9815,-.7353 48.9917,-1.0266 50.9463,-1.3645 52.8377,-1.7479 54.6587,-2.1759 56.4025,-2.6472 58.0628,-3.1606 59.634,-3.7147 61.1107,-4.308 62.4882,-4.9388 63.7625,-5.6054 64.9302,-6.3059 65.9882,-7.0385 66.9343,-7.8012 67.7669,-8.5918 68.4849,-9.4082 69.0878,-10.2481 69.5758,-11.1093 69.9496,-11.9894 70.2102,-12.886 70.3595,-13.7965 70.3997,-14.7186 70.3334,-15.6497 70.1636,-16.5873 69.8937,-17.5287 69.5276,-18.4713 69.0691,-19.4127 68.5225,-20.3503 67.8923,-21.2814 67.1831,-22.2035 66.3996,-23.114 65.5464,-24.0106 64.6285,-24.8907 63.6504,-25.7519 62.617,-26.5918 61.5329,-27.4082 60.4024,-28.1988 59.2299,-28.9615 58.0197,-29.6941 56.7755,-30.3946 55.5012,-31.0612 54.2002,-31.692 52.8757,-32.2853 51.5309,-32.8394 50.1684,-33.3528 48.7908,-33.8241 47.4003,-34.2521 45.9989,-34.6355 44.5886,-34.9734 43.1708,-35.2647 41.7472,-35.5087 40.3189,-35.7047 38.8872,-35.8521 37.4531,-35.9507 36.0175,-36 34.5815,-36 33.146,-35.9507 31.7119,-35.8521 30.2801,-35.7047 28.8519,-35.5087 27.4282,-35.2647 26.0105,-34.9734 24.6001,-34.6355 23.1988,-34.2521 21.8083,-33.8241 20.4306,-33.3528 19.0681,-32.8394 17.7233,-32.2853 16.3989,-31.692 15.0979,-31.0612 13.8236,-30.3946 12.5794,-29.6941 11.3691,-28.9615 10.1967,-28.1988 9.0662,-27.4082 7.982,-26.5918 6.9486,-25.7519 5.9706,-24.8907 5.0526,-24.0106 4.1995,-23.114 3.4159,-22.2035 2.7067,-21.2814 2.0765,-20.3503 1.53,-19.4127 1.0715,-18.4713 .7053,-17.5287 .4355,-16.5873 .2657,-15.6497 .1993,-14.7186 .2395,-13.7965 .3888,-12.886 .6495,-11.9894 1.0232,-11.1093 1.5112,-10.2481 2.1141,-9.4082 2.8321,-8.5918 3.6647,-7.8012 4.6109,-7.0385 5.6689,-6.3059 6.8365,-5.6054 8.1108,-4.9388 9.4884,-4.308 10.9651,-3.7147 12.5362,-3.1606 14.1966,-2.6472 15.9404,-2.1759 17.7614,-1.7479 19.6528,-1.3645 21.6074,-1.0266 23.6176,-.7353 25.6755,-.4913 27.7728,-.2953 29.901,-.1479 32.0515,-.0493 34.2154,0 36.3837,0 38.5476,-.0493\"/>\n",
       "<text text-anchor=\"middle\" x=\"35.2995\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">start</text>\n",
       "</g>\n",
       "<!-- start -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>start</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"139.7461\" cy=\"-18\" rx=\"33.2948\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"139.7461\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">start</text>\n",
       "</g>\n",
       "<!-- _start&#45;&gt;start -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_start&#45;&gt;start</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M69.7887,-18C78.2449,-18 87.405,-18 96.2245,-18\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"96.3586,-21.5001 106.3586,-18 96.3586,-14.5001 96.3586,-21.5001\"/>\n",
       "</g>\n",
       "<!-- double -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>double</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"251.1394\" cy=\"-18\" rx=\"42.4939\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"251.1394\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">double</text>\n",
       "</g>\n",
       "<!-- start&#45;&gt;double -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>start&#45;&gt;double</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M173.1678,-18C181.1824,-18 189.9338,-18 198.5839,-18\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"198.6416,-21.5001 208.6415,-18 198.6415,-14.5001 198.6416,-21.5001\"/>\n",
       "</g>\n",
       "<!-- add3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>add3</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"363.1827\" cy=\"-18\" rx=\"33.5952\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.1827\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">add3</text>\n",
       "</g>\n",
       "<!-- double&#45;&gt;add3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>double&#45;&gt;add3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M293.4982,-18C301.869,-18 310.6732,-18 319.0995,-18\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"319.1359,-21.5001 329.1358,-18 319.1358,-14.5001 319.1359,-21.5001\"/>\n",
       "</g>\n",
       "<!-- end -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>end</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"460.9272\" cy=\"-18\" rx=\"27.8951\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"460.9272\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">end</text>\n",
       "</g>\n",
       "<!-- add3&#45;&gt;end -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>add3&#45;&gt;end</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M397.0946,-18C405.4051,-18 414.3542,-18 422.8458,-18\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"422.9033,-21.5001 432.9032,-18 422.9032,-14.5001 422.9033,-21.5001\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f49543dfe90>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define a serving graph function with 3 steps (*2, +3, echo)\n",
    "serving = mlrun.new_function(\"serving\", kind=\"serving\", image=\"mlrun/mlrun\")\n",
    "graph = serving.set_topology(\"flow\")\n",
    "graph.to(name=\"start\", handler=\"serving.start\") \\\n",
    "     .to(name=\"double\", handler=\"serving.double\")\\\n",
    "     .to(name=\"add3\", handler=\"serving.add3\") \\\n",
    "     .to(name=\"end\", handler=\"serving.end\").respond()\n",
    "\n",
    "# plot the serving graph\n",
    "serving.spec.graph.plot(rankdir=\"LR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2042727a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-08-03 10:58:03,753 [info] function spec saved to path: serving.yaml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.serving.ServingRuntime at 0x7f4949750610>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# save the function definition into a .yaml file and register it in the project\n",
    "serving.export(\"serving.yaml\")\n",
    "project.set_function(\"serving.yaml\", name=\"serving\", with_repo=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "09fd202d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.projects.project.MlrunProject at 0x7f49499e16d0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36035b1f",
   "metadata": {},
   "source": [
    "### Done! \n",
    "**Now you have a project YAML for CI/CD Automation  - Later we will demostrate how to load a project and use this Project YAML**\n",
    "\n",
    "**TIP -** After creating the project YAML Upload/Save it in the source"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}