docs/python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-inference-byoc-gpu-cpu-aks.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ONNX Runtime on Azure Kubernetes Service\n",
    "**Deploying Facial Emotion Recognition (FER+) using Docker Images for ONNX Runtime with TensorRT**\n",
    "\n",
    "This example shows how to deploy an image classification neural network using ONNX Runtime on GPU compute SKUs in Azure. This example makes use of the Facial Expression Recognition ([FER](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data)) dataset and Open Neural Network eXchange format ([ONNX](http://aka.ms/onnxruntime)) on the Azure Machine Learning platform.\n",
    "\n",
    "![End-to-end pipeline with ONNX Runtime](https://raw.githubusercontent.com/vinitra/models/gtc-demo/gtc-demo/E2EPicture.png)\n",
    "\n",
    "#### Tutorial Roadmap:\n",
    "1. Obtain an ***external ONNX model*** for facial emotion recognition (FER+) from the [ONNX model zoo](https://github.com/onnx/models)\n",
    "2. ***Register our model*** in our Azure Machine Learning workspace\n",
    "3. ***Write a scoring file*** and environment file to evaluate our model with ONNX Runtime\n",
    "4. ***Build a container image*** using the ONNX Runtime + TensorRT base image from Microsoft Container Registry (MCR)\n",
    "5. ***Deploy to the cloud*** using an AKS cluster with GPU and use it to make predictions using ONNX Runtime Python APIs\n",
    "\n",
    "**Note:** You can also use the same notebook and code to deploy to a CPU cluster in addition to your GPU cluster. ONNX Runtime APIs remain unchanged across hardware endpoints. The default option for a secondary deployment with CPU is `False`, but you can change the variable below to `True` for a performance comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deploy_with_cpu = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Prerequisites\n",
    "\n",
    "Throughout this tutorial, we will be referring to **ONNX**, a neural network exchange format used to represent deep learning models. With ONNX, AI developers can more easily move models between state-of-the-art tools (CNTK, PyTorch, Caffe, MXNet, TensorFlow) and choose the combination that is best for them. ONNX is developed and supported by a community of partners including Microsoft AI, Facebook, and Amazon. For more information, explore the [ONNX website](http://onnx.ai) and [open source files](https://github.com/onnx).\n",
    "\n",
    "[ONNX Runtime](https://aka.ms/onnxruntime-python) is the runtime engine that enables evaluation of trained machine learning (traditional ML and Deep Learning) models with high performance and low resource utilization. We use the CPU version of **ONNX Runtime** in this tutorial, but will soon be releasing an additional tutorial for deploying this model using ONNX Runtime GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Azure ML SDK and create a new workspace\n",
    "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, please follow the [Azure ML configuration notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb) to set up your environment.\n",
    "\n",
    "### Install additional packages needed for this Notebook\n",
    "You need to install the popular plotting library matplotlib, the image manipulation library opencv, and the onnx library in the conda environment where Azure Maching Learning SDK is installed.\n",
    "\n",
    "```\n",
    "(myenv) $ pip install matplotlib onnx opencv-python\n",
    "```\n",
    "\n",
    "**Debugging tip**: Make sure that to activate your virtual environment (myenv) before you re-launch this notebook using the jupyter notebook comand. Choose the respective Python kernel for your new virtual environment using the Kernel > Change Kernel menu above. If you have completed the steps correctly, the upper right corner of your screen should state Python [conda env:myenv] instead of Python [default]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Obtain a model from the ONNX Model Zoo\n",
    "\n",
    "For more information on the Facial Emotion Recognition (FER+) model, you can explore the notebook explaning how to deploy [FER+ with ONNX Runtime on an ACI Instance](onnx-inference-facial-expression-recognition-deploy.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# urllib is a built-in Python library to download files from URLs\n",
    "\n",
    "# Objective: retrieve the latest version of the ONNX Emotion FER+ model files from the\n",
    "# ONNX Model Zoo and save it in the same folder as this tutorial\n",
    "\n",
    "import urllib.request\n",
    "\n",
    "onnx_model_url = \"https://www.cntk.ai/OnnxModels/emotion_ferplus/opset_7/emotion_ferplus.tar.gz\"\n",
    "\n",
    "urllib.request.urlretrieve(onnx_model_url, filename=\"emotion_ferplus.tar.gz\")\n",
    "\n",
    "# the ! magic command tells our jupyter notebook kernel to run the following line of \n",
    "# code from the command line instead of the notebook kernel\n",
    "\n",
    "# We use tar and xvcf to unzip the files we just retrieved from the ONNX model zoo\n",
    "\n",
    "!tar xvzf emotion_ferplus.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Registering your model with Azure ML\n",
    "\n",
    "### Load Azure ML workspace\n",
    "\n",
    "We begin by instantiating a workspace object from the existing workspace created earlier in the configuration notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check core SDK version number\n",
    "import azureml.core\n",
    "\n",
    "print(\"SDK version:\", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Workspace\n",
    "\n",
    "# read existing workspace from config.json\n",
    "ws = Workspace.from_config()\n",
    "\n",
    "print(ws.name, ws.location, ws.resource_group, sep = '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Register your ONNX model with Azure ML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_dir = \"emotion_ferplus\" # replace this with the location of your model files\n",
    "\n",
    "# leave as is if it's in the same folder as this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.model import Model\n",
    "\n",
    "# register the new model from local folder\n",
    "model = Model.register(model_path = model_dir + \"/\" + \"model.onnx\",\n",
    "                       model_name = \"onnx_emotion\",\n",
    "                       tags = {\"onnx\": \"demo\"},\n",
    "                       description = \"FER+ emotion recognition CNN from ONNX Model Zoo\",\n",
    "                       workspace = ws)\n",
    "\n",
    "# Alternative: uncomment the line below and point to the model file in the workspace's model registry\n",
    "# model = Model(name=\"onnx_emotion\", workspace=ws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional: Displaying your registered models\n",
    "\n",
    "This step is not required, so feel free to skip it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = ws.models\n",
    "for name, m in models.items():\n",
    "    print(\"Name:\", name,\"\\tVersion:\", m.version, \"\\tDescription:\", m.description, m.tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Specify our Score and Environment Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now going to deploy our ONNX Model on AKS with inference in ONNX Runtime. We begin by writing a score.py file, which will help us run the model in our Azure Kubernetes Cluster, and then specify our environment by writing a yml file. You will also notice that we import the onnxruntime library to do runtime inference on our ONNX models (passing in input and evaluating out model's predicted output). More information on the API and commands can be found in the [ONNX Runtime documentation](https://aka.ms/onnxruntime).\n",
    "\n",
    "### Write Score File\n",
    "\n",
    "A score file is what tells our Azure cloud service what to do. After initializing our model using azureml.core.model, we start an ONNX Runtime inference session to evaluate the data passed in on our function calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile score.py\n",
    "import json\n",
    "import numpy as np\n",
    "import onnxruntime\n",
    "import sys\n",
    "import os\n",
    "from azureml.core.model import Model\n",
    "import time\n",
    "\n",
    "def init():\n",
    "    global session, input_name, output_name\n",
    "    model = Model.get_model_path(model_name = 'onnx_emotion')\n",
    "    \n",
    "    # Load the model in onnx runtime to start the session    \n",
    "    session = onnxruntime.InferenceSession(model, None)\n",
    "    input_name = session.get_inputs()[0].name\n",
    "    output_name = session.get_outputs()[0].name \n",
    "    \n",
    "def run(input_data):\n",
    "    '''Purpose: evaluate test input in Azure Cloud using onnxruntime.\n",
    "        We will call the run function later from our Jupyter Notebook \n",
    "        so our azure service can evaluate our model input in the cloud. '''\n",
    "\n",
    "    try:\n",
    "        # load in our data, convert to readable format\n",
    "        data = np.array(json.loads(input_data)['data']).astype('float32')\n",
    "        \n",
    "        # pass input data to do model inference with ONNX Runtime\n",
    "        start = time.time()\n",
    "        r = session.run([output_name], {input_name : data})\n",
    "        end = time.time()\n",
    "        \n",
    "        result = emotion_map(postprocess(r[0]))\n",
    "        \n",
    "        result_dict = {\"result\": result,\n",
    "                      \"time_in_sec\": [end - start]}\n",
    "    except Exception as e:\n",
    "        result_dict = {\"error\": str(e)}\n",
    "    \n",
    "    return json.dumps(result_dict)\n",
    "\n",
    "def emotion_map(classes, N=1):\n",
    "    \"\"\"Take the most probable labels (output of postprocess) and returns the \n",
    "    top N emotional labels that fit the picture.\"\"\"\n",
    "    \n",
    "    emotion_table = {'neutral':0, 'happiness':1, 'surprise':2, 'sadness':3, \n",
    "                     'anger':4, 'disgust':5, 'fear':6, 'contempt':7}\n",
    "    \n",
    "    emotion_keys = list(emotion_table.keys())\n",
    "    emotions = []\n",
    "    for i in range(N):\n",
    "        emotions.append(emotion_keys[classes[i]])\n",
    "    return emotions\n",
    "\n",
    "def softmax(x):\n",
    "    \"\"\"Compute softmax values (probabilities from 0 to 1) for each possible label.\"\"\"\n",
    "    x = x.reshape(-1)\n",
    "    e_x = np.exp(x - np.max(x))\n",
    "    return e_x / e_x.sum(axis=0)\n",
    "\n",
    "def postprocess(scores):\n",
    "    \"\"\"This function takes the scores generated by the network and \n",
    "    returns the class IDs in decreasing order of probability.\"\"\"\n",
    "    prob = softmax(scores)\n",
    "    prob = np.squeeze(prob)\n",
    "    classes = np.argsort(prob)[::-1]\n",
    "    return classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write Environment File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.conda_dependencies import CondaDependencies \n",
    "\n",
    "myenv = CondaDependencies.create(pip_packages=[\"numpy\", \"azureml-core\"])\n",
    "\n",
    "with open(\"myenv.yml\",\"w\") as f:\n",
    "    f.write(myenv.serialize_to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Create the Container Image\n",
    "\n",
    "This step will take a few minutes if the container image is built for the first time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.image import ContainerImage\n",
    "from azureml.core.model import Model\n",
    "\n",
    "gpu_image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
    "                                                  runtime = \"python\",\n",
    "                                                  conda_file = \"myenv.yml\",\n",
    "                                                  description = \"Emotion ONNX Runtime container\",\n",
    "                                                  tags = {\"demo\": \"onnx\"})\n",
    "\n",
    "# Use the ONNX Runtime + TensorRT base image\n",
    "gpu_image_config.base_image = \"mcr.microsoft.com/azureml/onnxruntime:latest-tensorrt\"\n",
    "\n",
    "gpu_image = ContainerImage.create(name = \"onnximage.trt\",\n",
    "                              # this is the model object\n",
    "                              models = [model],\n",
    "                              image_config = gpu_image_config,\n",
    "                              workspace = ws)\n",
    "\n",
    "# Alternative: Re-use an image that you have already built from the workspace image registry\n",
    "# gpu_image = ContainerImage(name = \"onnximage.trt\", workspace = ws)\n",
    "\n",
    "gpu_image.wait_for_creation(show_output = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case you need to debug your code, the next line of code accesses the log file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(gpu_image.image_build_log_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This step is for a container image to target a CPU cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if deploy_with_cpu:\n",
    "    \n",
    "    cpu_image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
    "                                                      runtime = \"python\",\n",
    "                                                      conda_file = \"myenv.yml\",\n",
    "                                                      description = \"Emotion ONNX Runtime container\",\n",
    "                                                      tags = {\"demo\": \"onnx\"})\n",
    "\n",
    "    # use the ONNX Runtime CPU base image\n",
    "    cpu_image_config.base_image = \"mcr.microsoft.com/azureml/onnxruntime:latest\"\n",
    "\n",
    "    cpu_image = ContainerImage.create(name = \"onnximage.cpu\",\n",
    "                                      models = [model],\n",
    "                                      image_config = cpu_image_config,\n",
    "                                      workspace = ws)\n",
    "    \n",
    "    # Alternative: Re-use an image that you have already built from the workspace image registry\n",
    "    # cpu_image = ContainerImage(name = \"onnximage.cpu\", workspace = ws)\n",
    "\n",
    "    cpu_image.wait_for_creation(show_output = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case you need to debug your code, the next line of code accesses the log file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if deploy_with_cpu:\n",
    "    print(cpu_image.image_build_log_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're all done specifying what we want our virtual machine to do. Let's configure and deploy our container image.\n",
    "\n",
    "## 5. Deploy the container image\n",
    "\n",
    "Create a Azure Kubernetes Service (AKS) Compute Target for GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the AKS service with GPU nodes\n",
    "from azureml.core import Workspace\n",
    "from azureml.core.compute import AksCompute, ComputeTarget\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "from azureml.core.webservice import Webservice, AksWebservice\n",
    "from azureml.core.image import Image\n",
    "from azureml.core.model import Model\n",
    "\n",
    "gpu_aks_name = 'your-gpu-cluster'\n",
    "\n",
    "try:\n",
    "    gpu_aks_target = ComputeTarget(workspace = ws, name=gpu_aks_name)\n",
    "    print('Found existing cluster, use it.')\n",
    "except ComputeTargetException:\n",
    "    # Create a configuration (can also provide parameters to customize)\n",
    "    prov_config = AksCompute.provisioning_configuration(vm_size='Standard_NC6', location='East US2' )\n",
    "    # Create the cluster\n",
    "    gpu_aks_target = ComputeTarget.create(workspace = ws, \n",
    "                                  name = gpu_aks_name, \n",
    "                                  provisioning_configuration = prov_config)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "gpu_aks_target.wait_for_completion(show_output = True)\n",
    "print(gpu_aks_target.provisioning_state)\n",
    "print(gpu_aks_target.provisioning_errors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(Optional)** Configure another target for a CPU AKS cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CPU-cell\n",
    "\n",
    "if deploy_with_cpu: \n",
    "    \n",
    "    cpu_aks_name = 'your-cpu-cluster'\n",
    "\n",
    "    try:\n",
    "        cpu_aks_target = ComputeTarget(workspace = ws, name=cpu_aks_name)\n",
    "        print('Found existing cluster, use it.')\n",
    "\n",
    "    except ComputeTargetException:\n",
    "        # Create a configuration (can also provide parameters to customize)\n",
    "        prov_config = AksCompute.provisioning_configuration(vm_size='Standard_D3', location='East US2' )\n",
    "        # Create the cluster\n",
    "        cpu_aks_target = ComputeTarget.create(workspace = ws, \n",
    "                                      name = cpu_aks_name, \n",
    "                                      provisioning_configuration = prov_config)\n",
    "    \n",
    "    cpu_aks_target.wait_for_completion(show_output = True)\n",
    "    print(cpu_aks_target.provisioning_state)\n",
    "    print(cpu_aks_target.provisioning_errors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the web service configuration. In this case, we're using the default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gpu_aks_config = AksWebservice.deploy_configuration()\n",
    "\n",
    "if deploy_with_cpu:\n",
    "    cpu_aks_config = AksWebservice.deploy_configuration()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an AKS service for our GPU AKS cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "gpu_aks_service_name ='gpu-aks-service'\n",
    "\n",
    "gpu_aks_service = Webservice.deploy_from_image(workspace = ws, \n",
    "                                           name = gpu_aks_service_name,\n",
    "                                           image = gpu_image,\n",
    "                                           deployment_config = gpu_aks_config,\n",
    "                                           deployment_target = gpu_aks_target)\n",
    "gpu_aks_service.wait_for_deployment(show_output = True)\n",
    "print(gpu_aks_service.state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if gpu_aks_service.state != 'Healthy':\n",
    "    # run this command for debugging.\n",
    "    print(gpu_aks_service.get_logs())\n",
    "    \n",
    "    # If your deployment fails, make sure to delete your aci_service before trying again!\n",
    "    # gpu_aks_service.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(Optional)** Create an AKS service for our CPU AKS cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CPU-cell\n",
    "\n",
    "if deploy_with_cpu:\n",
    "    cpu_aks_service_name ='cpu-aks-service'\n",
    "\n",
    "    cpu_aks_service = Webservice.deploy_from_image(workspace = ws, \n",
    "                                               name = cpu_aks_service_name,\n",
    "                                               image = cpu_image,\n",
    "                                               deployment_config = cpu_aks_config,\n",
    "                                               deployment_target = cpu_aks_target)\n",
    "    \n",
    "    cpu_aks_service.wait_for_deployment(show_output = True)\n",
    "    print(cpu_aks_service.state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if deploy_with_cpu:\n",
    "    \n",
    "    if cpu_aks_service.state != 'Healthy':\n",
    "        # run this command for debugging.\n",
    "        print(cpu_aks_service.get_logs())\n",
    "\n",
    "        # If your deployment fails, make sure to delete your aks_service before trying again!\n",
    "        # cpu_aks_service.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Success!\n",
    "\n",
    "If you've made it this far, you've deployed a working AKS Cluster with a facial emotion recognition model running in the cloud using Azure ML. Congratulations!\n",
    "\n",
    "Let's see how well our model deals with our test images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing and Evaluation\n",
    "\n",
    "### Useful Helper Functions\n",
    "\n",
    "We preprocess and postprocess our data (see score.py file) using the helper functions specified in the [ONNX FER+ Model page in the Model Zoo repository](https://github.com/onnx/models/tree/master/emotion_ferplus)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def emotion_map(classes, N=1):\n",
    "    \"\"\"Take the most probable labels (output of postprocess) and returns the \n",
    "    top N emotional labels that fit the picture.\"\"\"\n",
    "    \n",
    "    emotion_table = {'neutral':0, 'happiness':1, 'surprise':2, 'sadness':3, \n",
    "                     'anger':4, 'disgust':5, 'fear':6, 'contempt':7}\n",
    "    \n",
    "    emotion_keys = list(emotion_table.keys())\n",
    "    emotions = []\n",
    "    for i in range(N):\n",
    "        emotions.append(emotion_keys[classes[i]])\n",
    "    return emotions\n",
    "\n",
    "def softmax(x):\n",
    "    \"\"\"Compute softmax values (probabilities from 0 to 1) for each possible label.\"\"\"\n",
    "    x = x.reshape(-1)\n",
    "    e_x = np.exp(x - np.max(x))\n",
    "    return e_x / e_x.sum(axis=0)\n",
    "\n",
    "def postprocess(scores):\n",
    "    \"\"\"This function takes the scores generated by the network and \n",
    "    returns the class IDs in decreasing order of probability.\"\"\"\n",
    "    prob = softmax(scores)\n",
    "    prob = np.squeeze(prob)\n",
    "    classes = np.argsort(prob)[::-1]\n",
    "    return classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try classifying your own images!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preprocessing functions take your image and format it so it can be passed\n",
    "# as input into our ONNX model\n",
    "import cv2\n",
    "\n",
    "def rgb2gray(rgb):\n",
    "    \"\"\"Convert the input image into grayscale\"\"\"\n",
    "    return np.dot(rgb[...,:3], [0.299, 0.587, 0.114])\n",
    "\n",
    "def resize_img(img):\n",
    "    \"\"\"Resize image to MNIST model input dimensions\"\"\"\n",
    "    img = cv2.resize(img, dsize=(64, 64), interpolation=cv2.INTER_AREA)\n",
    "    img.resize((1, 1, 64, 64))\n",
    "    return img\n",
    "\n",
    "def preprocess(img):\n",
    "    \"\"\"Resize input images and convert them to grayscale.\"\"\"\n",
    "    if img.shape == (64, 64):\n",
    "        img.resize((1, 1, 64, 64))\n",
    "        return img\n",
    "    \n",
    "    grayscale = rgb2gray(img)\n",
    "    processed_img = resize_img(grayscale)\n",
    "    return processed_img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace the following string with your own path/test image\n",
    "# Make sure your image is square and the dimensions are equal (i.e. 100 * 100 pixels or 28 * 28 pixels)\n",
    "\n",
    "# Any PNG or JPG image file should work\n",
    "# Make sure to include the entire path with // instead of /\n",
    "\n",
    "# e.g. your_test_image = \"C:/Users/vinitra.swamy/Pictures/face.png\"\n",
    "\n",
    "your_test_image = \"<path to file>\"\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import numpy as np\n",
    "import json\n",
    "import os\n",
    "\n",
    "if your_test_image != \"<path to file>\":\n",
    "    img = mpimg.imread(your_test_image)\n",
    "    plt.subplot(1,3,1)\n",
    "    plt.imshow(img, cmap = plt.cm.Greys)\n",
    "    print(\"Old Dimensions: \", img.shape)\n",
    "    img = preprocess(img)\n",
    "    print(\"New Dimensions: \", img.shape)\n",
    "else:\n",
    "    img = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if img is None:\n",
    "    print(\"Add the path for your image data.\")\n",
    "else:\n",
    "    input_data = json.dumps({'data': img.tolist()})\n",
    "\n",
    "    try:\n",
    "        r_gpu = json.loads(gpu_aks_service.run(input_data))\n",
    "        gpu_result = r_gpu['result'][0]\n",
    "        gpu_time_ms = np.round(r_gpu['time_in_sec'][0] * 1000, 2)\n",
    "        \n",
    "        if deploy_with_cpu:\n",
    "            r_cpu = json.loads(cpu_aks_service.run(input_data))\n",
    "            cpu_result = r_cpu['result'][0]\n",
    "            cpu_time_ms = np.round(r_cpu['time_in_sec'][0] * 1000, 2)\n",
    "        else:\n",
    "            cpu_result, cpu_time_ms = \"\", \"\"\n",
    "            \n",
    "    except Exception as e:\n",
    "        print(str(e))\n",
    "\n",
    "    plt.figure(figsize = (16, 6))\n",
    "    plt.subplot(1,8,1)\n",
    "    plt.axhline('')\n",
    "    plt.axvline('')\n",
    "    plt.text(x = -10, y = -40, s = \"Model prediction: \", fontsize = 14)\n",
    "    plt.text(x = -10, y = -25, s = \"Inference time (GPU, CPU): \", fontsize = 14)\n",
    "    plt.text(x = 100, y = -40, s = \"          \"+str(gpu_result)+ \"     \"+ str(cpu_result), fontsize = 14)\n",
    "    plt.text(x = 100, y = -25, s = \"          \"+str(gpu_time_ms) + \" ms\" + \"     \" +str(cpu_time_ms)+ \" ms\", fontsize = 14)\n",
    "    plt.text(x = -10, y = -10, s = \"Model Input image: \", fontsize = 14)\n",
    "    plt.imshow(img.reshape((64, 64)), cmap = plt.cm.gray)    \n",
    "     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CPU vs. GPU + TensorRT Performance comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if deploy_with_cpu:\n",
    "    x = ['GPU \\n(TensorRT)', 'CPU']\n",
    "    time = [gpu_time_ms, cpu_time_ms]\n",
    "\n",
    "    x_pos = [i for i, _ in enumerate(x)]\n",
    "\n",
    "    bar_graph = plt.barh(x_pos, time, color='green')\n",
    "    bar_graph[1].set_color('grey')\n",
    "    plt.ylabel(\"ONNX Runtime Deployment Type\")\n",
    "    plt.xlabel(\"Inference Time (ms)\")\n",
    "    plt.title(\"ONNX Runtime CPU vs GPU Performance Comparison\")\n",
    "\n",
    "    plt.yticks(x_pos, x)\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up our workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remember to delete your service after you are done using it!\n",
    "\n",
    "gpu_aks_service.delete()\n",
    "\n",
    "if deploy_with_cpu:\n",
    "    cpu_aks_service.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Congratulations!\n",
    "\n",
    "In this tutorial, you have:\n",
    "- familiarized yourself with ONNX Runtime inference and the pretrained models in the ONNX model zoo\n",
    "- understood a state-of-the-art convolutional neural net image classification model (FER+ in ONNX) and deployed it in the Azure ML cloud\n",
    "- ensured that your deep learning model is working perfectly (in the cloud) on test data, and checked it against some of your own!\n",
    "\n",
    "Next steps:\n",
    "- If you have not already, check out another interesting ONNX application that lets you set up a state-of-the-art [handwritten image classification model (MNIST)](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-inference-mnist-deploy.ipynb) on Azure! This tutorial deploys a pre-trained ONNX Computer Vision model for handwritten digit classification in an Azure ML Container Instance.\n",
    "- Contribute to our [open source ONNX repository on github](http://github.com/onnx/onnx) and/or add to our [ONNX model zoo](http://github.com/onnx/models)"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "viswamy"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "msauthor": "vinitra.swamy"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}