NNI — An AutoML Toolkit

An Automatic Neural Network Intelligence Python Package

Introduction

This blog-post will review the AutoML toolkit that is called Neural Network Intelligence (NNI). Let’s start with some basic questions, then we will go over the main usages and in more detail on the practical aspect.

What is AutoML?

AutoML is an abbreviated phrase for the term “Automatic Machine Learning”, which means a toolkit that runs machine learning models and implements experiments automatically. Those packages usually help to speed up the search operation and automate procedures e.g., feature engineering and hyper-parameters tuning.

So, what is NNI?

Neural Network Intelligence (NNI) is a python AutoML package that works on Linux and Windows. This package trains neural networks models and finds a tuple of hyper-parameters that yields an optimal model. Moreover, it enables to conduct researches and experiments fast, effectively and without too much efforts.

Installation & Settings

Download Anaconda

Let’s start with some settings and installations. First, I really recommend you to work with Anaconda (a free and easy-to-use python platform for data scientists) when using this toolkit. If you still don’t have Anaconda installed on your computer, please take few minutes to install it and than continue reading this tutorial — download Anaconda.

Create a New Conda Virtual Environment (Optional)

I recommend you also to open a new python environment for NNI to have the specific versions of packages that you need for the project. If you don’t know how and what does it mean, you can check it out in my previous blog-post.

Open a new conda terminal window: Start->Anaconda Prompt->Right click->“Run” and create a new virtual environment as shown in the following command:

conda create --name nni_env python=3.6

Activate the environment:

conda activate nni_env

NNI Installation on Windows

In this tutorial we will focus on how to install NNI on Windows, but you can also find a detailed example on how to do it on Linux here.

Use conda to install NNI in your environment:

conda install nni

Verify Installation

Let’s verify installation with some MNIST example that is built on TensorFlow. For that you will need to download the source code with Git.

For this example you will need also to install TensorFlow (this MNIST example uses version 1.14):

conda install tensorflow==1.14

Now, clone the NNI repository:

git clone -b v1.5 https://github.com/Microsoft/nni.git

Then, run the sample to verify installation:

nnictl create --config nni\examples\trials\mnist-tfv1\config_windows.yml

The command line tool nnictl above is used to control experiments and manage NNI WebUI.

After running this command, you should receive a success message as below:

INFO: Starting restful server...
INFO: Successfully started Restful server!
INFO: Setting local config...
INFO: Successfully set local config!
INFO: Starting experiment...
INFO: Successfully started experiment!

Trials

NNI runs experiment many times when it is training the model. Each of those attempts at applying a new configuration (e.g., a set of hyperparameter values, a specific neural architecture) is called a Trial. Later we will see how to define an NNI trial using a search space of hyperparameters.

NNI WebUI

NNI provides also a web interface that helps users investigate their trials and explore the experiment results. You have an option to open the NNI web interface locally using your browser (open in chrome, won’t work in explorer) by the URL address that is shown in your command line:

The experiment id is [id]
The Web UI urls are: http://[Your IP]:8080

There are few tabs in the main page of NNI webUI:

Overview Tab

In this tab you will have a summary on the current status of the experiment that includes the duration, number of trails, technical errors, status per trail and search space and configuration files.

Overview Tab

Trials Detail Tab

This tab includes few sub-tabs that provide information about the results of the experiment per trail.

Default Metric Sub-Tab: The graph shows the running score and the values of the hyper-parameters for each trial.

Default metric sub-tab

Hyper-parameter Sub-Tab: This presentation shows the values of the hyper-parameters for all the trials on one go. The red graphs shows the trials that received the best scores and enables to investigate better ranges for the hyper-parameters.

Hyper-parameter sub-tab

Trial Duration Sub-Tab: A bar-graph that presents the duration for each of the trials.

Intermediate Result Sub-Tab: A line graph that presents the intermediate results and shows the trend of the trials in the training process along the way.

Common Errors

There are few errors that you may encounter when launching an NNI experiment. To find the error messages of a specific experiment you can follow the experiment Id (which is shown on the right-top corner of webUI) to check the logs:

C:\Users\[Your_user_name]\nni\experiments

Dispatcher Error

This error may happen due to an invalid search space configurations.

Trial Fails

In this error you will see on Trials detail tab that the status of all of your trials is FAILED, which means that you had some problem while running your script. Usually, it happens when trying to import a package that is not installed.

First, check if the code runs without using NNI:

python nni\examples\trials\mnist-tfv1\mnist.py

If not, you can test your import settings. Start a new Python interpreter session by running:

python

Once the interpreter opens up, type:

>>> import tensorflow as tf

If the above code shows an error “ImportError: No module named tensorflow”, you will need to install the missing package.

Useful Commands

The following commands will help you to manage your experiment effectively through the command line:

         commands                       description
1. nnictl experiment show        show the information of experiments
2. nnictl trial ls               list all of trial jobs
3. nnictl top                    monitor the status of running experiments
4. nnictl log stderr             show stderr log content
5. nnictl log stdout             show stdout log content
6. nnictl stop                   stop an experiment
7. nnictl trial kill             kill a trial job by id
8. nnictl --help                 get help information about nnictl

For example, if you made some changes and you try to restart your experiment, but receive this error:

Shut down the experiment and refresh it with:

nnictl stop

And then restart the experiment by rerunning the create command:

nnictl create --config config_windows.yml

Adding NNI to Your Own Project

If everything worked fine in the verification step, let’s try to set it up on our project. For this part, I will use a simple Linear NN that was used in my previous tutorial just to demonstrate step-by-step how things work. The code for this part is stored here.

NNI enables to define a trial using: NNI API and NNI Python annotation. In this tutorial we will focus on the first approach but you can check also the other one and choose your favorite.

Search_space File

The hyper-parameters will be defined in a json file that will be saved in the same directory of your project. It’s important that you will name this file search_space.json and will define the search space for each of your hyper-parameters.

Here is an example of such a file:

{
   “lr”:{“_type”:”uniform”,”_value”:[0.0001, 0.1]}, 
   “batch_size”: {“_type”:”choice”, “_value”: [25, 50, 100, 200]},
   “num_epochs”: {“_type”:”choice”, “_value”: [5, 6, 7, 8, 10]}, 
   “dropout_keep_prob”:{“_type”:”uniform”,”_value”:[0.5, 1.0]}
}

There are few sampling strategies that can be used to define the hyper-parameters search space, for example:

Choice Strategy: This strategy chooses a different value Iteratively from a list of options.

“hyper_parameter_name”:{“_type”: “choice”, “_value”: options}

Random Strategy: This strategy will randomize a variable that ranges between lower and upper parameters.

“hyper_parameter_name”:{“_type”: “randint”, “_value”: [lower, upper]}

Uniform Strategy: This strategy will sample a uniform variable that ranges between low and high parameters.

“hyper_parameter_name”: {“_type”: “uniform”, “_value”: [low, high]}

Note: Each strategy is supported by a different tuner, so you will need to check it out before and configure the config file accordingly.

Config_windows.yml File

Another file that should be saved in the same directory is the config_windows.yml file.

An example for such a file:

authorName: default
experimentName: sentiment_analysis_experiment
trialConcurrency: 1
maxExecDuration: 1h
maxTrialNum: 10
#choice: local, remote, pai
trainingServicePlatform: local
searchSpacePath: search_space.json
#choice: true, false
useAnnotation: false
tuner:
 #choice: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner
 #SMAC (SMAC should be installed through nnictl)
 builtinTunerName: TPE
 classArgs:
 #choice: maximize, minimize
 optimize_mode: maximize
trial:
 command: python main.py
 codeDir: .
 gpuNum: 0

In this file you will need to change the experiment name on the top and enable NNI API mode by setting the useAnnotation to False. Furthermore, choose number of trials by maxTrialNum, provide the path of search_space file and set the command parameter to the name of your script. For more information on each of the parameters you can use this link.

Changes in the Source Code

import

There are few thing that you will need to add to your source code. First, import nni package:

import nni

search_space.json

Then, add a function that will take each time a subset of hyper-parameters from search_space.json file and will run the source code. get_next_parameter() function will be used in the main to transfer each tuple of hyper-parameter from params to the current trail as the example bellow:

params = nni.get_next_parameter()
batch_size = params['batch_size']
dropout = params['dropout']
learning_rate = params['learning_rate']

params is an object that contains a specific set of variables. For instance, in our case above for one trial attempt, params object will look like that:

{“batch_size”: 20, “learning_rate”: 0.058, “dropout”: 0.27}

nni.report_intermediate_result

NNI enables also to follow after the intermediate results of the trial. To do so, you can add the report_final_result function that receives the metric (e.g., test accuracy, loss) and reports it to the assessor. The assessor analyzes trial’s intermediate results and check whether this trial can be early stopped or not.

nni.report_intermediate_result(your_metric)

nni.report_final_result

To receive the a report of the final results we will also add report_final_result function that receives the metric and reports final result to the tuner:

nni.report_final_result(your_metric)

Debug

It is impossible to debug NNI with your IDE debugger but you can look for stderr file inside nni directory and check the error log.

Run Your Project with NNI

Finally, after we’re done with the settings it is time to run the project.

Open the terminal and enter the directory path in which the project is saved:

cd [your_directory_path]

Create a new experiment that will run on your project:

nnictl create --config config_windows.yml

View Experiment Results

Enter the URL that was shown in the terminal and open it locally in your browser. Now, you can change the modes to investigate your results and to find the best hyper-parameters.

End Notes

This blog-post reviewed the AutoML toolkit — NNI. Hopefully this package will help you from now on to improve your model performance and manage your experiments effectively.

The full code for this tutorial is stored in my Github.