Integrating a model into the V2 API (CURRENT)
If you want to see an example of a module integrated with deepaas, where those methods are actually implemented, please head over to the deephdc demo app.
Defining what to load
The DEEPaaS API uses Python’s Setuptools entry points that are dynamically loaded to offer the model functionality through the API. This allows you to offer several models using a single DEEPaaS instance, by defining different entry points for the different models.
Warning
Serving multiple models is marked as deprecated, and will be removed in a future major version of the API. Please ensure that you start using the model-name configuration option in your configuration file or the –model-name command line option as soon as possible.
When the DEEPaaS API is spawned it will look for the deepaas.v2.model
entrypoint namespace, loading and adding the names found into the API
namespace. In order to define your entry points, your module should leverage
setuptools and be ready to be installed in the system. Then, in order to define
your entry points, you should add the following to your setup.cfg
configuration file:
[entry_points]
deepaas.v2.model =
my_model = package_name.module
This will define an entry point in the deepaas.v2.model
namespace, called
my_model
. All the required functionality will be fetched from the
package_name.module
module. This means that module
should provide the
Entry point (model) API as described below.
If you provide a class with the required functionality, the entry point will be defined as follows:
[entry_points]
deepaas.v2.model =
my_model = package_name.module:Class
Again, this will define an entry point in the deepaas.v2.model
namespace,
called my_model
. All the required functionality will be fetched
from the package_name.module.Class
class, meaning that an object of
Class
will be created and used as entry point. This also means that
Class
objects should provide the Entry point (model) API as described below.
Entry point (model) API
Regardless on the way you implement your entry point (i.e. as a module or as an object), you should expose the following functions or methods:
Defining model metadata
Your model entry point must implement a get_medatata
function that will
return some basic metadata information about your model, as follows:
- get_metadata(self)
Return metadata from the exposed model.
The metadata that is expected should follow the schema that is shown below. This basically means that you should return a dictionary with the following aspect:
{ "author": "Author name", "description": "Model description", "license": "Model's license", "url": "URL for the model (e.g. GitHub repository)", "version": "Model version", }
If you want to integrate with the deephdc platform you should provide at least an [
name
,author
,author-email
,license
]. You can nevertheless set them toNone
if you don’t feel like providing the information.The schema that we are following is the following:
{ "id": = fields.Str(required=True, description='Model identifier'), "name": fields.Str(required=True, description='Model name'), "description": fields.Str(required=True, description='Model description'), "license": fields.Str(required=False, description='Model license'), "author": fields.Str(required=False, description='Model author'), "version": fields.Str(required=False, description='Model version'), "url": fields.Str(required=False, description='Model url'), "links": fields.List( fields.Nested( { "rel": fields.Str(required=True), "href": fields.Url(required=True), } ) ) }
- Returns:
dictionary containing the model’s metadata.
Warming a model
You can initialize your model before any prediction or train is done by
defining a warm
function. This function receives no arguments and returns
no result, but it will be call before the API is spawned.
You can use it to implement any loading or initialization that your model may use. This way, your model will be ready whenever a first prediction is done, reducint the waiting time.
- warm(self)
Warm (initialize, load) the model.
This is called when the model is loaded, before the API is spawned.
If implemented, it should prepare the model for execution. This is useful for loading it into memory, perform any kind of preliminary checks, etc.
Training
Regarding training there are two functions to be defined. First of all, you can
specify the training arguments to be defined (and published through the API)
with the get_train_args
function, as follows:
- get_train_args(self)
Return the arguments that are needed to train the application.
This function should return a dictionary of
webargs
fields (check here for more information). For example:from webargs import fields (...) def get_train_args(): return { "arg1": fields.Str( required=False, # force the user to define the value missing="foo", # default value to use enum=["choice1", "choice2"], # list of choices description="Argument one" # help string ), }
- Return dict:
A dictionary of
webargs
fields containing the application required arguments.
Then, you must implement the training function (named train
) that will
receive the defined arguments as keyword arguments:
- train(self, **kwargs)
Perform a training.
- Parameters:
kwargs – The keyword arguments that the predict method accepts must be defined by the
get_train_args()
method so the API is able to pass them down. Usually you would populate these with all the training hyper-parameters- Returns:
You can return any Python object that is JSON parseable (eg. dict, string, float).
Prediction and inference
For prediction, there are different functions to be implemented. First of all,
as for the training, you can specify the prediction arguments to be defined,
(and published through the API) with the get_predict_args
as follows:
- get_predict_args(self)
Return the arguments that are needed to perform a prediction.
This function should return a dictionary of
webargs
fields (check here for more information). For example:from webargs import fields (...) def get_predict_args(): return { "arg1": fields.Str( required=False, # force the user to define the value missing="foo", # default value to use enum=["choice1", "choice2"], # list of choices description="Argument one" # help string ), }
- Return dict:
A dictionary of
webargs
fields containing the application required arguments.
Do not forget to add an input argument to hold your data. If you want to upload
files for inference to the API, you should use a webargs.fields.Field
field created as follows:
def get_predict_args():
return {
"data": fields.Field(
description="Data file to perform inference on.",
required=False,
missing=None,
type="file",
location="form")
}
You can also predict data stored in an URL by using:
def get_predict_args():
return {
"url": fields.Url(
description="Url of data to perform inference on.",
required=False,
missing=None)
}
Important
do not forget to add the location="form"
and type="file"
to the
argument definition, otherwise it will not work as expected.
Once defined, you will receive an object of the class described below for each
of the file arguments you declare. You can open and read the file stored in the
filename
attribute.
- class UploadedFile(name, filename, content_type, original_filename)
Class to hold uploaded field metadata when passed to model’s methods
- name
Name of the argument where this file is being sent.
- filename
Complete file path to the temporary file in the filesystem,
- content_type
Content-type of the uploaded file
- original_filename
Filename of the original file being uploaded.
Then you should define the predict
function as indicated below. You will
receive all the arguments that have been parsed as keyword arguments:
- predict(self, **kwargs)
Prediction from incoming keyword arguments.
- Parameters:
kwargs – The keyword arguments that the predict method accepts must be defined by the
get_predict_args()
method so the API is able to pass them down.- Returns:
The response must be a str or a dict.
By default, the return values from these two functions will be casted into a string, and will be returned in the following JSON response:
{
"status": "OK",
"predictions": "<model response as string>"
}
However, it is recommended that you specify a custom response schema. This way the API exposed will be richer and it will be easier for developers to build applications against your API, as they will be able to discover the response schemas from your endpoints.
In order to define a custom response, the schema
attribute is used:
- schema = None
Returning different content types
Sometimes it is useful to return something different than a JSON file. For such
cases, you can define an additional argument accept
defining the content
types that you are able to return as follows:
def get_predict_args():
return {
'accept': fields.Str(description="Media type(s) that is/are acceptable for the response.",
missing='application/zip',
validate=validate.OneOf(['application/zip', 'image/png', 'application/json']))
}
Find in this link a comprehensive list of possible content types. Then the predict function will have to return the raw bytes of a file according to the user selection. For example:
def predict(**args):
# Run your prediction
# Return file according to user selection
if args['accept'] == 'image/png':
return open(img_path, 'rb')
elif args['accept'] == 'application/json':
return {'some': 'json'}
elif args['accept'] == 'application/zip':
return open(zip_path, 'rb')
If you want to return several content types at the same time (let’s say a JSON and an image), the easiest way it to return a zip file with all the files.
Using classes
Apart from using a module, you can base your entrypoints on classes. If you
want to do so, you may find useful to inhering from the
deepaas.model.v2.base.BaseModel
abstract class:
- class BaseModel
Base class for all models to be used with DEEPaaS.
Note that it is not needed for DEEPaaS to inherit from this abstract base class in order to expose the model functionality, but the entrypoint that is configured should expose the same API.
- abstract get_metadata()
Return metadata from the exposed model.
The metadata that is expected should follow the schema that is shown below. This basically means that you should return a dictionary with the following aspect:
{ "author": "Author name", "description": "Model description", "license": "Model's license", "url": "URL for the model (e.g. GitHub repository)", "version": "Model version", }
If you want to integrate with the deephdc platform you should provide at least an [
name
,author
,author-email
,license
]. You can nevertheless set them toNone
if you don’t feel like providing the information.The schema that we are following is the following:
{ "id": = fields.Str(required=True, description='Model identifier'), "name": fields.Str(required=True, description='Model name'), "description": fields.Str(required=True, description='Model description'), "license": fields.Str(required=False, description='Model license'), "author": fields.Str(required=False, description='Model author'), "version": fields.Str(required=False, description='Model version'), "url": fields.Str(required=False, description='Model url'), "links": fields.List( fields.Nested( { "rel": fields.Str(required=True), "href": fields.Url(required=True), } ) ) }
- Returns:
dictionary containing the model’s metadata.
- abstract get_predict_args()
Return the arguments that are needed to perform a prediction.
This function should return a dictionary of
webargs
fields (check here for more information). For example:from webargs import fields (...) def get_predict_args(): return { "arg1": fields.Str( required=False, # force the user to define the value missing="foo", # default value to use enum=["choice1", "choice2"], # list of choices description="Argument one" # help string ), }
- Return dict:
A dictionary of
webargs
fields containing the application required arguments.
- abstract get_train_args()
Return the arguments that are needed to train the application.
This function should return a dictionary of
webargs
fields (check here for more information). For example:from webargs import fields (...) def get_train_args(): return { "arg1": fields.Str( required=False, # force the user to define the value missing="foo", # default value to use enum=["choice1", "choice2"], # list of choices description="Argument one" # help string ), }
- Return dict:
A dictionary of
webargs
fields containing the application required arguments.
- abstract predict(**kwargs)
Prediction from incoming keyword arguments.
- Parameters:
kwargs – The keyword arguments that the predict method accepts must be defined by the
get_predict_args()
method so the API is able to pass them down.- Returns:
The response must be a str or a dict.
- schema = None
Must contain a valid schema for the model’s predictions or None.
A valid schema is either a
marshmallow.Schema
subclass or a dictionary schema that can be converted into a schema.In order to provide a consistent API specification we use this attribute to define the schema that all the prediction responses will follow, therefore: - If this attribute is set we will validate them against it. - If it is not set (i.e.
schema = None
), the model’s response will be converted into a string and the response will have the following form:{ "status": "OK", "predictions": "<model response as string>" }
As previously stated, there are two ways of defining an schema here. If our response have the following form:
{ "status": "OK", "predictions": [ { "label": "foo", "probability": 1.0, }, { "label": "bar", "probability": 0.5, }, ] }
We should define or schema as schema as follows:
Using a schema dictionary. This is the most straightforward way. In order to do so, you must use the
marshmallow
Python module, as follows:from marshmallow import fields schema = { "status": fields.Str( description="Model predictions", required=True ), "predictions": fields.List( fields.Nested( { "label": fields.Str(required=True), "probability": fields.Float(required=True), }, ) ) }
Using a
marshmallow.Schema
subclass. Note that the schema must be the class that you have created, not an object:import marshmallow from marshmallow import fields class Prediction(marshmallow.Schema): label = fields.Str(required=True) probability = fields.Float(required=True) class Response(marshmallow.Schema): status = fields.Str( description="Model predictions", required=True ) predictions = fields.List(fields.Nested(Prediction)) schema = Response
- abstract train(**kwargs)
Perform a training.
- Parameters:
kwargs – The keyword arguments that the predict method accepts must be defined by the
get_train_args()
method so the API is able to pass them down. Usually you would populate these with all the training hyper-parameters- Returns:
You can return any Python object that is JSON parseable (eg. dict, string, float).
- abstract warm()
Warm (initialize, load) the model.
This is called when the model is loaded, before the API is spawned.
If implemented, it should prepare the model for execution. This is useful for loading it into memory, perform any kind of preliminary checks, etc.
Warning
The API uses multiprocessing
for handling tasks. Therefore if you use
decorators around your methods, please follow best practices and use
functools.wraps
so that the methods are still pickable. Beware also of using global variables
that might not be shared between processes.