Why Use Multi-Container Endpoints instead of Multi-Model Endpoints?

Question

In the SageMaker documentation, both Multi-Model Endpoints and Multi-Container Endpoints with Direct Invocation are described as very similar methods to host multiple models on a single endpoint. The given use cases appear identical except that Multi-Model Endpoints include many more advanced features.

For example, Multi-Model Endpoints can host n number of models and support features such as resource sharing and model caching while Multi-Container Endpoints with Direct Invocation are limited to hosting only 5 models and lack model caching.

When does it make sense to use Multi-Container Endpoints with Direct Invocation instead of Multi-Model Endpoints?

Answer 1

If you want to serve multiple models from the same framework using the same endpoint then you can use multi-model endpoints. Due to using the same framework (e.g. only sklearn models), multi-model endpoints make it to the endpoint when they are called. You can have thousands of those models under one endpoint. Multi-container endpoints on the other hand allow serving models from multiple frameworks, e.g. one TensorFlow, one XGBoost and so on, with direct invocation again. However in this case there's limit of 5 different models on a single endpoint.

So depending on the problem you are working, if you need to use multiple frameworks on a single endpoint then you will need to use multi-container endpoint with direct invocation. Otherwise you can use the multi-model endpoint.

Reference