Whether it’s performing object detection in images or video, recommending restaurants, or translating the spoken word, inference is the mechanism that allows applications to derive valuable information from trained AI models. But many inference solutions are one-off designs that lack the performance and flexibility to be seamlessly deployed in modern data center environments.

NVIDIA TensorRT Inference Server lets you leverage inference in your application without needing to reinvent the wheel. Delivered as a ready-to-deploy container from NGC, NVIDIA’s registry for GPU-accelerated software containers, and as an open source project, NVIDIA TensorRT Inference Server is a microservice that enables applications to use AI models in data center production. It maximizes GPU utilization, supports all popular AI frameworks and model types, and provides packaging and documentation for deployment using Kubeflow.

• The internal architecture of TensorRT Inference Server and how it fits into a larger inference workflow with Kubernetes, Kubeflow, and other load balancing and container orchestration solutions;
• How TensorRT Inference Server maximizes GPU utilization with dynamic batching and concurrent model execution on each GPU; and
• Best practices for using metrics from TensorRT Inference Server for autoscaling, health, and utilization, with a demonstration using Kubeflow and Google Kubernetes Engine (GKE).
  

Content

content goes here