Model Serving and RAG Operations

Load the model you actually evaluated

Training writes a model artifact to durable storage. The registry records its version, input requirements, evaluation results, and artifact location. At startup, a serving process loads that exact artifact and its preprocessing configuration; it should not train a fresh model when a request arrives.

Choose batch predictions when the product can wait for a scheduled output. Use an online endpoint when a decision needs fresh request data. Measure latency and throughput with representative inputs, then set timeouts, capacity limits, and a fallback the product can actually use.