Inference is the process of using a trained model to produce a prediction or output for new input. For an LLM request, it includes processing the prompt and generating tokens.
Inference
Inference is the process of using a trained model to produce a prediction or output for new input. For an LLM request, it includes processing the prompt and generating tokens.