mindspore_lite.LLMEngineStatus

class mindspore_lite.LLMEngineStatus(status)[source]

LLMEngine Status, which can be got from LLEngine.fetch_status.

property block_size: Get block size of PagedAttention

property empty_max_prompt_kv: Get empty count of prompt KV cache of this LLMEngine object

property num_free_blocks: Get number of free blocks PagedAttention

property num_total_blocks: Get number of total blocks PagedAttention