torchserve : batch_size is always 1 even config.properties specify other value

Question

I think my torchserve loaded config.properties correctly because the number of worker is 2 as I set. But the batch_size is 1 instead of 20.

Anyone has an idea what might go wrong ? Thanks !

I have checked and torchserve load config.properties correctly, alas it ignored the batch_size and max_batch_delay specified in config.properties.

Here is my config.properties for the reference

inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
log_file=/ml_server/logs/torchserve.log
default_workers_per_model=2
number_of_netty_threads=32
job_queue_size=1000
batch_size=20
max_batch_delay=10

Below is the log, worker with batchSize: 1

ml-server  | 2024-03-06T00:11:11,091 [INFO ] W-9001-model_1.0-stdout MODEL_LOG - model_name: _model, batchSize: 1
ml-server  | 2024-03-06T00:11:11,091 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - model_name: _model, batchSize: 1

Answer 1

I figured it out. It is because my config.properties was wrong. This is correct.

models={\
  "prompt_injection_model": {\
    "1.0": {\
        "defaultVersion": true,\
        "marName": "prompt_injection_model.mar",\
        "minWorkers": 2,\
        "maxWorkers": 5,\
        "batchSize": 128,\
        "maxBatchDelay": 20,\
        "responseTimeout": 60\
    }\
  }\
}