Skip to content

Advanced Command Line Parameters

Pass-through of inference engine parameters

Parameter Passing Instructions

Tip

  • Router accepts only its documented worker options and no longer forwards arbitrary vllm/lmdeploy parameters. Configure those parameters on independently deployed upstream services.
  • Command-line options support both --foo value and --foo=value forms
  • If you want to learn more about vllm parameter usage, please refer to the vllm official documentation
  • If you want to learn more about lmdeploy parameter usage, please refer to the lmdeploy official documentation

GPU Device Selection and Configuration

CUDA_VISIBLE_DEVICES Basic Usage

Tip

  • For CUDA-backed Torch/vLLM/LMDeploy execution, CUDA_VISIBLE_DEVICES selects visible CUDA devices. It does not select ONNX DirectML/WebGPU, Metal, or llama.cpp Vulkan devices. The examples require the corresponding accelerator runtimes: 
    MINERU_MODEL_SMALL_BACKEND=torch MINERU_MODEL_VLM_ENGINE=vllm CUDA_VISIBLE_DEVICES=1 mineru-kit parse <input_path> -o <output_path>
  • Router sets worker visibility through --local-gpus; auto also consults the detected device family’s visibility mask. Use explicit worker IDs when assigning devices.

Common Device Configuration Examples

Tip

Here are some common CUDA_VISIBLE_DEVICES setting examples: 

CUDA_VISIBLE_DEVICES=1  # Only device 1 will be seen
CUDA_VISIBLE_DEVICES=0,1  # Devices 0 and 1 will be visible
CUDA_VISIBLE_DEVICES="0,1"  # Same as above, quotation marks are optional
CUDA_VISIBLE_DEVICES=0,2,3  # Devices 0, 2, 3 will be visible; device 1 is masked
CUDA_VISIBLE_DEVICES=""  # No CUDA GPU will be visible

Practical Application Scenarios

Tip

Here are some possible usage scenarios:

  • If you have multiple GPUs and need to start two openai-server services on GPU 0 and GPU 1, each listening on a different port, you can use the following commands: 

    # In terminal 1
CUDA_VISIBLE_DEVICES=0 mineru-kit vlm-server --engine vllm --port 30000
# In terminal 2
CUDA_VISIBLE_DEVICES=1 mineru-kit vlm-server --engine vllm --port 30001

  • If you have multiple GPUs and need to start two fastapi services on GPU 0 and GPU 1, each listening on a different port, you can use the following commands: 

    # In terminal 1
MINERU_MODEL_SMALL_BACKEND=torch MINERU_MODEL_VLM_ENGINE=vllm CUDA_VISIBLE_DEVICES=0 mineru-kit api-server --host 127.0.0.1 --port 8000
# In terminal 2
MINERU_MODEL_SMALL_BACKEND=torch MINERU_MODEL_VLM_ENGINE=vllm CUDA_VISIBLE_DEVICES=1 mineru-kit api-server --host 127.0.0.1 --port 8001

  • If you have multiple GPUs and want to use router to launch and manage fastapi services across four GPUs, you can use the following command: 

    mineru-kit router --host 127.0.0.1 --port 8002 --local-gpus 0,1,2,3