# `Vllm.Config` [🔗](https://github.com/nshkrdotcom/vllm/blob/v0.3.0/lib/snakebridge_generated/vllm/config/__init__.ex#L6) Submodule bindings for `vllm.config`. ## Version - Requested: 0.14.0 - Observed at generation: 0.14.0 ## Runtime Options All functions accept a `__runtime__` option for controlling execution behavior: Vllm.Config.some_function(args, __runtime__: [timeout: 120_000]) ### Supported runtime options - `:timeout` - Call timeout in milliseconds (default: 120,000ms / 2 minutes) - `:timeout_profile` - Use a named profile (`:default`, `:ml_inference`, `:batch_job`, `:streaming`) - `:stream_timeout` - Timeout for streaming operations (default: 1,800,000ms / 30 minutes) - `:session_id` - Override the session ID for this call - `:pool_name` - Target a specific Snakepit pool (multi-pool setups) - `:affinity` - Override session affinity (`:hint`, `:strict_queue`, `:strict_fail_fast`) ### Timeout Profiles - `:default` - 2 minute timeout for regular calls - `:ml_inference` - 10 minute timeout for ML/LLM workloads - `:batch_job` - Unlimited timeout for long-running jobs - `:streaming` - 2 minute timeout, 30 minute stream_timeout ### Example with timeout override # For a long-running ML inference call Vllm.Config.predict(data, __runtime__: [timeout_profile: :ml_inference]) # Or explicit timeout Vllm.Config.predict(data, __runtime__: [timeout: 600_000]) # Route to a pool and enforce strict affinity Vllm.Config.predict(data, __runtime__: [pool_name: :strict_pool, affinity: :strict_queue]) See `SnakeBridge.Defaults` for global timeout configuration. # `__all__` ```elixir @spec __all__() :: {:ok, [term()]} | {:error, Snakepit.Error.t()} ``` Python module attribute `vllm.config.__all__`. ## Returns - `list(term())` # `config` ```elixir @spec config( term(), keyword() ) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` A decorator that ensures all fields in a dataclass have default values and that each field has a docstring. If a `ConfigT` is used as a CLI argument itself, the `type` keyword argument provided by `get_kwargs` will be `pydantic.TypeAdapter(ConfigT).validate_json(cli_arg)` which treats the `cli_arg` as a JSON string which gets validated by `pydantic`. Config validation is performed by the tools/pre_commit/validate_config.py script, which is invoked during the pre-commit checks. ## Parameters - `cls` (term()) ## Returns - `term()` # `config_type` ```elixir @spec config_type(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` type(object) -> the object's type type(name, bases, dict, **kwds) -> a new type ## Parameters - `args` (term()) - `kwargs` (term()) ## Returns - `term()` # `get_attr_docs` ```elixir @spec get_attr_docs( term(), keyword() ) :: {:ok, %{optional(String.t()) => String.t()}} | {:error, Snakepit.Error.t()} ``` Get any docstrings placed after attribute assignments in a class body. https://davidism.com/mit-license/ ## Parameters - `cls` (term()) ## Returns - `%{optional(String.t()) => String.t()}` # `get_cached_compilation_config` ```elixir @spec get_cached_compilation_config(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` Cache config to avoid repeated calls to get_current_vllm_config() ## Returns - `term()` # `get_current_vllm_config` ```elixir @spec get_current_vllm_config(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` Python binding for `vllm.config.get_current_vllm_config`. ## Returns - `term()` # `get_current_vllm_config_or_none` ```elixir @spec get_current_vllm_config_or_none(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` Python binding for `vllm.config.get_current_vllm_config_or_none`. ## Returns - `term()` # `get_layers_from_vllm_config` ```elixir @spec get_layers_from_vllm_config(term(), term()) :: {:ok, %{optional(String.t()) => term()}} | {:error, Snakepit.Error.t()} ``` Get layers from the vLLM config. ## Parameters - `vllm_config` - The vLLM config. - `layer_type` - The type of the layer to get. - `layer_names` - The names of the layers to get. If None, return all layers. ## Returns - `%{optional(String.t()) => term()}` # `get_layers_from_vllm_config` ```elixir @spec get_layers_from_vllm_config(term(), term(), keyword()) :: {:ok, %{optional(String.t()) => term()}} | {:error, Snakepit.Error.t()} @spec get_layers_from_vllm_config(term(), term(), term()) :: {:ok, %{optional(String.t()) => term()}} | {:error, Snakepit.Error.t()} ``` # `get_layers_from_vllm_config` ```elixir @spec get_layers_from_vllm_config(term(), term(), term(), keyword()) :: {:ok, %{optional(String.t()) => term()}} | {:error, Snakepit.Error.t()} ``` # `is_init_field` ```elixir @spec is_init_field(term(), String.t(), keyword()) :: {:ok, boolean()} | {:error, Snakepit.Error.t()} ``` Python binding for `vllm.config.is_init_field`. ## Parameters - `cls` (term()) - `name` (String.t()) ## Returns - `boolean()` # `iter_architecture_defaults` ```elixir @spec iter_architecture_defaults(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` Python binding for `vllm.config.iter_architecture_defaults`. ## Returns - `term()` # `set_current_vllm_config` ```elixir @spec set_current_vllm_config(term()) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` Temporarily set the current vLLM config. Used during model initialization. We save the current vLLM config in a global variable, so that all modules can access it, e.g. custom ops can access the vLLM config to determine how to dispatch. ## Parameters - `vllm_config` (term()) - `check_compile` (term() default: False) - `prefix` (term() default: None) ## Returns - `term()` # `set_current_vllm_config` ```elixir @spec set_current_vllm_config( term(), keyword() ) :: {:ok, term()} | {:error, Snakepit.Error.t()} @spec set_current_vllm_config(term(), term()) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` # `set_current_vllm_config` ```elixir @spec set_current_vllm_config(term(), term(), keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()} @spec set_current_vllm_config(term(), term(), term()) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` # `set_current_vllm_config` ```elixir @spec set_current_vllm_config(term(), term(), term(), keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` # `str_dtype_to_torch_dtype` ```elixir @spec str_dtype_to_torch_dtype( String.t(), keyword() ) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` Python binding for `vllm.config.str_dtype_to_torch_dtype`. ## Parameters - `type` (String.t()) ## Returns - `term()` # `try_match_architecture_defaults` ```elixir @spec try_match_architecture_defaults( String.t(), keyword() ) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` Python binding for `vllm.config.try_match_architecture_defaults`. ## Parameters - `architecture` (String.t()) - `runner_type` (term() | nil keyword-only default: None) - `convert_type` (term() | nil keyword-only default: None) ## Returns - `term()` # `update_config` ```elixir @spec update_config(term(), %{optional(String.t()) => term()}, keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()} ``` Python binding for `vllm.config.update_config`. ## Parameters - `config` (term()) - `overrides` (%{optional(String.t()) => term()}) ## Returns - `term()` --- *Consult [api-reference.md](api-reference.md) for complete listing*