Plugin#

class tensorrt_llm.plugin.PluginConfig(
*,
dtype: str = 'float16',
bert_attention_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = 'auto',
gpt_attention_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = 'auto',
gemm_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', 'fp8', 'nvfp4', None] | None = None,
gemm_swiglu_plugin: Literal['fp8', None] | None = None,
fp8_rowwise_gemm_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = None,
qserve_gemm_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = None,
identity_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = None,
nccl_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = 'auto',
lora_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = None,
dora_plugin: bool = False,
weight_only_groupwise_quant_matmul_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = None,
weight_only_quant_matmul_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = None,
smooth_quant_plugins: bool = True,
smooth_quant_gemm_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = None,
layernorm_quantization_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = None,
rmsnorm_quantization_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = None,
quantize_per_token_plugin: bool = False,
quantize_tensor_plugin: bool = False,
moe_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = 'auto',
mamba_conv1d_plugin: Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None] | None = 'auto',
low_latency_gemm_plugin: Literal['fp8', None] | None = None,
low_latency_gemm_swiglu_plugin: Literal['fp8', None] | None = None,
gemm_allreduce_plugin: Literal['float16', 'bfloat16', None] | None = None,
context_fmha: bool = True,
bert_context_fmha_fp32_acc: bool = False,
paged_kv_cache: bool | None = None,
remove_input_padding: bool = True,
norm_quant_fusion: bool = False,
reduce_fusion: bool = False,
user_buffer: bool = False,
tokens_per_block: int = 32,
use_paged_context_fmha: bool = True,
use_fp8_context_fmha: bool = True,
fuse_fp4_quant: bool = False,
multiple_profiles: bool = False,
paged_state: bool = True,
streamingllm: bool = False,
manage_weights: bool = False,
use_fused_mlp: bool = True,
pp_reduce_scatter: bool = False,
)[source]#

Bases: StrictBaseModel

The config that manages plugin-related options.

There are two option categories: * Plugin options (typically with xxx_plugin naming). These options can be assigned with:

  • “float16”/”bfloat16”/”float32”/”int32”, which means the plugin is enabled with the specified precision; (Some plugins only support limited dtype, i.e., gemm_swiglu_plugin and low_latency_gemm_swiglu_plugin only supports fp8 now)

  • “auto”, which means the plugin is enabled with the precision of dtype field (the dtype field must be same to model dtype, i.e., the one in PretrainedConfig);

  • None, which means the plugin is disabled.

  • Other features. These options can be assigned with boolean:

    • True, which means the plugin is enabled;

    • False, which means the plugin is disabled.

model_config = {'extra': 'forbid', 'validate_assignment': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod convert_enable_disable(
value,
info: ValidationInfo,
)[source]#

Allow passing enable/disable strings which map to boolean/None values.

classmethod log_field_changes(
v: Any,
info: ValidationInfo,
) → Any[source]#

Log all field changes for debugging.

classmethod from_arguments(args: Namespace)[source]#

Create a PluginConfig from argparse arguments.

to_legacy_setting()[source]#

Legacy setting means that all of the plugins and features are disabled, this is needed for the legacy build.py script, which will be migrated to the centralized building script tensorrt_llm/commands/build.py.

After the migration is done, this function may or may not be deleted.

model_post_init(context: Any, /) → None#

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self – The BaseModel instance.

  • context – The context.