Logits Processor Zoo
https://github.com/nvidia/logits-processor-zoo- Category
- AI Tools
- Rank
- No. 940Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- nvidia
- GitHub
- 387 stars
- Date
About
Collection of logits processors to customize LLM generation for specific tasks. Drop-in for vLLM, HF, TRT-LLM.
What it does
It changes next-token scores while a language model writes. Rules can favor prompt vocabulary, control answer length, require a closing phrase, constrain multiple-choice answers, insert triggered text, stop after elapsed time, or substitute a fallback when confidence drops.
Why it's ranked here
The project solves several narrow generation-control problems with direct, understandable mechanisms. Its strongest case is portability across three inference stacks, although the supplied vLLM example requires the older engine because V1 does not accept logits processors.
What's good
The controls are composable and cover both soft steering and hard constraints. Users can bias prompt-derived tokens, force exact phrase sequences before termination, preserve sentence boundaries when stopping by time, and apply equivalent processor categories across supported runtimes.
Tradeoffs
Several controls depend on tokenizer details, token confidence, or wall-clock timing, so behavior needs model-specific testing. The vLLM example disables V1. TensorRT-LLM support requires version 0.20.0 or newer, and the core package directly depends on Torch, Transformers, and Accelerate.
How to use it well
Use it when Python inference code needs targeted, token-level output controls across Transformers, vLLM, or TensorRT-LLM. Start with deterministic examples, tune boosts against your chosen tokenizer, and test composed rules. Do not treat confidence-triggered fallback text as factual verification or full grammar enforcement.
Technical notes+
pyproject.toml defines a Poetry-built Python package requiring Python 3.10+, Torch, Transformers, and Accelerate, with vLLM as an optional extra. logits_processor_zoo/transformers/base.py tracks prompt and previous token tensors to detect generation resets. logits_processor_zoo/utils.py supplies token conversion, sentence-end checks, and score enforcement. logits_processor_zoo/vllm/max_time.py clones mutable state per prompt and enforces EOS or a replacement token after elapsed wall time. logits_processor_zoo/trtllm/max_time.py performs analogous mutation inside a CUDA stream. lpz_examples/vllm/utils.py explicitly sets VLLM_USE_V1 to 0 because V1 does not accept logits processors.
Observed
- License
- Apache License 2.0, stated in source SPDX headers and the README badge target
- Primary language
- Python
- Installation
- Published package installed with pip as logits-processor-zoo
- Packaging
- Poetry metadata with poetry-core build backend; Python 3.10 or newer
- Interface
- Python library integrated through inference framework logits-processor hooks
- Framework support
- Transformers, vLLM, and TensorRT-LLM 0.20.0 or newer
- Dependencies
- Torch, Transformers 4.41.2 or newer, and Accelerate 0.26.1 or newer; vLLM is optional
Read from README.md, pyproject.toml, logits_processor_zoo/utils.py, lpz_examples/vllm/utils.py, lpz_examples/trtllm/utils.py, lpz_examples/transformers/utils.py, logits_processor_zoo/vllm/__init__.py, logits_processor_zoo/vllm/max_time.py, logits_processor_zoo/trtllm/__init__.py, logits_processor_zoo/trtllm/max_time.py, logits_processor_zoo/vllm/cite_prompt.py, logits_processor_zoo/vllm/last_phrase.py, logits_processor_zoo/transformers/base.py, logits_processor_zoo/trtllm/cite_prompt.py, logits_processor_zoo/trtllm/last_phrase.py.
What it can do
Customize LLM text generation with custom logits processors
LLM model and custom logits processing rules → Modified text generation behavior
Integrate logits processors with vLLM framework
vLLM setup and logits processor configuration → Enhanced vLLM generation pipeline
Integrate logits processors with Hugging Face Transformers
Hugging Face model and processor specifications → Modified Hugging Face generation pipeline
Integrate logits processors with TensorRT-LLM
TensorRT-LLM model and processing configuration → Optimized TensorRT generation with custom logic
Apply task-specific generation constraints
Task requirements and generation parameters → LLM output tailored to specific task needs
Control token probability distributions during generation
Token probability preferences and generation context → Adjusted token selection probabilities
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.