chiennv2000/orthrus

Orthrus takes a different angle on LLM inference speed: instead of the usual speculative decoding tricks or quantization trade-offs, it uses a dual-view diffusion decoding approach that is claimed to be both fast and fully lossless. That combination is the headline claim worth stress-testing. Most speed-ups in this space involve a quality concession somewhere — either through approximation, reduced precision, or constrained token sampling. Orthrus says it avoids that. The open-source repo is early but the approach is grounded in diffusion-based generation rather than autoregressive shortcuts. Reservation: the project is a research-weight repo right now, not a production inference server. There is no obvious integration path to vLLM or llama.cpp out of the box, so expect engineering work before you can drop this into an existing pipeline. If the dual-view approach benchmarks well on your model family, it could be a real differentiator for latency-sensitive inference. -> Best for: ML researcher or AI engineer running their own inference infrastructure