The vllm openAI compatible server can make CFG(Classifier-Free Guidance) with outlines or lm-format-enforcer with Extra Parameters like guided_json, guided_regex. The openai compatible provider in Dify can not do this, but like gpt-4o can do same thing wit_assets/add_model.pngh json flag.

This plugin provide a vllm-openai provider upon Dify's openai compatible provider with guided_json, guided_regex, guided_grammar.

Add model same as openai compatible with vLLM-openai backend

Config model guided with `json schema`, `regex`, `grammar`

Dynamic Request guided

The guided param may various in same workflow, but Dify doesn't give a extra param in LLM to pass it.

This comes little hacky, when the prompt has 2nd part and it's assistant, this provider will use the assistant part as guided param. If it can be parsed as json.

The structure of param can be found in GuidedParam class.

README.md

vllm-openai dify provider plugin to support guided generate

Repo

Description

Add model same as openai compatible with vLLM-openai backend

Config model guided with json schema, regex, grammar

Dynamic Request guided

Config model guided with `json schema`, `regex`, `grammar`