Compare commits

...

8 Commits

Author SHA1 Message Date
f0a14f5fce Add Moodle data source integration (#11325)
### What problem does this PR solve?

This PR adds a native Moodle connector to sync content (courses,
resources, forums, assignments, pages, books) into RAGFlow.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2025-11-21 19:58:49 +08:00
174a2578e8 Feat: add auth header for Ollama chat model (#11452)
### What problem does this PR solve?

Add auth header for Ollama chat model. #11350

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-11-21 19:47:06 +08:00
a0959b9d38 Fix:Resolves the issue of sessions not being saved when the variable is array<object>. (#11446)
### What problem does this PR solve?

Fix:Resolves the issue of sessions not being saved when the variable is
array<object>.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-11-21 17:20:26 +08:00
13299197b8 Feat: Enable logical operators in metadata. #11387 #11376 (#11442)
### What problem does this PR solve?

Feat: Enable logical operators in metadata. #11387  #11376
### Type of change


- [x] New Feature (non-breaking change which adds functionality)
2025-11-21 16:21:27 +08:00
249296e417 Feat: API supports toc_enhance. (#11437)
### What problem does this PR solve?

Close #11433

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-11-21 14:51:58 +08:00
db0f6840d9 Feat: ignore chunk size when using custom delimiters (#11434)
### What problem does this PR solve?

Ignore chunk size when using custom delimiter.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-11-21 14:36:26 +08:00
1033a3ae26 Fix: improve PDF text type detection by expanding regex content (#11432)
- Add whitespace validation to the PDF English text checking regex
- Reduce false negatives in English PDF content recognition

### What problem does this PR solve?

The core idea is to **expand the regex content used for English text
detection** so it can accommodate more valid characters commonly found
in English PDFs. The modifications include:

- Adding support for **space** in the regex.
- Ensuring the update does not reduce existing detection accuracy.

### Type of change

- [] Bug Fix (non-breaking change which fixes an issue)
2025-11-21 14:33:29 +08:00
1845daf41f Fix: UI adjustments, replacing private components with public components (#11438)
### What problem does this PR solve?

Fix: UI adjustments, replacing private components with public components

- UI adjustments for public components (input, multiselect,
SliderInputFormField)

- Replacing the private LlmSettingFieldItems component in search with
the public LlmSettingFieldItems component


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-11-21 14:32:50 +08:00
47 changed files with 3884 additions and 3336 deletions

View File

@ -86,7 +86,7 @@ Try our demo at [https://demo.ragflow.io](https://demo.ragflow.io).
## 🔥 Latest Updates
- 2025-11-19 Supports Gemini 3 Pro.
- 2025-11-12 Supports data synchronization from Confluence, AWS S3, Discord, Google Drive.
- 2025-11-12 Supports data synchronization from Confluence, S3, Notion, Discord, Google Drive.
- 2025-10-23 Supports MinerU & Docling as document parsing methods.
- 2025-10-15 Supports orchestrable ingestion pipeline.
- 2025-08-08 Supports OpenAI's latest GPT-5 series models.

View File

@ -86,7 +86,7 @@ Coba demo kami di [https://demo.ragflow.io](https://demo.ragflow.io).
## 🔥 Pembaruan Terbaru
- 2025-11-19 Mendukung Gemini 3 Pro.
- 2025-11-12 Mendukung sinkronisasi data dari Confluence, AWS S3, Discord, Google Drive.
- 2025-11-12 Mendukung sinkronisasi data dari Confluence, S3, Notion, Discord, Google Drive.
- 2025-10-23 Mendukung MinerU & Docling sebagai metode penguraian dokumen.
- 2025-10-15 Dukungan untuk jalur data yang terorkestrasi.
- 2025-08-08 Mendukung model seri GPT-5 terbaru dari OpenAI.

View File

@ -67,7 +67,7 @@
## 🔥 最新情報
- 2025-11-19 Gemini 3 Proをサポートしています
- 2025-11-12 Confluence、AWS S3、Discord、Google Drive からのデータ同期をサポートします。
- 2025-11-12 Confluence、S3、Notion、Discord、Google Drive からのデータ同期をサポートします。
- 2025-10-23 ドキュメント解析方法として MinerU と Docling をサポートします。
- 2025-10-15 オーケストレーションされたデータパイプラインのサポート。
- 2025-08-08 OpenAI の最新 GPT-5 シリーズモデルをサポートします。

View File

@ -68,7 +68,7 @@
## 🔥 업데이트
- 2025-11-19 Gemini 3 Pro를 지원합니다.
- 2025-11-12 Confluence, AWS S3, Discord, Google Drive에서 데이터 동기화를 지원합니다.
- 2025-11-12 Confluence, S3, Notion, Discord, Google Drive에서 데이터 동기화를 지원합니다.
- 2025-10-23 문서 파싱 방법으로 MinerU 및 Docling을 지원합니다.
- 2025-10-15 조정된 데이터 파이프라인 지원.
- 2025-08-08 OpenAI의 최신 GPT-5 시리즈 모델을 지원합니다.

View File

@ -87,7 +87,7 @@ Experimente nossa demo em [https://demo.ragflow.io](https://demo.ragflow.io).
## 🔥 Últimas Atualizações
- 19-11-2025 Suporta Gemini 3 Pro.
- 12-11-2025 Suporta a sincronização de dados do Confluence, AWS S3, Discord e Google Drive.
- 12-11-2025 Suporta a sincronização de dados do Confluence, S3, Notion, Discord e Google Drive.
- 23-10-2025 Suporta MinerU e Docling como métodos de análise de documentos.
- 15-10-2025 Suporte para pipelines de dados orquestrados.
- 08-08-2025 Suporta a mais recente série GPT-5 da OpenAI.

View File

@ -86,7 +86,7 @@
## 🔥 近期更新
- 2025-11-19 支援 Gemini 3 Pro.
- 2025-11-12 支援從 Confluence、AWS S3、Discord、Google Drive 進行資料同步。
- 2025-11-12 支援從 Confluence、S3、Notion、Discord、Google Drive 進行資料同步。
- 2025-10-23 支援 MinerU 和 Docling 作為文件解析方法。
- 2025-10-15 支援可編排的資料管道。
- 2025-08-08 支援 OpenAI 最新的 GPT-5 系列模型。

View File

@ -86,7 +86,7 @@
## 🔥 近期更新
- 2025-11-19 支持 Gemini 3 Pro.
- 2025-11-12 支持从 Confluence、AWS S3、Discord、Google Drive 进行数据同步。
- 2025-11-12 支持从 Confluence、S3、Notion、Discord、Google Drive 进行数据同步。
- 2025-10-23 支持 MinerU 和 Docling 作为文档解析方法。
- 2025-10-15 支持可编排的数据管道。
- 2025-08-08 支持 OpenAI 最新的 GPT-5 系列模型。

View File

@ -32,7 +32,7 @@ class IterationParam(ComponentParamBase):
def __init__(self):
super().__init__()
self.items_ref = ""
self.veriable={}
self.variable={}
def get_input_form(self) -> dict[str, dict]:
return {

View File

@ -24,7 +24,7 @@ from flasgger import Swagger
from itsdangerous.url_safe import URLSafeTimedSerializer as Serializer
from quart_cors import cors
from common.constants import StatusEnum
from api.db.db_models import close_connection
from api.db.db_models import close_connection, APIToken
from api.db.services import UserService
from api.utils.json_encode import CustomJSONEncoder
from api.utils import commands
@ -124,6 +124,10 @@ def _load_user():
user = UserService.query(
access_token=access_token, status=StatusEnum.VALID.value
)
if not user and len(authorization.split()) == 2:
objs = APIToken.query(token=authorization.split()[1])
if objs:
user = UserService.query(id=objs[0].tenant_id, status=StatusEnum.VALID.value)
if user:
if not user[0].access_token or not user[0].access_token.strip():
logging.warning(f"User {user[0].email} has empty access_token in database")

View File

@ -1434,6 +1434,7 @@ async def retrieval_test(tenant_id):
question = req["question"]
doc_ids = req.get("document_ids", [])
use_kg = req.get("use_kg", False)
toc_enhance = req.get("toc_enhance", False)
langs = req.get("cross_languages", [])
if not isinstance(doc_ids, list):
return get_error_data_result("`documents` should be a list")
@ -1487,6 +1488,11 @@ async def retrieval_test(tenant_id):
highlight=highlight,
rank_feature=label_question(question, kbs),
)
if toc_enhance:
chat_mdl = LLMBundle(kb.tenant_id, LLMType.CHAT)
cks = settings.retriever.retrieval_by_toc(question, ranks["chunks"], tenant_ids, chat_mdl, size)
if cks:
ranks["chunks"] = cks
if use_kg:
ck = settings.kg_retriever.retrieval(question, [k.tenant_id for k in kbs], kb_ids, embd_mdl, LLMBundle(kb.tenant_id, LLMType.CHAT))
if ck["content_with_weight"]:

View File

@ -118,6 +118,7 @@ class FileSource(StrEnum):
SHAREPOINT = "sharepoint"
SLACK = "slack"
TEAMS = "teams"
MOODLE = "moodle"
class PipelineTaskType(StrEnum):

View File

@ -14,6 +14,7 @@ from .google_drive.connector import GoogleDriveConnector
from .jira.connector import JiraConnector
from .sharepoint_connector import SharePointConnector
from .teams_connector import TeamsConnector
from .moodle_connector import MoodleConnector
from .config import BlobType, DocumentSource
from .models import Document, TextSection, ImageSection, BasicExpertInfo
from .exceptions import (
@ -36,6 +37,7 @@ __all__ = [
"JiraConnector",
"SharePointConnector",
"TeamsConnector",
"MoodleConnector",
"BlobType",
"DocumentSource",
"Document",

View File

@ -48,6 +48,7 @@ class DocumentSource(str, Enum):
GOOGLE_DRIVE = "google_drive"
GMAIL = "gmail"
DISCORD = "discord"
MOODLE = "moodle"
S3_COMPATIBLE = "s3_compatible"

View File

@ -0,0 +1,378 @@
from __future__ import annotations
import logging
import os
from collections.abc import Generator
from datetime import datetime, timezone
from retry import retry
from typing import Any, Optional
from markdownify import markdownify as md
from moodle import Moodle as MoodleClient, MoodleException
from common.data_source.config import INDEX_BATCH_SIZE
from common.data_source.exceptions import (
ConnectorMissingCredentialError,
CredentialExpiredError,
InsufficientPermissionsError,
ConnectorValidationError,
)
from common.data_source.interfaces import LoadConnector, PollConnector, SecondsSinceUnixEpoch
from common.data_source.models import Document
from common.data_source.utils import batch_generator, rl_requests
logger = logging.getLogger(__name__)
class MoodleConnector(LoadConnector, PollConnector):
"""Moodle LMS connector for accessing course content"""
def __init__(self, moodle_url: str, batch_size: int = INDEX_BATCH_SIZE) -> None:
self.moodle_url = moodle_url.rstrip("/")
self.batch_size = batch_size
self.moodle_client: Optional[MoodleClient] = None
def _add_token_to_url(self, file_url: str) -> str:
"""Append Moodle token to URL if missing"""
if not self.moodle_client:
return file_url
token = getattr(self.moodle_client, "token", "")
if "token=" in file_url.lower():
return file_url
delimiter = "&" if "?" in file_url else "?"
return f"{file_url}{delimiter}token={token}"
def _log_error(self, context: str, error: Exception, level: str = "warning") -> None:
"""Simplified logging wrapper"""
msg = f"{context}: {error}"
if level == "error":
logger.error(msg)
else:
logger.warning(msg)
def _get_latest_timestamp(self, *timestamps: int) -> int:
"""Return latest valid timestamp"""
return max((t for t in timestamps if t and t > 0), default=0)
def _yield_in_batches(
self, generator: Generator[Document, None, None]
) -> Generator[list[Document], None, None]:
for batch in batch_generator(generator, self.batch_size):
yield batch
def load_credentials(self, credentials: dict[str, Any]) -> None:
token = credentials.get("moodle_token")
if not token:
raise ConnectorMissingCredentialError("Moodle API token is required")
try:
self.moodle_client = MoodleClient(
self.moodle_url + "/webservice/rest/server.php", token
)
self.moodle_client.core.webservice.get_site_info()
except MoodleException as e:
if "invalidtoken" in str(e).lower():
raise CredentialExpiredError("Moodle token is invalid or expired")
raise ConnectorMissingCredentialError(f"Failed to initialize Moodle client: {e}")
def validate_connector_settings(self) -> None:
if not self.moodle_client:
raise ConnectorMissingCredentialError("Moodle client not initialized")
try:
site_info = self.moodle_client.core.webservice.get_site_info()
if not site_info.sitename:
raise InsufficientPermissionsError("Invalid Moodle API response")
except MoodleException as e:
msg = str(e).lower()
if "invalidtoken" in msg:
raise CredentialExpiredError("Moodle token is invalid or expired")
if "accessexception" in msg:
raise InsufficientPermissionsError(
"Insufficient permissions. Ensure web services are enabled and permissions are correct."
)
raise ConnectorValidationError(f"Moodle validation error: {e}")
except Exception as e:
raise ConnectorValidationError(f"Unexpected validation error: {e}")
# -------------------------------------------------------------------------
# Data loading & polling
# -------------------------------------------------------------------------
def load_from_state(self) -> Generator[list[Document], None, None]:
if not self.moodle_client:
raise ConnectorMissingCredentialError("Moodle client not initialized")
logger.info("Starting full load from Moodle workspace")
courses = self._get_enrolled_courses()
if not courses:
logger.warning("No courses found to process")
return
yield from self._yield_in_batches(self._process_courses(courses))
def poll_source(
self, start: SecondsSinceUnixEpoch, end: SecondsSinceUnixEpoch
) -> Generator[list[Document], None, None]:
if not self.moodle_client:
raise ConnectorMissingCredentialError("Moodle client not initialized")
logger.info(
f"Polling Moodle updates between {datetime.fromtimestamp(start)} and {datetime.fromtimestamp(end)}"
)
courses = self._get_enrolled_courses()
if not courses:
logger.warning("No courses found to poll")
return
yield from self._yield_in_batches(self._get_updated_content(courses, start, end))
@retry(tries=3, delay=1, backoff=2)
def _get_enrolled_courses(self) -> list:
if not self.moodle_client:
raise ConnectorMissingCredentialError("Moodle client not initialized")
try:
return self.moodle_client.core.course.get_courses()
except MoodleException as e:
self._log_error("fetching courses", e, "error")
raise ConnectorValidationError(f"Failed to fetch courses: {e}")
@retry(tries=3, delay=1, backoff=2)
def _get_course_contents(self, course_id: int):
if not self.moodle_client:
raise ConnectorMissingCredentialError("Moodle client not initialized")
try:
return self.moodle_client.core.course.get_contents(courseid=course_id)
except MoodleException as e:
self._log_error(f"fetching course contents for {course_id}", e)
return []
def _process_courses(self, courses) -> Generator[Document, None, None]:
for course in courses:
try:
contents = self._get_course_contents(course.id)
for section in contents:
for module in section.modules:
doc = self._process_module(course, section, module)
if doc:
yield doc
except Exception as e:
self._log_error(f"processing course {course.fullname}", e)
def _get_updated_content(
self, courses, start: float, end: float
) -> Generator[Document, None, None]:
for course in courses:
try:
contents = self._get_course_contents(course.id)
for section in contents:
for module in section.modules:
times = [
getattr(module, "timecreated", 0),
getattr(module, "timemodified", 0),
]
if hasattr(module, "contents"):
times.extend(
getattr(c, "timemodified", 0)
for c in module.contents
if c and getattr(c, "timemodified", 0)
)
last_mod = self._get_latest_timestamp(*times)
if start < last_mod <= end:
doc = self._process_module(course, section, module)
if doc:
yield doc
except Exception as e:
self._log_error(f"polling course {course.fullname}", e)
def _process_module(
self, course, section, module
) -> Optional[Document]:
try:
mtype = module.modname
if mtype in ["label", "url"]:
return None
if mtype == "resource":
return self._process_resource(course, section, module)
if mtype == "forum":
return self._process_forum(course, section, module)
if mtype == "page":
return self._process_page(course, section, module)
if mtype in ["assign", "quiz"]:
return self._process_activity(course, section, module)
if mtype == "book":
return self._process_book(course, section, module)
except Exception as e:
self._log_error(f"processing module {getattr(module, 'name', '?')}", e)
return None
def _process_resource(self, course, section, module) -> Optional[Document]:
if not getattr(module, "contents", None):
return None
file_info = module.contents[0]
if not getattr(file_info, "fileurl", None):
return None
file_name = os.path.basename(file_info.filename)
ts = self._get_latest_timestamp(
getattr(module, "timecreated", 0),
getattr(module, "timemodified", 0),
getattr(file_info, "timemodified", 0),
)
try:
resp = rl_requests.get(self._add_token_to_url(file_info.fileurl), timeout=60)
resp.raise_for_status()
blob = resp.content
ext = os.path.splitext(file_name)[1] or ".bin"
semantic_id = f"{course.fullname} / {section.name} / {file_name}"
return Document(
id=f"moodle_resource_{module.id}",
source="moodle",
semantic_identifier=semantic_id,
extension=ext,
blob=blob,
doc_updated_at=datetime.fromtimestamp(ts or 0, tz=timezone.utc),
size_bytes=len(blob),
)
except Exception as e:
self._log_error(f"downloading resource {file_name}", e, "error")
return None
def _process_forum(self, course, section, module) -> Optional[Document]:
if not self.moodle_client or not getattr(module, "instance", None):
return None
try:
result = self.moodle_client.mod.forum.get_forum_discussions(forumid=module.instance)
disc_list = getattr(result, "discussions", [])
if not disc_list:
return None
markdown = [f"# {module.name}\n"]
latest_ts = self._get_latest_timestamp(
getattr(module, "timecreated", 0),
getattr(module, "timemodified", 0),
)
for d in disc_list:
markdown.append(f"## {d.name}\n\n{md(d.message or '')}\n\n---\n")
latest_ts = max(latest_ts, getattr(d, "timemodified", 0))
blob = "\n".join(markdown).encode("utf-8")
semantic_id = f"{course.fullname} / {section.name} / {module.name}"
return Document(
id=f"moodle_forum_{module.id}",
source="moodle",
semantic_identifier=semantic_id,
extension=".md",
blob=blob,
doc_updated_at=datetime.fromtimestamp(latest_ts or 0, tz=timezone.utc),
size_bytes=len(blob),
)
except Exception as e:
self._log_error(f"processing forum {module.name}", e)
return None
def _process_page(self, course, section, module) -> Optional[Document]:
if not getattr(module, "contents", None):
return None
file_info = module.contents[0]
if not getattr(file_info, "fileurl", None):
return None
file_name = os.path.basename(file_info.filename)
ts = self._get_latest_timestamp(
getattr(module, "timecreated", 0),
getattr(module, "timemodified", 0),
getattr(file_info, "timemodified", 0),
)
try:
resp = rl_requests.get(self._add_token_to_url(file_info.fileurl), timeout=60)
resp.raise_for_status()
blob = resp.content
ext = os.path.splitext(file_name)[1] or ".html"
semantic_id = f"{course.fullname} / {section.name} / {module.name}"
return Document(
id=f"moodle_page_{module.id}",
source="moodle",
semantic_identifier=semantic_id,
extension=ext,
blob=blob,
doc_updated_at=datetime.fromtimestamp(ts or 0, tz=timezone.utc),
size_bytes=len(blob),
)
except Exception as e:
self._log_error(f"processing page {file_name}", e, "error")
return None
def _process_activity(self, course, section, module) -> Optional[Document]:
desc = getattr(module, "description", "")
if not desc:
return None
mtype, mname = module.modname, module.name
markdown = f"# {mname}\n\n**Type:** {mtype.capitalize()}\n\n{md(desc)}"
ts = self._get_latest_timestamp(
getattr(module, "timecreated", 0),
getattr(module, "timemodified", 0),
getattr(module, "added", 0),
)
semantic_id = f"{course.fullname} / {section.name} / {mname}"
blob = markdown.encode("utf-8")
return Document(
id=f"moodle_{mtype}_{module.id}",
source="moodle",
semantic_identifier=semantic_id,
extension=".md",
blob=blob,
doc_updated_at=datetime.fromtimestamp(ts or 0, tz=timezone.utc),
size_bytes=len(blob),
)
def _process_book(self, course, section, module) -> Optional[Document]:
if not getattr(module, "contents", None):
return None
contents = module.contents
chapters = [
c for c in contents
if getattr(c, "fileurl", None) and os.path.basename(c.filename) == "index.html"
]
if not chapters:
return None
latest_ts = self._get_latest_timestamp(
getattr(module, "timecreated", 0),
getattr(module, "timemodified", 0),
*[getattr(c, "timecreated", 0) for c in contents],
*[getattr(c, "timemodified", 0) for c in contents],
)
markdown_parts = [f"# {module.name}\n"]
for ch in chapters:
try:
resp = rl_requests.get(self._add_token_to_url(ch.fileurl), timeout=60)
resp.raise_for_status()
html = resp.content.decode("utf-8", errors="ignore")
markdown_parts.append(md(html) + "\n\n---\n")
except Exception as e:
self._log_error(f"processing book chapter {ch.filename}", e)
blob = "\n".join(markdown_parts).encode("utf-8")
semantic_id = f"{course.fullname} / {section.name} / {module.name}"
return Document(
id=f"moodle_book_{module.id}",
source="moodle",
semantic_identifier=semantic_id,
extension=".md",
blob=blob,
doc_updated_at=datetime.fromtimestamp(latest_ts or 0, tz=timezone.utc),
size_bytes=len(blob),
)

View File

@ -1091,7 +1091,7 @@ class RAGFlowPdfParser:
logging.debug("Images converted.")
self.is_english = [
re.search(r"[a-zA-Z0-9,/¸;:'\[\]\(\)!@#$%^&*\"?<>._-]{30,}", "".join(random.choices([c["text"] for c in self.page_chars[i]], k=min(100, len(self.page_chars[i])))))
re.search(r"[ a-zA-Z0-9,/¸;:'\[\]\(\)!@#$%^&*\"?<>._-]{30,}", "".join(random.choices([c["text"] for c in self.page_chars[i]], k=min(100, len(self.page_chars[i])))))
for i in range(len(self.page_chars))
]
if sum([1 if e else 0 for e in self.is_english]) > len(self.page_images) / 2:
@ -1148,7 +1148,7 @@ class RAGFlowPdfParser:
if not self.is_english and not any([c for c in self.page_chars]) and self.boxes:
bxes = [b for bxs in self.boxes for b in bxs]
self.is_english = re.search(r"[\na-zA-Z0-9,/¸;:'\[\]\(\)!@#$%^&*\"?<>._-]{30,}", "".join([b["text"] for b in random.choices(bxes, k=min(30, len(bxes)))]))
self.is_english = re.search(r"[ \na-zA-Z0-9,/¸;:'\[\]\(\)!@#$%^&*\"?<>._-]{30,}", "".join([b["text"] for b in random.choices(bxes, k=min(30, len(bxes)))]))
logging.debug(f"Is it English: {self.is_english}")

View File

@ -2072,6 +2072,7 @@ Retrieves chunks from specified datasets.
- `"cross_languages"`: `list[string]`
- `"metadata_condition"`: `object`
- `"use_kg"`: `boolean`
- `"toc_enhance"`: `boolean`
##### Request example
```bash
@ -2122,6 +2123,8 @@ curl --request POST \
The number of chunks engaged in vector cosine computation. Defaults to `1024`.
- `"use_kg"`: (*Body parameter*), `boolean`
The search includes text chunks related to the knowledge graph of the selected dataset to handle complex multi-hop queries. Defaults to `False`.
- `"toc_enhance"`: (*Body parameter*), `boolean`
The search includes table of content enhancement in order to boost rank of relevant chunks. Files parsed with `TOC Enhance` enabled is prerequisite. Defaults to `False`.
- `"rerank_id"`: (*Body parameter*), `integer`
The ID of the rerank model.
- `"keyword"`: (*Body parameter*), `boolean`
@ -2136,6 +2139,9 @@ curl --request POST \
The languages that should be translated into, in order to achieve keywords retrievals in different languages.
- `"metadata_condition"`: (*Body parameter*), `object`
The metadata condition used for filtering chunks:
- `"logic"`: (*Body parameter*), `string`
- `"and"` Intersection of the result from each condition (default).
- `"or"` union of the result from each condition.
- `"conditions"`: (*Body parameter*), `array`
A list of metadata filter conditions.
- `"name"`: `string` - The metadata field name to filter by, e.g., `"author"`, `"company"`, `"url"`. Ensure this parameter before use. See [Set metadata](../guides/dataset/set_metadata.md) for details.

View File

@ -133,7 +133,7 @@ dependencies = [
"pyicu>=2.15.3,<3.0.0",
"flasgger>=0.9.7.1,<0.10.0",
"xxhash>=3.5.0,<4.0.0",
"trio>=0.29.0",
"trio>=0.17.0,<0.29.0",
"langfuse>=2.60.0",
"debugpy>=1.8.13",
"mcp>=1.9.4",
@ -148,6 +148,7 @@ dependencies = [
"markdownify>=1.2.0",
"captcha>=0.7.1",
"pip>=25.2",
"moodlepy>=0.23.0",
"pypandoc>=1.16",
"pyobvector==0.2.18",
]

View File

@ -1635,6 +1635,15 @@ class LiteLLMBase(ABC):
provider_cfg["allow_fallbacks"] = False
extra_body["provider"] = provider_cfg
completion_args.update({"extra_body": extra_body})
# Ollama deployments commonly sit behind a reverse proxy that enforces
# Bearer auth. Ensure the Authorization header is set when an API key
# is provided, while respecting any user-supplied headers. #11350
extra_headers = deepcopy(completion_args.get("extra_headers") or {})
if self.provider == SupportedLiteLLMProvider.Ollama and self.api_key and "Authorization" not in extra_headers:
extra_headers["Authorization"] = f"Bearer {self.api_key}"
if extra_headers:
completion_args["extra_headers"] = extra_headers
return completion_args
def chat_with_tools(self, system: str, history: list, gen_conf: dict = {}):

View File

@ -437,16 +437,16 @@ def not_title(txt):
return re.search(r"[,;,。;!!]", txt)
def tree_merge(bull, sections, depth):
if not sections or bull < 0:
return sections
if isinstance(sections[0], type("")):
sections = [(s, "") for s in sections]
# filter out position information in pdf sections
sections = [(t, o) for t, o in sections if
t and len(t.split("@")[0].strip()) > 1 and not re.match(r"[0-9]+$", t.split("@")[0].strip())]
def get_level(bull, section):
text, layout = section
text = re.sub(r"\u3000", " ", text).strip()
@ -465,7 +465,7 @@ def tree_merge(bull, sections, depth):
level, text = get_level(bull, section)
if not text.strip("\n"):
continue
lines.append((level, text))
level_set.add(level)
@ -608,6 +608,26 @@ def naive_merge(sections: str | list, chunk_token_num=128, delimiter="\n。
cks[-1] += t
tk_nums[-1] += tnum
custom_delimiters = [m.group(1) for m in re.finditer(r"`([^`]+)`", delimiter)]
has_custom = bool(custom_delimiters)
if has_custom:
custom_pattern = "|".join(re.escape(t) for t in sorted(set(custom_delimiters), key=len, reverse=True))
cks, tk_nums = [], []
for sec, pos in sections:
split_sec = re.split(r"(%s)" % custom_pattern, sec, flags=re.DOTALL)
for sub_sec in split_sec:
if re.fullmatch(custom_pattern, sub_sec or ""):
continue
text = "\n" + sub_sec
local_pos = pos
if num_tokens_from_string(text) < 8:
local_pos = ""
if local_pos and text.find(local_pos) < 0:
text += local_pos
cks.append(text)
tk_nums.append(num_tokens_from_string(text))
return cks
dels = get_delimiters(delimiter)
for sec, pos in sections:
if num_tokens_from_string(sec) < chunk_token_num:
@ -657,6 +677,29 @@ def naive_merge_with_images(texts, images, chunk_token_num=128, delimiter="\n。
result_images[-1] = concat_img(result_images[-1], image)
tk_nums[-1] += tnum
custom_delimiters = [m.group(1) for m in re.finditer(r"`([^`]+)`", delimiter)]
has_custom = bool(custom_delimiters)
if has_custom:
custom_pattern = "|".join(re.escape(t) for t in sorted(set(custom_delimiters), key=len, reverse=True))
cks, result_images, tk_nums = [], [], []
for text, image in zip(texts, images):
text_str = text[0] if isinstance(text, tuple) else text
text_pos = text[1] if isinstance(text, tuple) and len(text) > 1 else ""
split_sec = re.split(r"(%s)" % custom_pattern, text_str)
for sub_sec in split_sec:
if re.fullmatch(custom_pattern, sub_sec or ""):
continue
text_seg = "\n" + sub_sec
local_pos = text_pos
if num_tokens_from_string(text_seg) < 8:
local_pos = ""
if local_pos and text_seg.find(local_pos) < 0:
text_seg += local_pos
cks.append(text_seg)
result_images.append(image)
tk_nums.append(num_tokens_from_string(text_seg))
return cks, result_images
dels = get_delimiters(delimiter)
for text, image in zip(texts, images):
# if text is tuple, unpack it
@ -748,6 +791,23 @@ def naive_merge_docx(sections, chunk_token_num=128, delimiter="\n。"):
images[-1] = concat_img(images[-1], image)
tk_nums[-1] += tnum
custom_delimiters = [m.group(1) for m in re.finditer(r"`([^`]+)`", delimiter)]
has_custom = bool(custom_delimiters)
if has_custom:
custom_pattern = "|".join(re.escape(t) for t in sorted(set(custom_delimiters), key=len, reverse=True))
cks, images, tk_nums = [], [], []
pattern = r"(%s)" % custom_pattern
for sec, image in sections:
split_sec = re.split(pattern, sec)
for sub_sec in split_sec:
if not sub_sec or re.fullmatch(custom_pattern, sub_sec):
continue
text_seg = "\n" + sub_sec
cks.append(text_seg)
images.append(image)
tk_nums.append(num_tokens_from_string(text_seg))
return cks, images
dels = get_delimiters(delimiter)
pattern = r"(%s)" % dels
@ -789,7 +849,7 @@ class Node:
self.level = level
self.depth = depth
self.texts = texts or []
self.children = []
self.children = []
def add_child(self, child_node):
self.children.append(child_node)
@ -835,7 +895,7 @@ class Node:
return self
def get_tree(self):
tree_list = []
tree_list = []
self._dfs(self, tree_list, [])
return tree_list
@ -860,7 +920,7 @@ class Node:
# A leaf title within depth emits its title path as a chunk (header-only section)
elif not child and (1 <= level <= self.depth):
tree_list.append("\n".join(path_titles))
# Recurse into children with the updated title path
for c in child:
self._dfs(c, tree_list, path_titles)
self._dfs(c, tree_list, path_titles)

View File

@ -37,14 +37,8 @@ from api.db.services.connector_service import ConnectorService, SyncLogsService
from api.db.services.knowledgebase_service import KnowledgebaseService
from common import settings
from common.config_utils import show_configs
from common.data_source import BlobStorageConnector, NotionConnector, DiscordConnector, GoogleDriveConnector, MoodleConnector, JiraConnector
from common.constants import FileSource, TaskStatus
from common.data_source import (
BlobStorageConnector,
DiscordConnector,
GoogleDriveConnector,
JiraConnector,
NotionConnector,
)
from common.data_source.config import INDEX_BATCH_SIZE
from common.data_source.confluence_connector import ConfluenceConnector
from common.data_source.interfaces import CheckpointOutputWrapper
@ -418,6 +412,37 @@ class Teams(SyncBase):
pass
class Moodle(SyncBase):
SOURCE_NAME: str = FileSource.MOODLE
async def _generate(self, task: dict):
self.connector = MoodleConnector(
moodle_url=self.conf["moodle_url"],
batch_size=self.conf.get("batch_size", INDEX_BATCH_SIZE)
)
self.connector.load_credentials(self.conf["credentials"])
# Determine the time range for synchronization based on reindex or poll_range_start
if task["reindex"] == "1" or not task.get("poll_range_start"):
document_generator = self.connector.load_from_state()
begin_info = "totally"
else:
poll_start = task["poll_range_start"]
if poll_start is None:
document_generator = self.connector.load_from_state()
begin_info = "totally"
else:
document_generator = self.connector.poll_source(
poll_start.timestamp(),
datetime.now(timezone.utc).timestamp()
)
begin_info = "from {}".format(poll_start)
logging.info("Connect to Moodle: {} {}".format(self.conf["moodle_url"], begin_info))
return document_generator
func_factory = {
FileSource.S3: S3,
FileSource.NOTION: Notion,
@ -429,6 +454,7 @@ func_factory = {
FileSource.SHAREPOINT: SharePoint,
FileSource.SLACK: Slack,
FileSource.TEAMS: Teams,
FileSource.MOODLE: Moodle
}

5779
uv.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,4 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1230.87 315.18">
<path fill="#f98012" d="M289.61 309.77V201.51q0-33.94-28-33.95t-28.06 33.95v108.26H178.4V201.51q0-33.94-27.57-33.95-28.05 0-28 33.95v108.26H67.67V195.12q0-35.43 24.6-53.63 21.66-16.25 58.56-16.25 37.41 0 55.12 19.19 15.26-19.19 55.62-19.19 36.9 0 58.54 16.25 24.6 18.19 24.61 53.63v114.65Zm675.49-.5V0h55.16v309.27Zm-70.3 0v-18.22q-7.39 9.84-25.11 15.76a92.81 92.81 0 0 1-30.05 5.41q-39.4 0-63.28-27.09t-23.89-67c0-26.25 7.76-48.3 23.4-66 13.85-15.65 36.35-26.59 62.29-26.59 29.22 0 46.28 11 56.64 23.63V0h53.68v309.27Zm0-102.92q0-14.78-14-28.33T852 164.47q-21.16 0-33.48 17.24-10.85 15.3-10.84 37.43 0 21.68 10.84 36.94 12.3 17.75 33.48 17.73 12.81 0 27.83-12.07t15-24.86ZM648.57 314.19q-41.87 0-69.19-26.59T552 219.14q0-41.83 27.34-68.45t69.19-26.59q41.85 0 69.44 26.59t27.58 68.45q0 41.88-27.58 68.46t-69.4 26.59Zm0-145.77q-19.94 0-30.65 15.1t-10.71 35.88q0 20.78 10 35.13 11.46 16.34 31.4 16.32T680 254.53q10.46-14.34 10.46-35.13t-10-35.13q-11.46-15.86-31.89-15.85ZM449.13 314.19q-41.86 0-69.2-26.59t-27.33-68.46q0-41.83 27.33-68.45t69.2-26.59q41.83 0 69.44 26.59t27.57 68.45q0 41.88-27.57 68.46t-69.44 26.59Zm0-145.77q-19.94 0-30.66 15.1t-10.71 35.88q0 20.78 10 35.13 11.46 16.34 31.41 16.32t31.39-16.32Q491 240.19 491 219.4t-10-35.13q-11.44-15.86-31.87-15.85Zm636.45 67.47c1.18 13.13 18.25 41.37 46.31 41.37 27.31 0 40.23-15.77 40.87-22.16l58.11-.5c-6.34 19.39-32.1 60.58-100 60.58-28.24 0-54.08-8.79-72.64-26.35s-27.82-40.45-27.82-68.7q0-43.83 27.82-69.68t72.16-25.85q48.25 0 75.34 32 25.13 29.53 25.12 79.28Zm90.13-34c-2.3-11.83-7.23-21.49-14.77-29.06q-12.82-12.3-29.55-12.31-17.25 0-28.82 11.82t-15.5 29.55Z"/>
<path fill="#333" d="m174.74 116.9 54.74-40-.7-2.44C130 86.57 85.08 95.15 0 144.47l.79 2.24 6.76.07c-.62 6.81-1.7 23.64-.32 48.95-9.44 27.32-.24 45.88 8.4 66.07 1.37-21 1.23-44-5.22-66.89-1.35-25.14-.24-41.67.37-48.1l56.4.54a258 258 0 0 0 1.67 33.06c50.4 17.71 101.09-.06 128-43.72-7.47-8.37-22.11-19.79-22.11-19.79Z"/>
</svg>

After

Width:  |  Height:  |  Size: 2.0 KiB

View File

@ -356,6 +356,13 @@ const DynamicForm = {
...combinedErrors,
...fieldErrors,
} as any;
console.log('combinedErrors', combinedErrors);
for (const key in combinedErrors) {
if (Array.isArray(combinedErrors[key])) {
combinedErrors[key] = combinedErrors[key][0];
}
}
console.log('combinedErrors', combinedErrors);
return {
values: Object.keys(combinedErrors).length ? {} : data,
@ -720,9 +727,7 @@ const DynamicForm = {
type="button"
disabled={submitLoading}
onClick={() => {
console.log('form submit');
(async () => {
console.log('form submit2');
try {
let beValid = await form.formControl.trigger();
console.log('form valid', beValid, form, form.formControl);

View File

@ -1,6 +1,9 @@
import { ModelVariableType } from '@/constants/knowledge';
import {
ModelVariableType,
settledModelVariableMap,
} from '@/constants/knowledge';
import { useTranslate } from '@/hooks/common-hooks';
import { camelCase } from 'lodash';
import { camelCase, isEqual } from 'lodash';
import { useCallback } from 'react';
import { useFormContext } from 'react-hook-form';
import { z } from 'zod';
@ -25,6 +28,13 @@ import { useHandleFreedomChange } from './use-watch-change';
interface LlmSettingFieldItemsProps {
prefix?: string;
options?: any[];
showFields?: Array<
| 'temperature'
| 'top_p'
| 'presence_penalty'
| 'frequency_penalty'
| 'max_tokens'
>;
}
export const LLMIdFormField = {
@ -56,6 +66,13 @@ export const LlmSettingSchema = {
export function LlmSettingFieldItems({
prefix,
options,
showFields = [
'temperature',
'top_p',
'presence_penalty',
'frequency_penalty',
'max_tokens',
],
}: LlmSettingFieldItemsProps) {
const form = useFormContext();
const { t } = useTranslate('chat');
@ -72,14 +89,53 @@ export function LlmSettingFieldItems({
const parameterOptions = Object.values(ModelVariableType).map((x) => ({
label: t(camelCase(x)),
value: x,
}));
})) as { label: string; value: ModelVariableType | 'Custom' }[];
parameterOptions.push({
label: t(camelCase('Custom')),
value: 'Custom',
});
const checkParameterIsEqual = () => {
const [
parameter,
topPValue,
frequencyPenaltyValue,
temperatureValue,
presencePenaltyValue,
maxTokensValue,
] = form.getValues([
getFieldWithPrefix('parameter'),
getFieldWithPrefix('temperature'),
getFieldWithPrefix('top_p'),
getFieldWithPrefix('frequency_penalty'),
getFieldWithPrefix('presence_penalty'),
getFieldWithPrefix('max_tokens'),
]);
if (parameter && parameter !== 'Custom') {
const parameterValue =
settledModelVariableMap[parameter as keyof typeof ModelVariableType];
const parameterRealValue = {
top_p: topPValue,
temperature: temperatureValue,
frequency_penalty: frequencyPenaltyValue,
presence_penalty: presencePenaltyValue,
max_tokens: maxTokensValue,
};
if (!isEqual(parameterValue, parameterRealValue)) {
form.setValue(getFieldWithPrefix('parameter'), 'Custom');
}
}
};
return (
<div className="space-y-5">
<LLMFormField options={options}></LLMFormField>
<LLMFormField
options={options}
name={getFieldWithPrefix('llm_id')}
></LLMFormField>
<FormField
control={form.control}
name={'parameter'}
name={getFieldWithPrefix('parameter')}
render={({ field }) => (
<FormItem className="flex justify-between items-center">
<FormLabel className="flex-1">{t('freedom')}</FormLabel>
@ -107,45 +163,71 @@ export function LlmSettingFieldItems({
</FormItem>
)}
/>
<SliderInputSwitchFormField
name={getFieldWithPrefix('temperature')}
checkName="temperatureEnabled"
label="temperature"
max={1}
step={0.01}
min={0}
></SliderInputSwitchFormField>
<SliderInputSwitchFormField
name={getFieldWithPrefix('top_p')}
checkName="topPEnabled"
label="topP"
max={1}
step={0.01}
min={0}
></SliderInputSwitchFormField>
<SliderInputSwitchFormField
name={getFieldWithPrefix('presence_penalty')}
checkName="presencePenaltyEnabled"
label="presencePenalty"
max={1}
step={0.01}
min={0}
></SliderInputSwitchFormField>
<SliderInputSwitchFormField
name={getFieldWithPrefix('frequency_penalty')}
checkName="frequencyPenaltyEnabled"
label="frequencyPenalty"
max={1}
step={0.01}
min={0}
></SliderInputSwitchFormField>
<SliderInputSwitchFormField
name={getFieldWithPrefix('max_tokens')}
checkName="maxTokensEnabled"
label="maxTokens"
max={128000}
min={0}
></SliderInputSwitchFormField>
{showFields.some((item) => item === 'temperature') && (
<SliderInputSwitchFormField
name={getFieldWithPrefix('temperature')}
checkName="temperatureEnabled"
label="temperature"
max={1}
step={0.01}
min={0}
onChange={() => {
checkParameterIsEqual();
}}
></SliderInputSwitchFormField>
)}
{showFields.some((item) => item === 'top_p') && (
<SliderInputSwitchFormField
name={getFieldWithPrefix('top_p')}
checkName="topPEnabled"
label="topP"
max={1}
step={0.01}
min={0}
onChange={() => {
checkParameterIsEqual();
}}
></SliderInputSwitchFormField>
)}
{showFields.some((item) => item === 'presence_penalty') && (
<SliderInputSwitchFormField
name={getFieldWithPrefix('presence_penalty')}
checkName="presencePenaltyEnabled"
label="presencePenalty"
max={1}
step={0.01}
min={0}
onChange={() => {
checkParameterIsEqual();
}}
></SliderInputSwitchFormField>
)}
{showFields.some((item) => item === 'frequency_penalty') && (
<SliderInputSwitchFormField
name={getFieldWithPrefix('frequency_penalty')}
checkName="frequencyPenaltyEnabled"
label="frequencyPenalty"
max={1}
step={0.01}
min={0}
onChange={() => {
checkParameterIsEqual();
}}
></SliderInputSwitchFormField>
)}
{showFields.some((item) => item === 'max_tokens') && (
<SliderInputSwitchFormField
name={getFieldWithPrefix('max_tokens')}
checkName="maxTokensEnabled"
numberInputClassName="w-20"
label="maxTokens"
max={128000}
min={0}
onChange={() => {
checkParameterIsEqual();
}}
></SliderInputSwitchFormField>
)}
</div>
);
}

View File

@ -22,6 +22,7 @@ type SliderInputSwitchFormFieldProps = {
onChange?: (value: number) => void;
className?: string;
checkName: string;
numberInputClassName?: string;
};
export function SliderInputSwitchFormField({
@ -34,6 +35,7 @@ export function SliderInputSwitchFormField({
onChange,
className,
checkName,
numberInputClassName,
}: SliderInputSwitchFormFieldProps) {
const form = useFormContext();
const disabled = !form.watch(checkName);
@ -81,7 +83,10 @@ export function SliderInputSwitchFormField({
<FormControl>
<NumberInput
disabled={disabled}
className="h-7 w-20"
className={cn(
'h-6 w-10 p-1 border border-border-button rounded-sm',
numberInputClassName,
)}
max={max}
min={min}
step={step}

View File

@ -14,6 +14,7 @@ type MetadataFilterProps = {
export const MetadataFilterSchema = {
meta_data_filter: z
.object({
logic: z.string().optional(),
method: z.string().optional(),
manual: z
.array(

View File

@ -15,14 +15,17 @@ import {
} from '@/components/ui/form';
import { Input } from '@/components/ui/input';
import { Separator } from '@/components/ui/separator';
import { SwitchOperatorOptions } from '@/constants/agent';
import { SwitchLogicOperator, SwitchOperatorOptions } from '@/constants/agent';
import { useBuildSwitchOperatorOptions } from '@/hooks/logic-hooks/use-build-operator-options';
import { useBuildSwitchLogicOperatorOptions } from '@/hooks/logic-hooks/use-build-options';
import { useFetchKnowledgeMetadata } from '@/hooks/use-knowledge-request';
import { PromptEditor } from '@/pages/agent/form/components/prompt-editor';
import { Plus, X } from 'lucide-react';
import { useCallback } from 'react';
import { useFieldArray, useFormContext } from 'react-hook-form';
import { useTranslation } from 'react-i18next';
import { RAGFlowFormItem } from '../ragflow-form';
import { RAGFlowSelect } from '../ui/select';
export function MetadataFilterConditions({
kbIds,
@ -36,10 +39,13 @@ export function MetadataFilterConditions({
const { t } = useTranslation();
const form = useFormContext();
const name = prefix + 'meta_data_filter.manual';
const logic = prefix + 'meta_data_filter.logic';
const metadata = useFetchKnowledgeMetadata(kbIds);
const switchOperatorOptions = useBuildSwitchOperatorOptions();
const switchLogicOperatorOptions = useBuildSwitchLogicOperatorOptions();
const { fields, remove, append } = useFieldArray({
name,
control: form.control,
@ -47,13 +53,14 @@ export function MetadataFilterConditions({
const add = useCallback(
(key: string) => () => {
form.setValue(logic, SwitchLogicOperator.And);
append({
key,
value: '',
op: SwitchOperatorOptions[0].value,
});
},
[append],
[append, form, logic],
);
return (
@ -77,73 +84,92 @@ export function MetadataFilterConditions({
</DropdownMenuContent>
</DropdownMenu>
</div>
<div className="space-y-5">
{fields.map((field, index) => {
const typeField = `${name}.${index}.key`;
return (
<div key={field.id} className="flex w-full items-center gap-2">
<FormField
control={form.control}
name={typeField}
render={({ field }) => (
<FormItem className="flex-1 overflow-hidden">
<FormControl>
<Input
{...field}
placeholder={t('common.pleaseInput')}
></Input>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<Separator className="w-3 text-text-secondary" />
<FormField
control={form.control}
name={`${name}.${index}.op`}
render={({ field }) => (
<FormItem className="flex-1 overflow-hidden">
<FormControl>
<SelectWithSearch
{...field}
options={switchOperatorOptions}
></SelectWithSearch>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<Separator className="w-3 text-text-secondary" />
<FormField
control={form.control}
name={`${name}.${index}.value`}
render={({ field }) => (
<FormItem className="flex-1 overflow-hidden">
<FormControl>
{canReference ? (
<PromptEditor
{...field}
multiLine={false}
showToolbar={false}
></PromptEditor>
) : (
<Input
placeholder={t('common.pleaseInput')}
{...field}
/>
<section className="flex">
{fields.length > 1 && (
<div className="relative min-w-14">
<RAGFlowFormItem
name={logic}
className="absolute top-1/2 -translate-y-1/2 right-1 left-0 z-10 bg-bg-base"
>
<RAGFlowSelect
options={switchLogicOperatorOptions}
triggerClassName="w-full text-xs px-1 py-0 h-6"
></RAGFlowSelect>
</RAGFlowFormItem>
<div className="absolute border-l border-y w-5 right-0 top-4 bottom-4 rounded-l-lg"></div>
</div>
)}
<div className="space-y-5 flex-1">
{fields.map((field, index) => {
const typeField = `${name}.${index}.key`;
return (
<section key={field.id} className="flex gap-2">
<div className="w-full space-y-2">
<div className="flex items-center gap-1">
<FormField
control={form.control}
name={typeField}
render={({ field }) => (
<FormItem className="flex-1 overflow-hidden">
<FormControl>
<Input
{...field}
placeholder={t('common.pleaseInput')}
></Input>
</FormControl>
<FormMessage />
</FormItem>
)}
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<Button variant={'ghost'} onClick={() => remove(index)}>
<X className="text-text-sub-title-invert " />
</Button>
</div>
);
})}
</div>
/>
<Separator className="w-1 text-text-secondary" />
<FormField
control={form.control}
name={`${name}.${index}.op`}
render={({ field }) => (
<FormItem className="flex-1 overflow-hidden">
<FormControl>
<SelectWithSearch
{...field}
options={switchOperatorOptions}
></SelectWithSearch>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
</div>
<FormField
control={form.control}
name={`${name}.${index}.value`}
render={({ field }) => (
<FormItem className="flex-1 overflow-hidden">
<FormControl>
{canReference ? (
<PromptEditor
{...field}
multiLine={false}
showToolbar={false}
></PromptEditor>
) : (
<Input
placeholder={t('common.pleaseInput')}
{...field}
/>
)}
</FormControl>
<FormMessage />
</FormItem>
)}
/>
</div>
<Button variant={'ghost'} onClick={() => remove(index)}>
<X className="text-text-sub-title-invert " />
</Button>
</section>
);
})}
</div>
</section>
</section>
);
}

View File

@ -32,13 +32,13 @@ const Input = function ({
type={type}
data-slot="input"
className={cn(
'border-input file:text-foreground placeholder:text-muted-foreground/70 flex h-9 w-full min-w-0 rounded-md border bg-transparent px-3 py-1 text-sm shadow-xs transition-[color,box-shadow] outline-none file:inline-flex file:h-7 file:border-0 file:bg-transparent file:text-sm file:font-medium disabled:pointer-events-none disabled:cursor-not-allowed disabled:opacity-50',
'focus-visible:border-ring focus-visible:ring-ring/50 focus-visible:ring-[3px]',
'border-border-button file:text-foreground placeholder:text-text-disabled flex h-9 w-full min-w-0 rounded-md border bg-transparent px-3 py-1 text-sm shadow-xs transition-[color,box-shadow] outline-none file:inline-flex file:h-7 file:border-0 file:bg-transparent file:text-sm file:font-medium disabled:pointer-events-none disabled:cursor-not-allowed disabled:opacity-50',
'focus-visible:border-border-button focus-visible:ring-text-primary/50 focus-visible:ring-1',
'aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 aria-invalid:border-destructive',
type === 'search' &&
'[&::-webkit-search-cancel-button]:appearance-none [&::-webkit-search-decoration]:appearance-none [&::-webkit-search-results-button]:appearance-none [&::-webkit-search-results-decoration]:appearance-none',
type === 'file' &&
'text-muted-foreground/70 file:border-input file:text-foreground p-0 pr-3 italic file:me-3 file:h-full file:border-0 file:border-r file:border-solid file:bg-transparent file:px-3 file:text-sm file:font-medium file:not-italic',
'text-text-disabled file:border-input file:text-foreground p-0 pr-3 italic file:me-3 file:h-full file:border-0 file:border-r file:border-solid file:bg-transparent file:px-3 file:text-sm file:font-medium file:not-italic',
icon && iconPosition === 'left' && 'pl-7',
icon && iconPosition === 'right' && 'pr-7',
className,

View File

@ -221,10 +221,12 @@ const RaptorFormFields = ({
defaultValue={0}
type="number"
suffix={
<Shuffle
className="size-3.5 cursor-pointer"
onClick={handleGenerate}
/>
<div className="w-7 flex justify-center items-center">
<Shuffle
className="size-3.5 cursor-pointer"
onClick={handleGenerate}
/>
</div>
}
/>
</FormControl>

View File

@ -59,6 +59,7 @@ interface SimilaritySliderFormFieldProps {
similarityName?: string;
vectorSimilarityWeightName?: string;
isTooltipShown?: boolean;
numberInputClassName?: string;
}
export const initialSimilarityThresholdValue = {
@ -86,6 +87,7 @@ export function SimilaritySliderFormField({
similarityName = 'similarity_threshold',
vectorSimilarityWeightName = 'vector_similarity_weight',
isTooltipShown,
numberInputClassName,
}: SimilaritySliderFormFieldProps) {
const { t } = useTranslate('knowledgeDetails');
const form = useFormContext();
@ -101,6 +103,7 @@ export function SimilaritySliderFormField({
step={0.01}
layout={FormLayout.Vertical}
tooltip={isTooltipShown && t('similarityThresholdTip')}
numberInputClassName={numberInputClassName}
></SliderInputFormField>
<FormField
control={form.control}
@ -124,7 +127,7 @@ export function SimilaritySliderFormField({
isVector ? 'vectorSimilarityWeight' : 'keywordSimilarityWeight',
)}
</FormLabel>
<div className={cn('flex items-end gap-14 justify-between')}>
<div className={cn('flex items-end gap-4 justify-between')}>
<FormControl>
<div className="flex flex-col flex-1 gap-2">
<div className="flex justify-between items-center">
@ -158,6 +161,7 @@ export function SimilaritySliderFormField({
className={cn(
'h-6 w-10 p-0 text-center bg-bg-input border-border-default border text-text-secondary',
'[appearance:textfield] [&::-webkit-outer-spin-button]:appearance-none [&::-webkit-inner-spin-button]:appearance-none',
numberInputClassName,
)}
max={1}
min={0}

View File

@ -25,6 +25,7 @@ type SliderInputFormFieldProps = {
tooltip?: ReactNode;
defaultValue?: number;
className?: string;
numberInputClassName?: string;
} & FormLayoutType;
export function SliderInputFormField({
@ -36,6 +37,7 @@ export function SliderInputFormField({
tooltip,
defaultValue,
className,
numberInputClassName,
layout = FormLayout.Horizontal,
}: SliderInputFormFieldProps) {
const form = useFormContext();
@ -61,7 +63,7 @@ export function SliderInputFormField({
</FormLabel>
<div
className={cn(
'flex items-center gap-14 justify-between',
'flex items-center gap-4 justify-between',
{ 'w-3/4': isHorizontal },
className,
)}
@ -82,6 +84,7 @@ export function SliderInputFormField({
className={cn(
'h-6 w-10 p-0 text-center bg-bg-input border border-border-default text-text-secondary',
'[appearance:textfield] [&::-webkit-outer-spin-button]:appearance-none [&::-webkit-inner-spin-button]:appearance-none',
numberInputClassName,
)}
max={max}
min={min}

View File

@ -47,7 +47,7 @@ const CommandInput = React.forwardRef<
<CommandPrimitive.Input
ref={ref}
className={cn(
'flex h-11 w-full rounded-md bg-transparent py-3 text-sm outline-none placeholder:text-muted-foreground disabled:cursor-not-allowed disabled:opacity-50',
'flex min-h-8 w-full rounded-md bg-transparent py-2 text-sm outline-none placeholder:text-text-secondary disabled:cursor-not-allowed disabled:opacity-50',
className,
)}
{...props}

View File

@ -17,7 +17,7 @@ const Divider: React.FC<DividerProps> = ({
direction = 'horizontal',
type = 'horizontal',
text,
color = 'border-muted-foreground/50',
color = 'border-border-button',
margin = 'my-4',
className = '',
}) => {

View File

@ -2,7 +2,7 @@ import * as React from 'react';
import { cn } from '@/lib/utils';
import { Eye, EyeOff, Search } from 'lucide-react';
import { useState } from 'react';
import { useEffect, useMemo, useRef, useState } from 'react';
export interface InputProps
extends Omit<React.InputHTMLAttributes<HTMLInputElement>, 'prefix'> {
@ -17,6 +17,20 @@ const Input = React.forwardRef<HTMLInputElement, InputProps>(
const { defaultValue, ...restProps } = props;
const inputValue = isControlled ? value : defaultValue;
const [showPassword, setShowPassword] = useState(false);
const [prefixWidth, setPrefixWidth] = useState(0);
const [suffixWidth, setSuffixWidth] = useState(0);
const prefixRef = useRef<HTMLSpanElement>(null);
const suffixRef = useRef<HTMLSpanElement>(null);
useEffect(() => {
if (prefixRef.current) {
setPrefixWidth(prefixRef.current.offsetWidth);
}
if (suffixRef.current) {
setSuffixWidth(suffixRef.current.offsetWidth);
}
}, [prefix, suffix, prefixRef, suffixRef]);
const handleChange: React.ChangeEventHandler<HTMLInputElement> = (e) => {
if (type === 'number') {
const numValue = e.target.value === '' ? '' : Number(e.target.value);
@ -34,42 +48,60 @@ const Input = React.forwardRef<HTMLInputElement, InputProps>(
const isPasswordInput = type === 'password';
const inputEl = (
<input
ref={ref}
type={isPasswordInput && showPassword ? 'text' : type}
className={cn(
'peer/input',
'flex h-8 w-full rounded-md border-0.5 border-border-button bg-bg-input px-3 py-2 outline-none text-sm text-text-primary',
'file:border-0 file:bg-transparent file:text-sm file:font-medium file:text-foreground placeholder:text-text-disabled',
'focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-accent-primary',
'disabled:cursor-not-allowed disabled:opacity-50 transition-colors',
{
'pl-[calc(1em+1.25rem)]': !!prefix,
'pr-[calc(1em+1.25rem)]': !!suffix || isPasswordInput,
'pr-[calc(2em+2rem)]': !!suffix && isPasswordInput,
},
type === 'number' &&
'[appearance:textfield] [&::-webkit-outer-spin-button]:appearance-none [&::-webkit-inner-spin-button]:appearance-none',
className,
)}
value={inputValue ?? ''}
onChange={handleChange}
{...restProps}
/>
const inputEl = useMemo(
() => (
<input
ref={ref}
type={isPasswordInput && showPassword ? 'text' : type}
className={cn(
'peer/input',
'flex h-8 w-full rounded-md border-0.5 border-border-button bg-bg-input px-3 py-2 outline-none text-sm text-text-primary',
'file:border-0 file:bg-transparent file:text-sm file:font-medium file:text-foreground placeholder:text-text-disabled',
'focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-accent-primary',
'disabled:cursor-not-allowed disabled:opacity-50 transition-colors',
type === 'number' &&
'[appearance:textfield] [&::-webkit-outer-spin-button]:appearance-none [&::-webkit-inner-spin-button]:appearance-none',
className,
)}
style={{
paddingLeft: !!prefix ? `${prefixWidth}px` : '',
paddingRight: isPasswordInput
? '40px'
: !!suffix
? `${suffixWidth}px`
: '',
}}
value={inputValue ?? ''}
onChange={handleChange}
{...restProps}
/>
),
[
prefixWidth,
suffixWidth,
isPasswordInput,
inputValue,
className,
handleChange,
restProps,
],
);
if (prefix || suffix || isPasswordInput) {
return (
<div className="relative">
{prefix && (
<span className="absolute left-0 top-[50%] translate-y-[-50%]">
<span
ref={prefixRef}
className="absolute left-0 top-[50%] translate-y-[-50%]"
>
{prefix}
</span>
)}
{inputEl}
{suffix && (
<span
ref={suffixRef}
className={cn('absolute right-0 top-[50%] translate-y-[-50%]', {
'right-14': isPasswordInput,
})}

View File

@ -289,12 +289,12 @@ export const MultiSelect = React.forwardRef<
{...props}
onClick={handleTogglePopover}
className={cn(
'flex w-full p-1 rounded-md border min-h-10 h-auto items-center justify-between bg-inherit hover:bg-inherit [&_svg]:pointer-events-auto',
'flex w-full p-1 rounded-md border border-border-button min-h-10 h-auto placeholder:text-text-disabled items-center justify-between bg-bg-input hover:bg-bg-input [&_svg]:pointer-events-auto',
className,
)}
>
{selectedValues.length > 0 ? (
<div className="flex justify-between items-center w-full">
<div className="flex justify-between items-center w-full group">
<div className="flex flex-wrap items-center">
{selectedValues?.slice(0, maxCount)?.map((value) => {
const option = flatOptions.find((o) => o.value === value);
@ -348,9 +348,9 @@ export const MultiSelect = React.forwardRef<
</Badge>
)}
</div>
<div className="flex items-center justify-between">
<div className="flex items-center justify-between ">
<XIcon
className="h-4 mx-2 cursor-pointer text-muted-foreground"
className="h-4 mx-2 cursor-pointer text-text-secondary hidden group-hover:block"
onClick={(event) => {
event.stopPropagation();
handleClear();
@ -358,17 +358,17 @@ export const MultiSelect = React.forwardRef<
/>
<Separator
orientation="vertical"
className="flex min-h-6 h-full"
className="min-h-6 h-full hidden group-hover:flex"
/>
<ChevronDown className="h-4 mx-2 cursor-pointer text-muted-foreground" />
<ChevronDown className="h-4 mx-2 cursor-pointer text-text-secondary" />
</div>
</div>
) : (
<div className="flex items-center justify-between w-full mx-auto">
<span className="text-sm text-muted-foreground mx-3">
<span className="text-sm text-text-secondary mx-3">
{placeholder}
</span>
<ChevronDown className="h-4 cursor-pointer text-muted-foreground mx-2" />
<ChevronDown className="h-4 cursor-pointer text-text-secondary mx-2" />
</div>
)}
</Button>
@ -379,14 +379,16 @@ export const MultiSelect = React.forwardRef<
onEscapeKeyDown={() => setIsPopoverOpen(false)}
>
<Command className="p-5 pb-8">
<CommandInput
placeholder={t('common.search') + '...'}
onKeyDown={handleInputKeyDown}
/>
{options && options.length > 0 && (
<CommandInput
placeholder={t('common.search') + '...'}
onKeyDown={handleInputKeyDown}
/>
)}
<CommandList className="mt-2">
<CommandEmpty>No results found.</CommandEmpty>
<CommandGroup>
{showSelectAll && (
{showSelectAll && options && options.length > 0 && (
<CommandItem
key="all"
onSelect={toggleAll}
@ -454,12 +456,14 @@ export const MultiSelect = React.forwardRef<
/>
</>
)}
<CommandItem
onSelect={() => setIsPopoverOpen(false)}
className="flex-1 justify-center cursor-pointer max-w-full"
>
{t('common.close')}
</CommandItem>
{options && options.length > 0 && (
<CommandItem
onSelect={() => setIsPopoverOpen(false)}
className="flex-1 justify-center cursor-pointer max-w-full"
>
{t('common.close')}
</CommandItem>
)}
</div>
</CommandGroup>
</CommandList>

View File

@ -179,3 +179,8 @@ export enum JsonSchemaDataType {
Array = 'array',
Object = 'object',
}
export enum SwitchLogicOperator {
And = 'and',
Or = 'or',
}

View File

@ -0,0 +1,12 @@
import { SwitchLogicOperator } from '@/constants/agent';
import { buildOptions } from '@/utils/form';
import { useTranslation } from 'react-i18next';
export function useBuildSwitchLogicOperatorOptions() {
const { t } = useTranslation();
return buildOptions(
SwitchLogicOperator,
t,
'flow.switchLogicOperatorOptions',
);
}

View File

@ -736,9 +736,15 @@ Example: https://fsn1.your-objectstorage.com`,
google_drivePrimaryAdminTip:
'Email address that has access to the Drive content being synced.',
google_driveMyDriveEmailsTip:
'Comma-separated emails whose My Drive contents should be indexed (include the primary admin).',
'Comma-separated emails whose "My Drive" contents should be indexed (include the primary admin).',
google_driveSharedFoldersTip:
'Comma-separated Google Drive folder links to crawl.',
moodleDescription:
'Connect to your Moodle LMS to sync course content, forums, and resources.',
moodleUrlTip:
'The base URL of your Moodle instance (e.g., https://moodle.university.edu). Do not include /webservice or /login.',
moodleTokenTip:
'Generate a web service token in Moodle: Go to Site administration → Server → Web services → Manage tokens. The user must be enrolled in the courses you want to sync.',
jiraDescription:
'Connect your Jira workspace to sync issues, comments, and attachments.',
jiraBaseUrlTip:
@ -1046,7 +1052,7 @@ Example: https://fsn1.your-objectstorage.com`,
downloadFileType: 'Download file type',
formatTypeError: 'Format or type error',
variableNameMessage:
'Variable name can only contain letters and underscores',
'Variable name can only contain letters and underscores and numbers',
variableDescription: 'Variable Description',
defaultValue: 'Default Value',
conversationVariable: 'Conversation variable',

View File

@ -980,7 +980,7 @@ General实体和关系提取提示来自 GitHub - microsoft/graphrag基于
downloadFileTypeTip: '文件下载的类型',
downloadFileType: '文件类型',
formatTypeError: '格式或类型错误',
variableNameMessage: '名称只能包含字母和下划线',
variableNameMessage: '名称只能包含字母,数字和下划线',
variableDescription: '变量的描述',
defaultValue: '默认值',
conversationVariable: '会话变量',

View File

@ -10,6 +10,7 @@ import {
JsonSchemaDataType,
Operator,
ProgrammingLanguage,
SwitchLogicOperator,
SwitchOperatorOptions,
initialLlmBaseValues,
} from '@/constants/agent';
@ -51,8 +52,6 @@ import {
export const BeginId = 'begin';
export const SwitchLogicOperatorOptions = ['and', 'or'];
export const CommonOperatorList = Object.values(Operator).filter(
(x) => x !== Operator.Note,
);
@ -308,7 +307,7 @@ export const initialExeSqlValues = {
export const initialSwitchValues = {
conditions: [
{
logical_operator: SwitchLogicOperatorOptions[0],
logical_operator: SwitchLogicOperator.And,
items: [
{
operator: SwitchOperatorOptions[0].value,

View File

@ -11,16 +11,17 @@ import {
import { RAGFlowSelect } from '@/components/ui/select';
import { Separator } from '@/components/ui/separator';
import { Textarea } from '@/components/ui/textarea';
import { SwitchLogicOperator } from '@/constants/agent';
import { useBuildSwitchOperatorOptions } from '@/hooks/logic-hooks/use-build-operator-options';
import { useBuildSwitchLogicOperatorOptions } from '@/hooks/logic-hooks/use-build-options';
import { cn } from '@/lib/utils';
import { zodResolver } from '@hookform/resolvers/zod';
import { t } from 'i18next';
import { X } from 'lucide-react';
import { memo, useCallback, useMemo } from 'react';
import { memo, useCallback } from 'react';
import { useFieldArray, useForm, useFormContext } from 'react-hook-form';
import { useTranslation } from 'react-i18next';
import { z } from 'zod';
import { SwitchLogicOperatorOptions } from '../../constant';
import { IOperatorForm } from '../../interface';
import { FormWrapper } from '../components/form-wrapper';
import { QueryVariable } from '../components/query-variable';
@ -185,12 +186,7 @@ function SwitchForm({ node }: IOperatorForm) {
control: form.control,
});
const switchLogicOperatorOptions = useMemo(() => {
return SwitchLogicOperatorOptions.map((x) => ({
value: x,
label: t(`flow.switchLogicOperatorOptions.${x}`),
}));
}, [t]);
const switchLogicOperatorOptions = useBuildSwitchLogicOperatorOptions();
useWatchFormChange(node?.id, form);
@ -253,7 +249,7 @@ function SwitchForm({ node }: IOperatorForm) {
<BlockButton
onClick={() =>
append({
logical_operator: SwitchLogicOperatorOptions[0],
logical_operator: SwitchLogicOperator.And,
[ItemKey]: [
{
operator: switchOperatorOptions[0].value,

View File

@ -18,7 +18,7 @@ export const GlobalFormFields = [
placeholder: t('common.namePlaceholder'),
required: true,
validation: {
pattern: /^[a-zA-Z_]+$/,
pattern: /^[a-zA-Z_0-9]+$/,
message: t('flow.variableNameMessage'),
},
type: FormFieldType.Text,

View File

@ -3,6 +3,7 @@ import { BlockButton, Button } from '@/components/ui/button';
import { Input } from '@/components/ui/input';
import { Segmented } from '@/components/ui/segmented';
import { t } from 'i18next';
import { isEmpty } from 'lodash';
import { Trash2, X } from 'lucide-react';
import { useCallback } from 'react';
import { FieldValues } from 'react-hook-form';
@ -36,14 +37,19 @@ export const useObjectFields = () => {
path: (string | number)[] = [],
): Array<{ path: (string | number)[]; message: string }> => {
const errors: Array<{ path: (string | number)[]; message: string }> = [];
if (obj !== null && typeof obj === 'object' && !Array.isArray(obj)) {
if (typeof obj === 'object' && !Array.isArray(obj)) {
if (isEmpty(obj)) {
errors.push({
path: [...path],
message: 'No empty parameters are allowed.',
});
}
for (const key in obj) {
if (obj.hasOwnProperty(key)) {
if (!/^[a-zA-Z_]+$/.test(key)) {
if (!/^[a-zA-Z_0-9]+$/.test(key)) {
errors.push({
path: [...path, key],
message: `Key "${key}" is invalid. Keys can only contain letters and underscores.`,
message: `Key "${key}" is invalid. Keys can only contain letters and underscores and numbers.`,
});
}
const nestedErrors = validateKeys(obj[key], [...path, key]);
@ -108,6 +114,21 @@ export const useObjectFields = () => {
}
}, []);
const arrayObjectValidate = useCallback((value: any) => {
try {
if (validateKeys(value, [])?.length > 0) {
throw new Error(t('flow.formatTypeError'));
}
if (value && typeof value === 'string' && !JSON.parse(value)) {
throw new Error(t('flow.formatTypeError'));
}
return true;
} catch (e) {
console.log('object-render-error', e, value);
throw new Error(t('flow.formatTypeError'));
}
}, []);
const arrayStringRender = useCallback((field: FieldValues, type = 'text') => {
const values = Array.isArray(field.value)
? field.value
@ -253,8 +274,9 @@ export const useObjectFields = () => {
const handleCustomValidate = (value: TypesWithArray) => {
switch (value) {
case TypesWithArray.Object:
case TypesWithArray.ArrayObject:
return objectValidate;
case TypesWithArray.ArrayObject:
return arrayObjectValidate;
case TypesWithArray.ArrayString:
return arrayStringValidate;
case TypesWithArray.ArrayNumber:
@ -284,6 +306,7 @@ export const useObjectFields = () => {
return {
objectRender,
objectValidate,
arrayObjectValidate,
arrayStringRender,
arrayStringValidate,
arrayNumberRender,

View File

@ -1,236 +0,0 @@
import { SliderInputSwitchFormField } from '@/components/llm-setting-items/slider';
import { SelectWithSearch } from '@/components/originui/select-with-search';
import {
FormControl,
FormField,
FormItem,
FormLabel,
FormMessage,
} from '@/components/ui/form';
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from '@/components/ui/select';
import {
LlmModelType,
ModelVariableType,
settledModelVariableMap,
} from '@/constants/knowledge';
import { useTranslate } from '@/hooks/common-hooks';
import { useComposeLlmOptionsByModelTypes } from '@/hooks/llm-hooks';
import { camelCase, isEqual } from 'lodash';
import { useCallback } from 'react';
import { useFormContext } from 'react-hook-form';
import { z } from 'zod';
interface LlmSettingFieldItemsProps {
prefix?: string;
options?: any[];
}
const LlmSettingEnableSchema = {
temperatureEnabled: z.boolean(),
topPEnabled: z.boolean(),
presencePenaltyEnabled: z.boolean(),
frequencyPenaltyEnabled: z.boolean(),
};
export const LlmSettingSchema = {
llm_id: z.string(),
parameter: z.string().optional(),
temperature: z.coerce.number().optional(),
top_p: z.coerce.number().optional(),
presence_penalty: z.coerce.number().optional(),
frequency_penalty: z.coerce.number().optional(),
...LlmSettingEnableSchema,
// maxTokensEnabled: z.boolean(),
};
export function LlmSettingFieldItems({
prefix,
options,
}: LlmSettingFieldItemsProps) {
const form = useFormContext();
const { t } = useTranslate('chat');
const modelOptions = useComposeLlmOptionsByModelTypes([
LlmModelType.Chat,
LlmModelType.Image2text,
]);
const handleChange = useCallback(
(parameter: string) => {
const values =
settledModelVariableMap[
parameter as keyof typeof settledModelVariableMap
];
const enabledKeys = Object.keys(LlmSettingEnableSchema);
for (const key in values) {
if (Object.prototype.hasOwnProperty.call(values, key)) {
const element = values[key as keyof typeof values];
form.setValue(`${prefix}.${key}`, element);
}
}
if (enabledKeys && enabledKeys.length) {
for (const key of enabledKeys) {
form.setValue(`${prefix}.${key}`, true);
}
}
},
[form, prefix],
);
const parameterOptions = Object.values(ModelVariableType).map((x) => ({
label: t(camelCase(x)),
value: x,
})) as unknown as { label: string; value: ModelVariableType | 'Custom' }[];
parameterOptions.push({
label: t(camelCase('Custom')),
value: 'Custom',
});
const getFieldWithPrefix = useCallback(
(name: string) => {
return prefix ? `${prefix}.${name}` : name;
},
[prefix],
);
const checkParameterIsEquel = () => {
const [
parameter,
topPValue,
frequencyPenaltyValue,
temperatureValue,
presencePenaltyValue,
] = form.getValues([
getFieldWithPrefix('parameter'),
getFieldWithPrefix('temperature'),
getFieldWithPrefix('top_p'),
getFieldWithPrefix('frequency_penalty'),
getFieldWithPrefix('presence_penalty'),
]);
if (parameter && parameter !== 'Custom') {
const parameterValue =
settledModelVariableMap[parameter as keyof typeof ModelVariableType];
const parameterRealValue = {
top_p: topPValue,
temperature: temperatureValue,
frequency_penalty: frequencyPenaltyValue,
presence_penalty: presencePenaltyValue,
};
if (!isEqual(parameterValue, parameterRealValue)) {
form.setValue(getFieldWithPrefix('parameter'), 'Custom');
}
}
};
return (
<div className="space-y-5">
<FormField
control={form.control}
name={getFieldWithPrefix('llm_id')}
render={({ field }) => (
<FormItem>
<FormLabel>
<span className="text-destructive mr-1"> *</span>
{t('model')}
</FormLabel>
<FormControl>
<SelectWithSearch
options={options || modelOptions}
triggerClassName="!bg-bg-input"
{...field}
></SelectWithSearch>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name={getFieldWithPrefix('parameter')}
render={({ field }) => (
<FormItem className="flex justify-between gap-4 items-center">
<FormLabel>{t('freedom')}</FormLabel>
<FormControl>
<div className="w-28">
<Select
{...field}
onValueChange={(val) => {
handleChange(val);
field.onChange(val);
}}
>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent>
{parameterOptions.map((x) => (
<SelectItem value={x.value} key={x.value}>
{x.label}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<SliderInputSwitchFormField
name={getFieldWithPrefix('temperature')}
checkName={getFieldWithPrefix('temperatureEnabled')}
label="temperature"
max={1}
min={0}
step={0.01}
onChange={() => {
checkParameterIsEquel();
}}
></SliderInputSwitchFormField>
<SliderInputSwitchFormField
name={getFieldWithPrefix('top_p')}
checkName={getFieldWithPrefix('topPEnabled')}
label="topP"
max={1}
step={0.01}
min={0}
onChange={() => {
checkParameterIsEquel();
}}
></SliderInputSwitchFormField>
<SliderInputSwitchFormField
name={getFieldWithPrefix('presence_penalty')}
checkName={getFieldWithPrefix('presencePenaltyEnabled')}
label="presencePenalty"
max={1}
step={0.01}
min={0}
onChange={() => {
checkParameterIsEquel();
}}
></SliderInputSwitchFormField>
<SliderInputSwitchFormField
name={getFieldWithPrefix('frequency_penalty')}
checkName={getFieldWithPrefix('frequencyPenaltyEnabled')}
label="frequencyPenalty"
max={1}
step={0.01}
min={0}
onChange={() => {
checkParameterIsEquel();
}}
></SliderInputSwitchFormField>
{/* <SliderInputSwitchFormField
name={getFieldWithPrefix('max_tokens')}
checkName="maxTokensEnabled"
label="maxTokens"
max={128000}
></SliderInputSwitchFormField> */}
</div>
);
}

View File

@ -1,6 +1,10 @@
// src/pages/next-search/search-setting.tsx
import { AvatarUpload } from '@/components/avatar-upload';
import {
LlmSettingFieldItems,
LlmSettingSchema,
} from '@/components/llm-setting-items/next';
import {
MetadataFilter,
MetadataFilterSchema,
@ -46,10 +50,10 @@ import {
IllmSettingProps,
useUpdateSearch,
} from '../next-searches/hooks';
import {
LlmSettingFieldItems,
LlmSettingSchema,
} from './search-setting-aisummery-config';
// import {
// LlmSettingFieldItems,
// LlmSettingSchema,
// } from './search-setting-aisummery-config';
interface SearchSettingProps {
open: boolean;
@ -397,6 +401,7 @@ const SearchSetting: React.FC<SearchSettingProps> = ({
isTooltipShown
similarityName="search_config.similarity_threshold"
vectorSimilarityWeightName="search_config.vector_similarity_weight"
numberInputClassName="rounded-sm"
></SimilaritySliderFormField>
{/* Rerank Model */}
<FormField
@ -462,7 +467,7 @@ const SearchSetting: React.FC<SearchSettingProps> = ({
<FormControl>
<Input
type={'number'}
className="h-7 w-20 bg-bg-card"
className="h-7 w-20 bg-bg-card border border-border-button rounded-sm"
max={2048}
min={0}
step={1}
@ -493,9 +498,19 @@ const SearchSetting: React.FC<SearchSettingProps> = ({
)}
/>
{aiSummaryDisabled && (
// <LlmSettingFieldItems
// prefix="search_config.llm_setting"
// options={aiSummeryModelOptions}
// ></LlmSettingFieldItems>
<LlmSettingFieldItems
prefix="search_config.llm_setting"
options={aiSummeryModelOptions}
showFields={[
'temperature',
'top_p',
'presence_penalty',
'frequency_penalty',
]}
></LlmSettingFieldItems>
)}
{/* Feature Controls */}

View File

@ -9,7 +9,8 @@ export enum DataSourceKey {
NOTION = 'notion',
DISCORD = 'discord',
GOOGLE_DRIVE = 'google_drive',
// GMAIL = 'gmail',
MOODLE = 'moodle',
// GMAIL = 'gmail',
JIRA = 'jira',
// SHAREPOINT = 'sharepoint',
// SLACK = 'slack',
@ -42,6 +43,11 @@ export const DataSourceInfo = {
description: t(`setting.${DataSourceKey.GOOGLE_DRIVE}Description`),
icon: <SvgIcon name={'data-source/google-drive'} width={38} />,
},
[DataSourceKey.MOODLE]: {
name: 'Moodle',
description: t(`setting.${DataSourceKey.MOODLE}Description`),
icon: <SvgIcon name={'data-source/moodle'} width={38} />,
},
[DataSourceKey.JIRA]: {
name: 'Jira',
description: t(`setting.${DataSourceKey.JIRA}Description`),
@ -116,7 +122,7 @@ export const DataSourceFormFields = {
required: false,
placeholder: 'https://fsn1.your-objectstorage.com',
tooltip: t('setting.S3CompatibleEndpointUrlTip'),
shouldRender: (formValues) => {
shouldRender: (formValues: any) => {
return formValues?.config?.bucket_type === 's3_compatible';
},
},
@ -287,6 +293,21 @@ export const DataSourceFormFields = {
defaultValue: 'uploaded',
},
],
[DataSourceKey.MOODLE]: [
{
label: 'Moodle URL',
name: 'config.moodle_url',
type: FormFieldType.Text,
required: true,
placeholder: 'https://moodle.example.com',
},
{
label: 'API Token',
name: 'config.credentials.moodle_token',
type: FormFieldType.Password,
required: true,
},
],
[DataSourceKey.JIRA]: [
{
label: 'Jira Base URL',
@ -456,6 +477,16 @@ export const DataSourceFormDefaultValues = {
},
},
},
[DataSourceKey.MOODLE]: {
name: '',
source: DataSourceKey.MOODLE,
config: {
moodle_url: '',
credentials: {
moodle_token: '',
},
},
},
[DataSourceKey.JIRA]: {
name: '',
source: DataSourceKey.JIRA,

View File

@ -44,6 +44,12 @@ const dataSourceTemplates = [
description: DataSourceInfo[DataSourceKey.NOTION].description,
icon: DataSourceInfo[DataSourceKey.NOTION].icon,
},
{
id: DataSourceKey.MOODLE,
name: DataSourceInfo[DataSourceKey.MOODLE].name,
description: DataSourceInfo[DataSourceKey.MOODLE].description,
icon: DataSourceInfo[DataSourceKey.MOODLE].icon,
},
{
id: DataSourceKey.JIRA,
name: DataSourceInfo[DataSourceKey.JIRA].name,