Refactor Document API (#2833)

### What problem does this PR solve?

Refactor Document API

### Type of change


- [x] Refactoring

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
This commit is contained in:
liuhua
2024-10-14 20:03:33 +08:00
committed by GitHub
parent df223eddf3
commit 6329427ad5
11 changed files with 393 additions and 418 deletions

View File

@ -24,7 +24,7 @@ Creates a knowledge base (dataset).
### Parameters
#### name: *Required*
#### name: `str`, *Required*
The unique name of the dataset to create. It must adhere to the following requirements:
@ -36,81 +36,70 @@ The unique name of the dataset to create. It must adhere to the following requir
- Maximum 65,535 characters.
- Case-insensitive.
#### avatar
#### avatar: `str`
Base64 encoding of the avatar. Defaults to `""`
#### tenant_id
#### tenant_id: `str`
The id of the tenant associated with the created dataset is used to identify different users. Defaults to `None`.
- When creating a dataset, `tenant_id` must not be provided.
- When updating a dataset, `tenant_id` cannot be changed.
- If creating a dataset, tenant_id must not be provided.
- If updating a dataset, tenant_id can't be changed.
#### description
#### description: `str`
The description of the created dataset. Defaults to `""`.
#### language
#### language: `str`
The language setting of the created dataset. Defaults to `"English"`.
The language setting of the created dataset. Defaults to `"English"`. ????????????
#### embedding_model
#### embedding_model: `str`
The specific model used by the dataset to generate vector embeddings. Defaults to `""`.
- When creating a dataset, `embedding_model` must not be provided.
- When updating a dataset, `embedding_model` cannot be changed.
- If creating a dataset, embedding_model must not be provided.
- If updating a dataset, embedding_model can't be changed.
#### permission
#### permission: `str`
The person who can operate on the dataset. Defaults to `"me"`.
Specify who can operate on the dataset. Defaults to `"me"`.
#### document_count
#### document_count: `int`
The number of documents associated with the dataset. Defaults to `0`.
:::tip NOTE
When updating a dataset, `document_count` cannot be changed.
:::
- If updating a dataset, `document_count` can't be changed.
#### chunk_count
#### chunk_count: `int`
The number of data chunks generated or processed by the created dataset. Defaults to `0`.
:::tip NOTE
When updating a dataset, `chunk_count` cannot be changed.
:::
- If updating a dataset, chunk_count can't be changed.
#### parse_method
#### parse_method, `str`
The method used by the dataset to parse and process data. Defaults to `"naive"`.
The method used by the dataset to parse and process data.
:::tip NOTE
When updating `parse_method` in a dataset, `chunk_count` must be greater than 0.
:::
- If updating parse_method in a dataset, chunk_count must be greater than 0. Defaults to `"naive"`.
#### parser_config
#### parser_config, `Dataset.ParserConfig`
The parser configuration of the dataset. A `ParserConfig` object contains the following attributes:
- `chunk_token_count`: Defaults to `128`.
- `layout_recognize`: Defaults to `True`.
- `delimiter`: Defaults to `'\n!?。;!?'`.
- `task_page_size`: Defaults to `12`.
The configuration settings for the parser used by the dataset.
### Returns
- Success: A `dataset` object.
- Failure: `Exception`
```python
DataSet
description: dataset object
```
### Examples
```python
from ragflow import RAGFlow
rag_object = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
ds = rag_object.create_dataset(name="kb_1")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
ds = rag.create_dataset(name="kb_1")
```
---
@ -118,27 +107,28 @@ ds = rag_object.create_dataset(name="kb_1")
## Delete knowledge bases
```python
RAGFlow.delete_datasets(ids: list[str] = None)
RAGFlow.delete_datasets(ids: List[str] = None)
```
Deletes knowledge bases by name or ID.
Deletes knowledge bases.
### Parameters
#### ids
#### ids: `List[str]`
The ids of the datasets to be deleted.
The IDs of the knowledge bases to delete.
### Returns
- Success: No value is returned.
- Failure: `Exception`
```python
no return
```
### Examples
#### Delete knowledge bases by name
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
rag.delete_datasets(ids=["id_1","id_2"])
```
@ -154,84 +144,76 @@ RAGFlow.list_datasets(
desc: bool = True,
id: str = None,
name: str = None
) -> list[DataSet]
) -> List[DataSet]
```
Lists all knowledge bases.
Lists all knowledge bases in the RAGFlow system.
### Parameters
#### page
#### page: `int`
The current page number to retrieve from the paginated data. This parameter determines which set of records will be fetched. Defaults to `1`.
#### page_size
#### page_size: `int`
The number of records to retrieve per page. This controls how many records will be included in each page. Defaults to `1024`.
#### order_by
#### order_by: `str`
The attribute by which the results are sorted. Defaults to `"create_time"`.
The field by which the records should be sorted. This specifies the attribute or column used to order the results. Defaults to `"create_time"`.
#### desc
#### desc: `bool`
Indicates whether to sort the results in descending order. Defaults to `True`.
Whether the sorting should be in descending order. Defaults to `True`.
#### id
#### id: `str`
The ID of the dataset to retrieve. Defaults to `None`.
The id of the dataset to be got. Defaults to `None`.
#### name
#### name: `str`
The name of the dataset to retrieve. Defaults to `None`.
The name of the dataset to be got. Defaults to `None`.
### Returns
- Success: A list of `DataSet` objects representing the retrieved knowledge bases.
- Failure: `Exception`.
### Examples
#### Retrieve a list of knowledge bases associated with the current user
```python
for ds in rag_object.list_datasets():
print(ds.name)
List[DataSet]
description:the list of datasets.
```
#### Retrieve a knowledge base by ID
```python
ds = rag_object.list_datasets(id = "id_1")
print(ds.name)
```
---
## Update knowledge base
```python
DataSet.update(update_message: dict)
```
Updates the current knowledge base.
### Parameters
#### update_message
### Returns
- Success: No value is returned.
- Failure: `Exception`
### Examples
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
ds = rag.list_datasets(name="kb_1")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
for ds in rag.list_datasets():
print(ds)
```
---
## Update knowledge base
```python
DataSet.update(update_message: dict)
```
### Returns
```python
no return
```
### Examples
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
ds = rag.get_dataset(name="kb_1")
ds.update({"parse_method":"manual", ...}}
```
@ -249,8 +231,6 @@ RAGFLOW.upload_document(ds:DataSet, name:str, blob:bytes)-> bool
### Parameters
#### ds
#### name
#### blob
@ -354,7 +334,7 @@ Duration of the processing in seconds or minutes. Defaults to `0.0`.
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
doc = rag.get_document(id="wdfxb5t547d",name='testdocument.txt')
print(doc)
```
@ -376,7 +356,7 @@ bool
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
doc = rag.get_document(id="wdfxb5t547d")
doc.parser_method= "manual"
doc.save()
@ -399,7 +379,7 @@ bytes of the document.
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
doc = rag.get_document(id="wdfxb5t547d")
open("~/ragflow.txt", "w+").write(doc.download())
print(doc)
@ -410,7 +390,7 @@ print(doc)
## List documents
```python
Dataset.list_docs(keywords: str=None, offset: int=0, limit:int = -1) -> list[Document]
Dataset.list_docs(keywords: str=None, offset: int=0, limit:int = -1) -> List[Document]
```
### Parameters
@ -425,18 +405,18 @@ The beginning number of records for paging. Defaults to `0`.
#### limit: `int`
Records number to return, -1 means all of them.
Records number to return, -1 means all of them. Records number to return, -1 means all of them.
### Returns
list[Document]
List[Document]
### Examples
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
ds = rag.create_dataset(name="kb_1")
filename1 = "~/ragflow.txt"
@ -466,7 +446,7 @@ description: delete success or not
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
ds = rag.create_dataset(name="kb_1")
filename1 = "~/ragflow.txt"
@ -599,7 +579,7 @@ chunk
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
doc = rag.get_document(id="wdfxb5t547d")
chunk = doc.add_chunk(content="xxxxxxx")
```
@ -621,7 +601,7 @@ bool
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
doc = rag.get_document(id="wdfxb5t547d")
chunk = doc.add_chunk(content="xxxxxxx")
chunk.delete()
@ -644,7 +624,7 @@ bool
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
doc = rag.get_document(id="wdfxb5t547d")
chunk = doc.add_chunk(content="xxxxxxx")
chunk.content = "sdfx"
@ -656,7 +636,7 @@ chunk.save()
## Retrieval
```python
RAGFlow.retrieval(question:str, datasets:list[Dataset], document=list[Document]=None, offset:int=0, limit:int=6, similarity_threshold:float=0.1, vector_similarity_weight:float=0.3, top_k:int=1024) -> list[Chunk]
RAGFlow.retrieval(question:str, datasets:List[Dataset], document=List[Document]=None, offset:int=0, limit:int=6, similarity_threshold:float=0.1, vector_similarity_weight:float=0.3, top_k:int=1024) -> List[Chunk]
```
### Parameters
@ -665,11 +645,11 @@ RAGFlow.retrieval(question:str, datasets:list[Dataset], document=list[Document]=
The user query or query keywords. Defaults to `""`.
#### datasets: `list[Dataset]`, *Required*
#### datasets: `List[Dataset]`, *Required*
The scope of datasets.
#### document: `list[Document]`
#### document: `List[Document]`
The scope of document. `None` means no limitation. Defaults to `None`.
@ -695,14 +675,14 @@ Number of records engaged in vector cosine computaton. Defaults to `1024`.
### Returns
list[Chunk]
List[Chunk]
### Examples
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
ds = rag.get_dataset(name="ragflow")
name = 'ragflow_test.txt'
path = 'test_data/ragflow_test.txt'
@ -733,7 +713,7 @@ Chat APIs
RAGFlow.create_chat(
name: str = "assistant",
avatar: str = "path",
knowledgebases: list[DataSet] = ["kb1"],
knowledgebases: List[DataSet] = ["kb1"],
llm: Chat.LLM = None,
prompt: Chat.Prompt = None
) -> Chat
@ -754,7 +734,7 @@ The name of the created chat. Defaults to `"assistant"`.
The icon of the created chat. Defaults to `"path"`.
#### knowledgebases: `list[DataSet]`
#### knowledgebases: `List[DataSet]`
Select knowledgebases associated. Defaults to `["kb1"]`.
@ -796,7 +776,7 @@ You are an intelligent assistant. Please summarize the content of the knowledge
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
kb = rag.get_dataset(name="kb_1")
assi = rag.create_chat("Miss R", knowledgebases=[kb])
```
@ -820,7 +800,7 @@ no return
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
kb = rag.get_knowledgebase(name="kb_1")
assi = rag.create_chat("Miss R" knowledgebases=[kb])
assi.update({"temperature":0.8})
@ -831,7 +811,7 @@ assi.update({"temperature":0.8})
## Delete chats
```python
RAGFlow.delete_chats(ids: list[str] = None)
RAGFlow.delete_chats(ids: List[str] = None)
```
### Parameters
@ -851,7 +831,7 @@ no return
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
rag.delete_chats(ids=["id_1","id_2"])
```
@ -867,7 +847,7 @@ RAGFlow.list_chats(
desc: bool = True,
id: str = None,
name: str = None
) -> list[Chat]
) -> List[Chat]
```
### Parameters
@ -910,7 +890,7 @@ A list of chat objects.
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
for assi in rag.list_chats():
print(assi)
```
@ -940,7 +920,7 @@ The id of the created session is used to identify different sessions.
The name of the created session. Defaults to `"New session"`.
#### messages: `list[Message]`
#### messages: `List[Message]`
The messages of the created session.
- messages cannot be provided.
@ -963,7 +943,7 @@ The id of associated chat
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
assi = rag.list_chats(name="Miss R")
assi = assi[0]
sess = assi.create_session()
@ -985,7 +965,7 @@ no return
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
assi = rag.list_chats(name="Miss R")
assi = assi[0]
sess = assi.create_session("new_session")
@ -1023,7 +1003,7 @@ The id of the message. `id` is automatically generated. Defaults to `None`. ????
The content of the message. Defaults to `"Hi! I am your assistant, can I help you?"`.
#### reference: `list[Chunk]`
#### reference: `List[Chunk]`
The auto-generated reference of the message. Each `chunk` object includes the following attributes:
@ -1045,7 +1025,7 @@ The auto-generated reference of the message. Each `chunk` object includes the fo
A similarity score based on vector representations. This score is obtained by converting texts, words, or objects into vectors and then calculating the cosine similarity or other distance measures between these vectors to determine the similarity in vector space. A higher value indicates greater similarity in the vector space. Defaults to `None`. ?????????????????????????????????
- **term_similarity**: `float`
The similarity score based on terms or keywords. This score is calculated by comparing the similarity of key terms between texts or datasets, typically measuring how similar two words or phrases are in meaning or context. A higher value indicates a stronger similarity between terms. Defaults to `None`. ???????????????????
- **position**: `list[string]`
- **position**: `List[string]`
Indicates the position or index of keywords or specific terms within the text. An array is typically used to mark the location of keywords or specific elements, facilitating precise operations or analysis of the text. Defaults to `None`. ??????????????
### Examples
@ -1053,7 +1033,7 @@ The auto-generated reference of the message. Each `chunk` object includes the fo
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
assi = rag.list_chats(name="Miss R")
assi = assi[0]
sess = assi.create_session()
@ -1084,12 +1064,12 @@ Chat.list_sessions(
desc: bool = True,
id: str = None,
name: str = None
) -> list[Session]
) -> List[Session]
```
### Returns
list[Session]
List[Session]
description: the List contains information about multiple assistant object, with each dictionary containing information about one assistant.
### Examples
@ -1097,7 +1077,7 @@ description: the List contains information about multiple assistant object, with
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
assi = rag.list_chats(name="Miss R")
assi = assi[0]
for sess in assi.list_sessions():
@ -1140,7 +1120,7 @@ The name of the chat to be retrieved.
## Delete session
```python
Chat.delete_sessions(ids:list[str] = None)
Chat.delete_sessions(ids:List[str] = None)
```
### Returns
@ -1152,13 +1132,13 @@ no return
```python
from ragflow import RAGFlow
rag = RAGFlow(api_key="<YOUR_API_KEY>", base_url="http://<YOUR_BASE_URL>:9380")
rag = RAGFlow(api_key="xxxxxx", base_url="http://xxx.xx.xx.xxx:9380")
assi = rag.list_chats(name="Miss R")
assi = assi[0]
assi.delete_sessions(ids=["id_1","id_2"])
```
### Parameters
#### ids: `list[string]`
#### ids: `List[string]`
IDs of the sessions to be deleted.
- `None`