mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-02-06 02:25:05 +08:00
feat/add MySQL and PostgreSQL data source connectors (#12817)
### What problem does this PR solve? This PR adds MySQL and PostgreSQL as data source connectors, allowing users to import data directly from relational databases into RAGFlow for RAG workflows. Many users store their knowledge in databases (product catalogs, documentation, FAQs, etc.) and currently have no way to sync this data into RAGFlow without exporting to files first. This feature lets them connect directly to their databases, run SQL queries, and automatically create documents from the results. Closes #763 Closes #11560 ### Type of change - [ ] Bug Fix (non-breaking change which fixes an issue) - [x] New Feature (non-breaking change which adds functionality) - [ ] Documentation Update - [ ] Refactoring - [ ] Performance Improvement - [ ] Other (please describe): ### What this PR does **New capabilities:** - Connect to MySQL and PostgreSQL databases - Run custom SQL queries to extract data - Map database columns to document content (vectorized) and metadata (searchable) - Support incremental sync using a timestamp column - Full frontend UI with connection form and tooltips **Files changed:** Backend: - `common/constants.py` - Added MYSQL/POSTGRESQL to FileSource enum - `common/data_source/config.py` - Added to DocumentSource enum - `common/data_source/rdbms_connector.py` - New connector (368 lines) - `common/data_source/__init__.py` - Exported the connector - `rag/svr/sync_data_source.py` - Added MySQL and PostgreSQL sync classes - `pyproject.toml` - Added mysql-connector-python dependency Frontend: - `web/src/pages/user-setting/data-source/constant/index.tsx` - Form fields - `web/src/locales/en.ts` - English translations - `web/src/assets/svg/data-source/mysql.svg` - MySQL icon - `web/src/assets/svg/data-source/postgresql.svg` - PostgreSQL icon ### Testing done Tested with MySQL 8.0 and PostgreSQL 16: - Connection validation works correctly - Full sync imports all query results as documents - Incremental sync only fetches rows updated since last sync - Custom SQL queries filter data as expected - Invalid credentials show clear error messages - Lint checks pass (`ruff check` returns no errors) --------- Co-authored-by: mkdev11 <YOUR_GITHUB_ID+MkDev11@users.noreply.github.com>
This commit is contained in:
MkDev11
3
web/src/assets/svg/data-source/mysql.svg
Normal file
3
web/src/assets/svg/data-source/mysql.svg
Normal file
@ -0,0 +1,3 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128">
|
||||
<path fill="#00618A" d="M116.948 97.807c-6.863-.187-12.104.452-16.585 2.341-1.273.537-3.305.552-3.513 2.147.7.733.807 1.83 1.365 2.731 1.07 1.73 2.876 4.052 4.488 5.268 1.762 1.33 3.577 2.751 5.465 3.902 3.358 2.047 7.107 3.217 10.34 5.268 1.906 1.21 3.799 2.733 5.658 4.097.92.675 1.537 1.724 2.732 2.147v-.194c-.628-.79-.79-1.86-1.365-2.731l-2.537-2.537c-2.48-3.292-5.629-6.184-8.976-8.585-2.669-1.916-8.642-4.504-9.755-7.609l-.195-.195c1.892-.214 4.107-.898 5.854-1.367 2.934-.786 5.556-.583 8.585-1.365l4.097-1.171v-.78c-1.531-1.571-2.623-3.651-4.292-5.072-4.37-3.72-9.138-7.437-14.048-10.537-2.724-1.718-6.089-2.835-8.976-4.292-.971-.491-2.677-.746-3.318-1.562-1.517-1.932-2.342-4.382-3.511-6.633-2.449-4.717-4.854-9.868-7.024-14.831-1.48-3.384-2.447-6.72-4.292-9.756-8.85-14.567-18.396-23.358-33.169-32-3.144-1.838-6.929-2.563-10.929-3.513l-6.439-.391c-1.307-.547-2.666-2.149-3.902-2.927C17.811 4.565 5.257-2.16 1.633 6.682c-2.289 5.581 3.421 11.025 5.462 13.854 1.434 1.982 3.269 4.207 4.293 6.438.674 1.467.79 2.938 1.367 4.489 1.417 3.822 2.652 7.98 4.487 11.511.927 1.788 1.949 3.67 3.122 5.268.718.981 1.951 1.413 2.145 2.927-1.204 1.686-1.273 4.304-1.95 6.44-3.05 9.615-1.899 21.567 2.537 28.683 1.36 2.186 4.567 6.871 8.975 5.073 3.856-1.57 2.995-6.438 4.098-10.732.249-.973.096-1.689.585-2.341v.195l3.513 7.024c2.6 4.187 7.212 8.562 11.122 11.514 2.027 1.531 3.623 4.177 6.244 5.073v-.196h-.195c-.508-.791-1.303-1.119-1.951-1.755-1.527-1.497-3.225-3.358-4.487-5.073-3.556-4.827-6.698-10.11-9.561-15.609-1.368-2.627-2.557-5.523-3.709-8.196-.444-1.03-.438-2.589-1.364-3.122-1.263 1.958-3.122 3.542-4.098 5.854-1.561 3.696-1.762 8.204-2.341 12.878-.342.122-.19.038-.391.194-2.718-.655-3.672-3.452-4.683-5.853-2.554-6.07-3.029-15.842-.781-22.829.582-1.809 3.21-7.501 2.146-9.172-.508-1.666-2.184-2.63-3.121-3.903-1.161-1.574-2.319-3.646-3.124-5.464-2.09-4.731-3.066-10.044-5.267-14.828-1.053-2.287-2.832-4.602-4.293-6.634-1.617-2.253-3.429-3.912-4.683-6.635-.446-.968-1.051-2.518-.391-3.513.21-.671.507-.951 1.171-1.17 1.132-.873 4.284.29 5.462.779 3.129 1.3 5.741 2.538 8.392 4.294 1.271.84 2.557 2.471 4.097 2.926h1.756c2.747.631 5.824.195 8.391.975 4.536 1.378 8.601 3.523 12.292 5.854 11.246 7.102 20.442 17.21 26.732 29.269 1.012 1.942 1.45 3.794 2.341 5.854 1.798 4.153 4.063 8.426 5.852 12.488 1.786 4.052 3.526 8.141 6.05 11.513 1.327 1.772 6.451 2.723 8.781 3.708 1.632.689 4.307 1.409 5.854 2.34 2.953 1.782 5.815 3.903 8.586 5.855 1.383.975 5.64 3.116 5.852 4.879zM29.729 23.466c-1.431-.027-2.443.156-3.513.389v.195h.195c.683 1.402 1.888 2.306 2.731 3.513.65 1.367 1.301 2.732 1.952 4.097l.194-.193c1.209-.853 1.762-2.214 1.755-4.294-.484-.509-.555-1.147-.975-1.755-.556-.811-1.635-1.272-2.339-1.952z"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.7 KiB |
6
web/src/assets/svg/data-source/postgresql.svg
Normal file
6
web/src/assets/svg/data-source/postgresql.svg
Normal file
@ -0,0 +1,6 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128">
|
||||
<path d="M93.809 92.112c.785-6.533.55-7.492 5.416-6.433l1.235.108c3.742.17 8.637-.602 11.513-1.938 6.191-2.873 9.861-7.668 3.758-6.409-13.924 2.873-14.881-1.842-14.881-1.842 14.703-21.815 20.849-49.508 15.543-56.287-14.47-18.489-39.517-9.746-39.936-9.52l-.134.025c-2.751-.571-5.83-.912-9.289-.968-6.301-.104-11.082 1.652-14.709 4.402 0 0-44.683-18.409-42.604 23.151.442 8.841 12.672 66.898 27.26 49.362 5.332-6.412 10.484-11.834 10.484-11.834 2.558 1.699 5.622 2.567 8.834 2.255l.249-.212c-.078.796-.044 1.575.099 2.497-3.757 4.199-2.653 4.936-10.166 6.482-7.602 1.566-3.136 4.355-.221 5.084 3.535.884 11.712 2.136 17.238-5.598l-.22.882c1.474 1.18 1.375 8.477 1.583 13.69.208 5.214.558 10.079 1.621 12.948 1.063 2.868 2.317 10.256 12.191 8.14 8.252-1.764 14.561-4.309 15.136-27.985" fill="#336791"/>
|
||||
<path d="M66.509 129.502c-.169-2.431-.311-4.739-.311-5.868 0-14.703.837-18.013 3.047-19.678 2.212-1.665 6.505-2.332 7.327-10.148.468-4.448-2.016-6.448-2.016-6.448s3.798-16.395 3.798-31.193c0-14.798-7.326-23.263-7.326-23.263s15.756-7.326 18.924-7.326c3.169 0 7.326 2.363 7.326 10.956 0 8.592-3.169 30.847-3.169 30.847s2.363 1.712 2.363 4.879c0 3.167-3.169 25.968-3.169 25.968s-7.326 3.169-7.326 10.495c0 3.975.837 7.326.837 7.326M45.607 104.059c-.418-3.921-.418-7.842-.418-7.842 0-7.326 3.169-18.088 3.169-18.088s-3.169-1.665-3.169-8.99c0-7.326 3.169-23.264 3.169-23.264s-7.327 0-10.956-3.629c-3.629-3.629-7.326-11.362-7.326-16.241 0-4.879 3.697-7.326 7.326-7.326 3.629 0 25.968 10.956 25.968 10.956" fill="none" stroke="#fff" stroke-width="4.8" stroke-linecap="round" stroke-linejoin="round"/>
|
||||
<path d="M93.809 92.112c.785-6.533.55-7.492 5.416-6.433l1.235.108c3.742.17 8.637-.602 11.513-1.938 6.191-2.873 9.861-7.668 3.758-6.409-13.924 2.873-14.881-1.842-14.881-1.842 14.703-21.815 20.849-49.508 15.543-56.287-14.47-18.489-39.517-9.746-39.936-9.52l-.134.025c-2.751-.571-5.83-.912-9.289-.968-6.301-.104-11.082 1.652-14.709 4.402 0 0-44.683-18.409-42.604 23.151.442 8.841 12.672 66.898 27.26 49.362 5.332-6.412 10.484-11.834 10.484-11.834 2.558 1.699 5.622 2.567 8.834 2.255l.249-.212c-.078.796-.044 1.575.099 2.497-3.757 4.199-2.653 4.936-10.166 6.482-7.602 1.566-3.136 4.355-.221 5.084 3.535.884 11.712 2.136 17.238-5.598l-.22.882c1.474 1.18 1.375 8.477 1.583 13.69.208 5.214.558 10.079 1.621 12.948 1.063 2.868 2.317 10.256 12.191 8.14 8.252-1.764 14.561-4.309 15.136-27.985" fill="none" stroke="#fff" stroke-width="2.4" stroke-linecap="round" stroke-linejoin="round"/>
|
||||
<path d="M60.087 89.226c-.157 7.853.256 15.735 1.231 19.469 1.503 5.751 4.696 9.168 13.295 7.36 6.739-1.418 9.246-4.622 10.76-9.553.853-2.781 1.435-16.308 1.595-22.48" fill="none" stroke="#fff" stroke-width="2.4" stroke-linecap="round" stroke-linejoin="round"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.7 KiB |
@ -36,6 +36,8 @@ export enum DataSourceKey {
|
||||
BITBUCKET = 'bitbucket',
|
||||
ZENDESK = 'zendesk',
|
||||
SEAFILE = 'seafile',
|
||||
MYSQL = 'mysql',
|
||||
POSTGRESQL = 'postgresql',
|
||||
// SHAREPOINT = 'sharepoint',
|
||||
// SLACK = 'slack',
|
||||
// TEAMS = 'teams',
|
||||
@ -161,6 +163,16 @@ export const generateDataSourceInfo = (t: TFunction) => {
|
||||
description: t(`setting.${DataSourceKey.SEAFILE}Description`),
|
||||
icon: <SvgIcon name={'data-source/seafile'} width={38} />,
|
||||
},
|
||||
[DataSourceKey.MYSQL]: {
|
||||
name: 'MySQL',
|
||||
description: t(`setting.${DataSourceKey.MYSQL}Description`),
|
||||
icon: <SvgIcon name={'data-source/mysql'} width={38} />,
|
||||
},
|
||||
[DataSourceKey.POSTGRESQL]: {
|
||||
name: 'PostgreSQL',
|
||||
description: t(`setting.${DataSourceKey.POSTGRESQL}Description`),
|
||||
icon: <SvgIcon name={'data-source/postgresql'} width={38} />,
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
@ -854,6 +866,106 @@ export const DataSourceFormFields = {
|
||||
tooltip: t('setting.seafileBatchSizeTip'),
|
||||
},
|
||||
],
|
||||
[DataSourceKey.MYSQL]: [
|
||||
{
|
||||
label: 'Host',
|
||||
name: 'config.host',
|
||||
type: FormFieldType.Text,
|
||||
required: true,
|
||||
placeholder: 'localhost',
|
||||
},
|
||||
{
|
||||
label: 'Port',
|
||||
name: 'config.port',
|
||||
type: FormFieldType.Number,
|
||||
required: true,
|
||||
placeholder: '3306',
|
||||
},
|
||||
{
|
||||
label: 'Database',
|
||||
name: 'config.database',
|
||||
type: FormFieldType.Text,
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
label: 'Username',
|
||||
name: 'config.credentials.username',
|
||||
type: FormFieldType.Text,
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
label: 'Password',
|
||||
name: 'config.credentials.password',
|
||||
type: FormFieldType.Password,
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
label: 'SQL Query',
|
||||
name: 'config.query',
|
||||
type: FormFieldType.Textarea,
|
||||
required: false,
|
||||
placeholder: 'Leave empty to load all tables',
|
||||
tooltip: t('setting.mysqlQueryTip'),
|
||||
},
|
||||
{
|
||||
label: 'Content Columns',
|
||||
name: 'config.content_columns',
|
||||
type: FormFieldType.Text,
|
||||
required: false,
|
||||
placeholder: 'title,description,content',
|
||||
tooltip: t('setting.mysqlContentColumnsTip'),
|
||||
},
|
||||
],
|
||||
[DataSourceKey.POSTGRESQL]: [
|
||||
{
|
||||
label: 'Host',
|
||||
name: 'config.host',
|
||||
type: FormFieldType.Text,
|
||||
required: true,
|
||||
placeholder: 'localhost',
|
||||
},
|
||||
{
|
||||
label: 'Port',
|
||||
name: 'config.port',
|
||||
type: FormFieldType.Number,
|
||||
required: true,
|
||||
placeholder: '5432',
|
||||
},
|
||||
{
|
||||
label: 'Database',
|
||||
name: 'config.database',
|
||||
type: FormFieldType.Text,
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
label: 'Username',
|
||||
name: 'config.credentials.username',
|
||||
type: FormFieldType.Text,
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
label: 'Password',
|
||||
name: 'config.credentials.password',
|
||||
type: FormFieldType.Password,
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
label: 'SQL Query',
|
||||
name: 'config.query',
|
||||
type: FormFieldType.Textarea,
|
||||
required: false,
|
||||
placeholder: 'Leave empty to load all tables',
|
||||
tooltip: t('setting.postgresqlQueryTip'),
|
||||
},
|
||||
{
|
||||
label: 'Content Columns',
|
||||
name: 'config.content_columns',
|
||||
type: FormFieldType.Text,
|
||||
required: false,
|
||||
placeholder: 'title,description,content',
|
||||
tooltip: t('setting.postgresqlContentColumnsTip'),
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
export const DataSourceFormDefaultValues = {
|
||||
@ -1135,7 +1247,6 @@ export const DataSourceFormDefaultValues = {
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
[DataSourceKey.SEAFILE]: {
|
||||
name: '',
|
||||
source: DataSourceKey.SEAFILE,
|
||||
@ -1148,4 +1259,40 @@ export const DataSourceFormDefaultValues = {
|
||||
},
|
||||
},
|
||||
},
|
||||
[DataSourceKey.MYSQL]: {
|
||||
name: '',
|
||||
source: DataSourceKey.MYSQL,
|
||||
config: {
|
||||
host: 'localhost',
|
||||
port: 3306,
|
||||
database: '',
|
||||
query: '',
|
||||
content_columns: '',
|
||||
metadata_columns: '',
|
||||
id_column: '',
|
||||
timestamp_column: '',
|
||||
credentials: {
|
||||
username: '',
|
||||
password: '',
|
||||
},
|
||||
},
|
||||
},
|
||||
[DataSourceKey.POSTGRESQL]: {
|
||||
name: '',
|
||||
source: DataSourceKey.POSTGRESQL,
|
||||
config: {
|
||||
host: 'localhost',
|
||||
port: 5432,
|
||||
database: '',
|
||||
query: '',
|
||||
content_columns: '',
|
||||
metadata_columns: '',
|
||||
id_column: '',
|
||||
timestamp_column: '',
|
||||
credentials: {
|
||||
username: '',
|
||||
password: '',
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user