Rodrigo Nader 70eaacfd36 feat: improve gitloader component (#5351)
* feat: improve gitloader component

* [autofix.ci] apply automated fixes

* feat(git.py): refactor GitLoaderComponent to support asynchronous operations and improve temporary directory management

- Convert methods to async to enhance performance with file operations.
- Implement async context manager for handling temporary clone directories.
- Update binary file check and content filtering to be asynchronous.

* fix(git.py): enhance GitLoaderComponent with improved file filtering and binary check

- Refactor file filtering logic to utilize fnmatch for pattern matching.
- Introduce a new method to check for binary files based on null byte detection.
- Update content filtering to handle exceptions more gracefully.
- Modify temporary directory cleanup to use rmdir instead of remove for better directory management.
- Adjust load_documents method to utilize asyncio.to_thread for lazy loading of documents.

* refactor(git.py): enhance GitLoaderComponent with improved file filtering and binary check

- Refactor binary file check to be synchronous and handle exceptions more gracefully.
- Introduce new methods for checking file patterns and content patterns, allowing for more flexible filtering.
- Consolidate filtering logic into a single method for better maintainability.
- Update load_documents method to run lazy loading in a separate thread for improved performance.

* feat(tests): add unit tests for GitLoaderComponent file filtering and binary detection

- Introduced new test suite for GitLoaderComponent, including tests for binary file detection and file pattern matching.
- Implemented temporary file creation for testing various file types and permissions.
- Added tests for combined filtering functionality, ensuring robust handling of file and content patterns.

* refactor(git.py): improve pattern handling and content filtering in GitLoaderComponent

- Refactored pattern handling to use a more descriptive variable name `pattern_list` for clarity.
- Enhanced content filtering by ensuring proper encoding when reading file content.
- Updated regex validation to include a test string check for better error handling.
- Removed unnecessary comments to streamline the code and improve readability.

* fix: exception TypeError

TypeError: object async_generator can't be used in 'await' expression

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org>
Co-authored-by: italojohnny <italojohnnydosanjos@gmail.com>
2025-01-02 16:39:16 +00:00
2024-12-23 14:53:37 +00:00
2024-12-12 19:01:16 +00:00

Langflow

Langflow is a low-code app builder for RAG and multi-agent AI applications. Its Python-based and agnostic to any model, API, or database.

Docs - Free Cloud Service - Self Managed

README in English README in Portuguese README in Spanish README in Simplified Chinese README in Japanese README in KOREAN README in French

Core features

  1. Python-based and agnostic to models, APIs, data sources, or databases.
  2. Visual IDE for drag-and-drop building and testing of workflows.
  3. Playground to immediately test and iterate workflows with step-by-step control.
  4. Multi-agent orchestration and conversation management and retrieval.
  5. Free cloud service to get started in minutes with no setup.
  6. Publish as an API or export as a Python application.
  7. Observability with LangSmith, LangFuse, or LangWatch integration.
  8. Enterprise-grade security and scalability with free DataStax Langflow cloud service.
  9. Customize workflows or create flows entirely just using Python.
  10. Ecosystem integrations as reusable components for any model, API or database.

Integrations

📦 Quickstart

  • Install with uv (recommended) (Python 3.10 to 3.12):
uv pip install langflow
  • Install with pip (Python 3.10 to 3.12):
pip install langflow

Getting Started

Stay up-to-date

Star Langflow on GitHub to be instantly notified of new releases.

Star Langflow

👋 Contribute

We welcome contributions from developers of all levels. If you'd like to contribute, please check our contributing guidelines and help make Langflow more accessible.


Star History Chart

❤️ Contributors

langflow contributors

Description
Langflow is a powerful tool for building and deploying AI-powered agents and workflows.
Readme MIT 2.3 GiB
Languages
Python 64.5%
TypeScript 23.4%
JavaScript 11.4%
CSS 0.3%
Makefile 0.2%
Other 0.1%