refine text decode (#657)

### What problem does this PR solve?
#651 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
This commit is contained in:
KevinHuSh
2024-05-07 12:25:47 +08:00
committed by GitHub
parent de839fc3f0
commit 7013d7f620
7 changed files with 7 additions and 7 deletions

View File

@ -91,7 +91,7 @@ def chunk(filename, binary=None, from_page=0, to_page=100000,
txt = ""
if binary:
encoding = find_codec(binary)
txt = binary.decode(encoding)
txt = binary.decode(encoding, errors="ignore")
else:
with open(filename, "r") as f:
while True: