RepositoryCurrent collected state Current
Observed since 20 Oct, 19:10 UTC
No superseded metadata states collected. Unchanged observations do not create new revisions.
Back to packageRELEASE DETAILS
0.0.1
PyPIa-data-processing
A library that prepares raw documents for downstream ML tasks.
Published2 Feb 2024
Direct dependencies27
Transitive dependenciesNot calculated
README
Version 0.0.1 documentation# Data Processing ## Current Version Main Features Data Processing is used for data processing through MinIO, databases, Web APIs, etc. The data types handled include: - txt - json - doc - html - excel - csv - pdf - markdown - ppt ### Current Text Type Processing The data processing process includes: cleaning abnormal data, filtering, de-duplication, and anonymization. ## Design  ## Local Development ### Software Requirements Before setting up the local data-process environment, please make sure the following software is installed: - Python 3.10.x ### Environment Setup Install the Python dependencies in the requirements.txt file ### Running Run the server.py file in the src directory # isort isort is a tool for sorting imports alphabetically within your Python code. It helps maintain a consistent and clean import order. ## install ```shell pip install isort ``` ## isort a file ```shell isort src/server.py ``` ## isort a directory ```shell isort . ```
Downloads & popularity
Awaiting observationsNo metric observations yet
Download counts and popularity will appear here as the collector receives them.
Declared dependencies 27
Native requirements27 direct targetsNot calculated transitive count
| Package / target | Requirement | Relationship |
|---|---|---|
pandasEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | pandas ==2.1.2 | runtime |
numpyEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | numpy ==1.26.1 | runtime |
sanicEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | sanic ==23.6.0 | runtime |
sanic-corsEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | sanic-cors ==2.2.0 | runtime |
aiohttpEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | aiohttp ==3.8.6 | runtime |
ulidEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | ulid ==1.1 | runtime |
minioEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | minio ==7.1.17 | runtime |
zhipuaiEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | zhipuai ==1.0.7 | runtime |
langchainEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | langchain ==0.0.354 | runtime |
spacyEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | spacy ==3.5.4 | runtime |
pypdfEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | pypdf ==3.17.1 | runtime |
emojiEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | emoji ==2.2.0 | runtime |
ftfyEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | ftfy ==6.1.1 | runtime |
psycopg2-binaryEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | psycopg2-binary ==2.9.9 | runtime |
kubernetesEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | kubernetes ==25.3.0 | runtime |
duckdbEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | duckdb ==0.9.2 | runtime |
DBUtilsEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | DBUtils ==3.0.3 | runtime |
pyyamlEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | pyyaml ==6.0.1 | runtime |
openccEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | opencc ==0.2 | runtime |
opencc-python-reimplementedEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | opencc-python-reimplemented ==0.1.7 | runtime |
selectolaxEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | selectolax ==0.3.17 | runtime |
openaiEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | openai ==1.3.7 | runtime |
python-docxEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | python-docx ==1.1.0 | runtime |
bs4Environment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | bs4 ==0.0.1 | runtime |
playwrightEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | playwright ==1.40.0 | runtime |
pillowEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | pillow ==10.2.0 | runtime |
html2textEnvironment{
"extras": "",
"marker": null,
"requires_python": ">=3.9.0,<3.12",
"evaluation": "NOT_EVALUATED"
} | html2text ==2020.1.16 | runtime |
Distribution files
2 collecteda_data_processing-0.0.1-py3-none-any.whl
Source filebdist_wheel · 2.04 KB · Python py3
a-data-processing-0.0.1.tar.gz
Source filesdist · 2.73 KB · Python source
Repository
PENDINGMetadata history
Meaningful changesRecord identity
package_version:363cb8b1492275829ee11b0e9639e28c4c19268f44e1f0688034f4d3e6808013