GitHub - mirth/chonky: Fully neural approach for text chunking
Chonky is a Python library that intelligently segments text into meaningful semantic chunks using a fine-tuned transformer model. This library can be used in the RAG systems.from chonky import TextSplitter
# on the first run it will download the transformer model
splitter = TextSplitter(device="cpu")
text = """Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and prob...
Read more at github.com