Parts
Every part is checked against a reference implementation and benchmarked across input sizes. Pick one, copy it, keep the proof.
AI
Data processing around LLMs: chunking, normalization, dedup, token counting.
File Hash
Streaming BLAKE3 and SHA-256 hashing for file integrity checks and exact-duplicate detection in datasets.
- Verified against
- hashlib
- Up to
- 5.7× faster
MinHash
Near-duplicate detection with MinHash signatures, bit-for-bit compatible with datasketch. Estimates Jaccard similarity between token sets in constant space, for deduplicating RAG corpora and training data.
- Verified against
- datasketch
- Up to
- 19× faster
Text Chunker
Recursive text splitter for RAG. Splits documents on paragraph, sentence and word boundaries into chunks of a target size, with overlap.
- Verified against
- langchain-text-splitters
- Up to
- 26× faster
Token Counter
Counts BPE tokens for OpenAI-compatible encodings (cl100k_base, o200k_base) without building the token list.
- Verified against
- tiktoken
- Up to
- 1.7× faster
Unicode Normalize
Unicode cleanup for text pipelines: NFC/NFKC normalization, whitespace folding and removal of invisible characters.
- Verified against
- unicodedata
- Up to
- 2.2× faster
Security
Log parsing, pattern matching, hashing, binary analysis.
Byte Entropy
Shannon entropy over sliding windows of a binary, for spotting packed or encrypted sections.
- Verified against
- scipy.stats.entropy
- Up to
- 35× faster
File Hash
Streaming BLAKE3 and SHA-256 hashing for file integrity checks and exact-duplicate detection in datasets.
- Verified against
- hashlib
- Up to
- 5.7× faster
Log Parse
Parses syslog (RFC 3164 / 5424) and common web server access logs into structured records.
- Verified against
- re
- Up to
- 20× faster
Multi-pattern Match
Aho–Corasick matching of thousands of patterns in one pass, for scanning logs and payloads for indicators of compromise.
- Verified against
- pyahocorasick
- Up to
- 5.0× faster