Hugging Face가 Apache Arrow의 Parquet Content-Defined Chunking을 Xet 스토리지 레이어와 통합해 변경된 데이터 청크만 업로드/다운로드하도록 구현
Parquet Content-Defined Chunking
Parquet Content-Defined Chunking
Xet is on the Hub
From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub
From Files to Chunks: Improving HF Storage Efficiency