File Splitter
📖 技能介绍
name: file-splitter description: > Split large files into smaller chunks with semantic boundary detection. Supports JSON, Markdown, and TXT formats. Preserves data integrity by splitting at natural boundaries (JSON array elements, MD headings, TXT paragraphs). Use when: user needs to split large files, chunk datasets, segment corpora, or break down files into manageable pieces for processing or analysis. Triggers: split file, chunk, segment, file splitter, JSON split, MD split, TXT split, corpus segmentation, data chunking.
File Splitter - Universal File Splitting Tool
Split large files into smaller, manageable chunks while preserving semantic structure.
Quick Start
python <skill_dir>/scripts/split_files.py --input <input_folder> --output <output_folder> [options]
Parameters
| Parameter | Required | Default | Description |
|---|---|---|---|
--input |
Yes | - | Source folder containing files to split |
--output |
Yes | - | Output folder for split chunks |
--max-size |
No | 512000 (500KB) | Maximum bytes per chunk |
--min-size |
No | 409600 (400KB) | Minimum bytes per chunk |
--seq-digits |
No | 9 | Number of digits in sequence numbers |
--formats |
No | json,md,txt | File formats to process (comma-separated) |
--dry-run |
No | false | Preview mode - show what would be split without executing |
Examples
# Default 500KB split
python split_files.py --input "./corpus" --output "./corpus/chunks"
# Custom 200KB chunks
python split_files.py --input "./notes" --output "./notes/chunks" --max-size 204800 --min-size 153600
# JSON files only
python split_files.py --input "./data" --output "./data/out" --formats json
# Preview mode
python split_files.py --input "./data" --output "./data/out" --dry-run
Splitting Rules
JSON Files
- Splits at JSON array element boundaries
- Each chunk is a valid JSON array
[...] - Automatically extracts list values if top-level is an object
- Never cuts individual records in half
Markdown Files
- Splits at heading boundaries (
#through######) - Each chunk maintains complete heading structure
- Never cuts content within a heading section
TXT Files
- Prefers splitting at paragraph boundaries (empty lines)
- Falls back to line-by-line splitting if no paragraphs exist
- Never cuts within a paragraph
Output Naming Convention
Format: {source_filename_without_extension}{9-digit_sequence_number}{extension}
推荐访问7w4.net获取更多AI技能。
Examples:
- dataset000000001.json
- dataset000000002.json
- notes000000001.md
Safety Features
- Source File Preservation: Read-only access to source files; never deletes or modifies originals
- Duplicate Detection: Automatically skips files that already have N-digit sequence suffixes to avoid re-splitting
- Small File Skip: Files ≤ max-size are automatically skipped (no need to split)
- Sequential Processing: Processes files one at a time to ensure stability
- Data Validation: Compares total size/record count before and after splitting; reports verification results
- UTF-8 Encoding: Forces UTF-8 for all read/write operations to avoid encoding issues on Windows
Notes
- Console may display garbled Chinese characters on Windows, but functionality is unaffected
- If a single data block/paragraph exceeds max-size, it becomes its own chunk (integrity takes priority over size limits)
- Output folder is automatically created if it doesn't exist
- License: MIT-0
🤖 AI 评测
这个文件拆分工具整体质量不错，功能实用且文档清晰。优点是拆分逻辑合理、支持多种格式、预览模式和安全保障做得较好，能有效避免误操作和数据丢失。不足之处是功能相对单一，仅支持三种基础格式；部分边界情况的错误提示还可以更友好；没有示例或测试文件，新手上手可能需要多花时间理解。总体而言，是一个可靠可用的工具，适合有明确拆分需求的用户使用。
