Checks
dropoutt 1.4.1 includes 36 checks. Identifiers are never renumbered. Run dropoutt checks to list them or pass a check id for a detailed description.
Tier 0: structural
Tier 0 checks identify exact structural and formatting defects.
| ID | Description |
|---|---|
| T0-SCHEMA-001 | Files are not training data: agent session logs and telemetry that look like chat |
| T0-SCHEMA-002 | One folder mixes several record layouts |
| T0-SCHEMA-003 | Records failed to parse |
| T0-SCHEMA-004 | Message content was not a string |
| T0-SCHEMA-005 | Content sits in keys the layout never reads |
| T0-FORMAT-001 | Plain-text files are holding structured records |
| T0-GEN-001 | Generator scaffolding outside the records |
| T0-REASON-001 | Only some responses carry a reasoning trace |
| T0-TRUNC-002 | Responses stop at a generation length cap |
| T0-QUAL-001 | Documents whose lines mostly lack punctuation |
| T0-QUAL-002 | Documents built mostly from very short lines |
| T0-QUAL-003 | Documents repeating their own lines |
| T0-ROLE-001 | Conversation role structure is invalid |
| T0-ROLE-002 | Role names are not the canonical vocabulary |
| T0-TMPL-001 | Data is already formatted with a chat template |
| T0-TMPL-002 | Records fail to render with the target chat template |
| T0-MASK-001 | Records contribute zero trainable tokens |
| T0-MASK-002 | Stop token is outside the trainable span |
| T0-TRUNC-001 | Records exceed the sequence length |
| T0-PACK-001 | Packing efficiency under concat-and-chunk |
| T0-ENC-001 | Text encoding is damaged: mojibake from a file decoded as the wrong codec |
| T0-DUP-001 | Exact and whitespace-identical duplicates |
| T0-DEGEN-001 | Degenerate responses |
Tier 1: statistical
| ID | Description |
|---|---|
| T1-NDUP-001 | Near-duplicate records (MinHash; reports, never deletes) |
| T1-DUP-002 | Same prompt answered two different ways |
| T1-OVERLAP-001 | Datasets overlap with each other, directionally |
| T1-ATLAS-001 | Corpus sits in very few topical regions, dropoutt atlas only |
| T1-ATLAS-002 | A crowded region holds near-identical records, dropoutt atlas only |
| T1-CONTAM-001 | Training data overlaps evaluation benchmarks |
| T1-LANG-001 | Language composition and detection confidence |
| T1-LANG-002 | Records deviate from the dataset’s main language |
| T1-LANG-003 | Script does not match the detected language: Turkish that lost its diacritics reads as a different language |
| T1-PII-001 | Personal data and credentials in training text |
| T1-IDENT-001 | Assistant identity leakage and refusal boilerplate |
| T1-STYLE-001 | Formulaic response openings |
| T1-LIC-001 | Datasets have no recorded licence |
Behavior and scope
- Blocking requires an explicit
--target. - Checks report recommended actions but do not delete or rewrite data.
- PII matches are masked in reports.
- PDF text extraction is not supported. Scan extracted text instead.
- Nothing phones home. Network access is limited to fetching tokenizers and the atlas encoder, and
--offlinedisables it.
Every finding is labelled unverified: no calibration corpus yet links acting on a finding to a measured change in model quality. Structural checks identify deterministic defects such as empty loss masks and fully truncated responses; treat statistical findings as diagnostic signals and validate changes against the target model and evaluation suite.