Get started

Install dropoutt 1.4.1 on Python 3.10 or newer, then run a scan against a file or directory.

Install

pip install dropoutt
uv tool install dropoutt        # or as a standalone tool

If dropoutt is not on your PATH, run the package through the active Python interpreter: python -m dropoutt.

First run

Run scan against a training-data file or directory.

dropoutt scan ./data

The scan writes five files to ./data/.dropoutt/: report.html, report.md, report.json, findings.jsonl, and fingerprint.json. On a desktop, the HTML report opens automatically.

Default checks

With no flags, a scan runs 34 of the 36 checks. The other two belong to the map and run under dropoutt atlas.

File inventory and detected dataset layouts.
Exact duplicates, near duplicates, and overlap between datasets.
Overlap with supported evaluation benchmarks.
Language composition across 97 languages, including script mismatches and damaged text encoding.
Personal data and credentials, with matched values masked in every report.
Estimated token counts under 5 common tokenizers.

Token and loss-mask checks require --model. Blocking findings require --target. The report lists any checks that could not run and the option required to enable them.

Map coverage

Run dropoutt atlas to place records on a fixed map of public training data. The command writes atlas.html, atlas.md, and atlas.json beside the scan reports.

dropoutt atlas ./data
dropoutt atlas ./data --sampling 500     # a quick look

The command places records on atlas-v3, the one map that ships, in a terminal and in CI alike.

The first run downloads the encoder, 142 MB once cached. Placement needs at least 40 characters of text per record.

Supported input

JSON, JSONL/NDJSON, TXT, Markdown, CSV/TSV, Parquet, Arrow, Feather, ORC, .mds (MosaicML Streaming) and .tar (WebDataset). Text formats may be compressed with gzip, bzip2, xz or zstd. A .tar.gz is handled by the archive reader itself.

Directories that never hold training data, such as node_modules, site-packages and .git, are not descended into, and toolchain files such as package.json and requirements.txt are passed over even though their extensions match.

Prepare for offline use

The atlas maps and benchmark indices ship inside the package. The atlas encoder and model tokenizers are downloaded on first use. Run dropoutt fetch to download them before moving to an offline environment.

dropoutt fetch
dropoutt fetch --model qwen3      # one tokenizer as well

NextCommand reference