Skip to content

Changelog

All notable changes to dataveil are listed here. The format follows Keep a Changelog, and the project uses Semantic Versioning (while on 0.x, a minor release can change behavior).

Unreleased

0.4.0 - 2026-10-20

Added

  • dataveil.toml configuration file, found through DATAVEIL_CONFIG or in the working directory. It sets the profile and classification thresholds, turns built-in classifiers off, and configures the MCP server's adapter and audit log. Privacy thresholds can only be made stricter than the defaults, and unknown keys are rejected.
  • Custom regex classifiers in dataveil.toml ([[classify.custom]], with a tag, an anchored pattern and optional name hints), for identifiers such as internal customer numbers. They run as aggregate counts like the built-in classifiers, on every adapter. Patterns that are invalid, unanchored, or would match differently on DuckDB and Postgres are rejected at load time.
  • Adapter and classifier plugins through the dataveil.adapters and dataveil.classifiers entry point groups. The built-in adapters and classifiers are registered the same way. A classifier plugin provides definitions, not code, so it can only produce aggregate counts. Plugin adapters get their settings from [adapter.options].
  • benchmarks/run.py times profiling and classification on synthetic tables of 1M and 10M rows (DuckDB, and Postgres) and counts the queries per column. A non-blocking CI job runs it weekly and on changes to the core.
  • Documentation site (MkDocs Material) with getting started, configuration reference, one page per adapter, plugins, privacy and known limits, operations, and benchmark results. The README keeps the overview.
  • profile_table, classify_table and classify_column take an optional config argument. Without it they behave as before.

Changed

  • Classification reads each string column once: the non-null count and every regex classifier's match count come from one aggregate query, instead of one query per classifier. Results are unchanged.
  • The Luhn and IBAN checksum functions only run on values that match the classifier's pattern. On DuckDB they ran on every value before, which was most of the classification time.
  • The Postgres checksum functions are declared PARALLEL SAFE, so queries that use them can run on several Postgres workers. They were serial before, and the one-query-per-column change made that cost visible.
  • The person-name model no longer runs on a column that a regex classifier has already tagged, such as a column of email addresses.
  • On a 10M-row table, classification now takes 19 s instead of 75 s on DuckDB, 35 s instead of 135 s with the person-name model, and 54 s instead of 115 s on Postgres. See the benchmarks page.

0.3.0 - 2026-10-09

Security

  • min and max are only returned when at least 5 rows share that exact value (MIN_EXTREME_COUNT). Before, a large table returned them even when they were a single row's value, such as the largest salary.
  • A format signature is only returned when at least 5 non-null values share it (MIN_SIGNATURE_COUNT). Rarer signatures, and any beyond the first 20, are reported together as <other signatures> with their total count. Before, a signature with count 1 described a single row.

Added

  • PII:PERSON_NAME classifier for free-text person names, using a local spaCy model (optional ner extra and en_core_web_sm). Runs on DuckDB and on SQLMesh or dbt projects on DuckDB; skipped elsewhere.
  • DbtAdapter for dbt projects: lists the models from dbt's target/manifest.json and runs SQL through a DuckDB or Postgres adapter. No dbt dependency.
  • The SQLMesh adapter registers the Luhn and IBAN checksum functions when the project runs on DuckDB or Postgres, so the credit-card and IBAN classifiers work there. Other engines skip them as before. Pass register_checksum_functions=False to turn this off.
  • py.typed marker, so type checkers use dataveil's type hints.
  • Contributor guide (CONTRIBUTING.md), security policy (SECURITY.md), pull request template and issue forms.
  • make check (lint, format check, mypy, tests), pre-commit hooks, and CI on Python 3.13.

Fixed

  • Creating a second DuckDBAdapter on the same connection no longer fails with "function already created".

Changed

  • The README and the audit log docs state the privacy guarantee precisely: no raw rows or samples are returned, and some aggregates (for example min/max on a large table) can still reveal a value. Before, the README opened with "never raw values", which the Known limits section contradicted.
  • The package passes mypy --strict. Public functions return precise types (for example dict[str, Any] instead of a bare dict).

0.2.0 - 2026-10-03

Fixed

  • Format signatures folded digits to a, so an SSN read aaa-aa-aaaa instead of ddd-dd-dddd. Letters now fold to a and digits to d.

Security

  • Numeric stats (min, max, mean, stddev, percentiles) are withheld when the table has fewer than 10 rows or the column fewer than 3 distinct values. In 0.1.0 they could reveal the value of a single row.
  • Characters outside printable ASCII fold to a in format signatures, and a value made only of punctuation or whitespace is reported as <punctuation or whitespace only>.

Changed

  • The README has a "Known limits" section that states what aggregate-only does and does not guarantee.

0.1.0 - 2026-10-03

Added

  • Core engine: aggregate-only profiling, pattern- and checksum-based PII classification, and plan validation and execution against a closed, versioned operation vocabulary.
  • Adapters for DuckDB (reference), SQLMesh and Postgres.
  • Audit logging without values, and an optional plan-approval gate.
  • Standalone MCP server (dataveil-mcp).