Skip to content

Changelog

All notable changes to this project are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning under the policy of ADR 0021 (amendment 11 from 1.0.0): a patch only fixes, a minor adds and deprecates, and only a major breaks the Python API, each break listed under "Breaking" with what to change. Before 1.0.0 a minor release could break it. The state a stream leaves behind (offsets, checkpoint layout, table schemas) never breaks without a migration path, and within 1.x it only grows. Every release says what it does to that state in its "State compatibility" line.

Unreleased

1.0.0rc1 - 2026-10-15

State compatibility: offsets and checkpoints are unchanged. The facts table gets migration 15, which rewrites the detail comment only (no mssql_cdc.min_version: older releases keep writing it). A 'capture_instance_switched' row's detail is JSON from this release; rows written before keep their text (ADR 0021 amendment 12). Other payloads are unchanged. A micro-batch row's warnings may hold one more message (on Spark without admission control).

Breaking

  • Typing: the results that Spark Connect returns as its own class are typed as the union (ADR 0031 amendment 2). to_delta returns StreamingQueryLike (was StreamingQuery), start_many a dict[str, StreamingQueryLike], and spark.get_spark a SparkSessionLike (was SparkSession). A type checker now flags code that annotated them with the classic class or used a member only one of the two has. At run time nothing changed. To migrate, annotate with the *Like types, or narrow (isinstance) or cast where you need the classic class. The new DataFrameLike types ReconcileResult.report, seed()'s df and the function delta_sink returns; it is the same type as pyspark.sql.DataFrame for a type checker (the Connect DataFrame subclasses it), so nothing to change there.
  • HAS_ADMISSION_CONTROL, OPERATIONS, MssqlCdcDataSource and make_client left mssql_cdc.__all__ (see "Deprecated"). A type checker now rejects importing them from mssql_cdc, and from mssql_cdc import * no longer binds them, without a warning. Import them from their modules instead: the first three from mssql_cdc.source, make_client from mssql_cdc.client, internal names that may change in any release.
  • CdcClient left mssql_cdc.__all__ and the API reference: it is internal, and a minor release may add methods to it (ADR 0030 amendment 1; see "Deprecated"). A type checker now rejects importing it from mssql_cdc, and from mssql_cdc import * no longer binds it, without a warning. Import it from mssql_cdc.client. Backend and Lsn stay public.

Added

  • Build provenance: each release's sdist and wheel get a GitHub artifact attestation (SLSA build provenance, actions/attest) from the job that publishes them, before the upload; the TestPyPI dry run attests too. PyPI's PEP 740 attestations, uploaded with every file since 0.1.0, are now documented as well. SECURITY.md, "Verifying a release", shows how to check both (gh attestation verify ... --signer-workflow .../release.yml --source-ref refs/tags/vX.Y.Z, pypi-attestations verify pypi).
  • CaptureInstanceSwitchedDetail, exported from mssql_cdc with the other payload types: the JSON of a 'capture_instance_switched' row's detail.

Changed

  • ADR 0021 amendment 11 sets the 1.x policy. From 1.0.0 on, versioning follows Semantic Versioning strictly: a patch fixes, a minor adds, a major breaks, and a type-only break counts. Anything public is deprecated in a 1.x minor before 2.0 removes it. The state contract only grows. It now names the keys of the generation state file and the Delta txnAppIds that make replays idempotent. Dropping support for a Python, Spark, Delta or SQL Server version that has reached its end of life is a minor release, announced here at least one release before; 1.0 keeps Python 3.10. The warnings, a task's reason and the text of a 'capture_instance_switched' row's detail are text for people and may be reworded in any release.
  • A 'capture_instance_switched' facts row's detail is JSON, {"old", "new", "start_lsn", "not_captured", "text"}: the capture instances, the newer one's start LSN (the row's min_lsn), the columns the query reads that the newer one does not capture, and text, the line the row held before (old -> new, then those columns). A query no longer parses text for the instance names. Rows written before keep the line: read both with coalesce(get_json_object(detail, '$.text'), detail), or take the names from the JSON only on rows whose detail starts with { (ADR 0023 amendment 8).
  • The files in metricsPath are internal: the options reference no longer documents their names or contents (a chunk's chunk-<n>.json was), which may change in any release. The facts table is the public record of what they carry (ADR 0021 amendment 11).
  • FinalizationListener is public for join, last_error and failures. Create it with finalization.track(); its constructor is not public, because a listener built by hand never started its worker.
  • The documentation site is published per version, with a version selector: dev follows main, each release tag publishes its own version, and stable, the site's root, is the latest final release. A release candidate is published under its version without moving stable, and leaves the selector once its final release is published. Links from before the versions open the same page in stable (dev until 1.0.0) (ADR 0024 amendments 1 and 2).

Deprecated

  • Importing HAS_ADMISSION_CONTROL, OPERATIONS, MssqlCdcDataSource or make_client from mssql_cdc. The API reference never listed them, so they were already internal (ADR 0021). They are no longer in mssql_cdc.__all__. Importing one from mssql_cdc warns at run time (DeprecationWarning), and 2.0 removes them from there. mssql_cdc.source and mssql_cdc.client keep them as internal names.
  • Importing CdcClient from mssql_cdc: it warns at run time (DeprecationWarning), and 2.0 removes it from there. mssql_cdc.client keeps it, internal: the data source builds its own client, so none of one's own plugs in, and a minor release may add methods to it.
  • reconcile(seed=...): renamed sample_seed, which says what it seeds (the sample of MATCH buckets compared row by row). seed keeps working until 2.0 and warns (DeprecationWarning); passing both raises TypeError.

Fixed

  • On a Spark without admission control, maxCommitsPerBatch is still ignored, but now with a warning in the driver's log and in the next micro-batch row's warnings (with metricsPath). Before, it was ignored silently.
  • Documentation: finalization.finalized_until has a docstring, and get_spark says a local session gets the UTC time zone and 4 shuffle partitions. Public docstrings and docs/reference/options.md no longer point at internal names.

0.8.0 - 2026-10-14

State compatibility: offsets and checkpoints unchanged. Facts migration 14 and snapshot plans migration 1 rewrite column comments only (metadata), the next time a stream opens those tables. The 'snapshot_open' payload gains the optional key switch_batch_id (ADR 0021 amendment 10); no other payload key is new. No mssql_cdc.min_version is set.

  • A chunked snapshot opened with snapshotKeys records those columns as its open's keys, the columns its chunks are cut on, as before. A backfill() before 0.8.0 refuses such a snapshot, as it refuses one whose unique index changed.
  • A release before 0.8.0 cannot complete a chunked snapshot that snapshot_on_switch opened once its generation's own snapshot has completed, because Delta skips its completion row. Run backfill() with 0.8.0.
  • finalization.freshness() only reads the control table.

Added

  • Chunked snapshots of tables without a unique index. The snapshotKeys option already cut the snapshot reader's ranges; now it also names the columns to_delta(..., snapshot="chunked") cuts a snapshot's chunks on, unique or not (ADR 0028, Amendment 6). Each chunk is a [lo, hi) range of the column values, NULL first, so a row is in one chunk only and a value is never split: a value held by more rows than chunk_rows makes a bigger chunk. The columns are recorded with the snapshot in its plans table, and its plan stays on them. The open refuses a column CDC does not capture, of a type no bound can be bound as, or one SQL Server cannot sort (bit, text, ntext, image, xml). Such a snapshot fills bronze only: apply_changes still needs keys that identify a row.
  • to_delta(snapshot="chunked", snapshot_on_switch=True), which raised ValueError before. The batch that first reads a newer capture instance now opens a chunked snapshot at that instance's start LSN, which the stream has already read past, instead of reading the whole table inside the batch. backfill() reads it next to the stream, and apply_changes applies its waves and rebuilds silver at its bootstrap row (ADR 0023 Amendment 7). It replaces a chunked snapshot of the stream that is still open, and it holds the snapshot mode like the others: a snapshot="full" run raises until it completes. Its 'snapshot_open' row has kind 'bootstrap' and the new key switch_batch_id, the batch_id of the switch's 'capture_instance_switched' row; its open and completion rows use Delta keys based on its S (<app_id>#snapopen.<S>, <app_id>#snapdone.<S>, version 0), because the generation's own keys may already be taken.
  • finalization.freshness(spark, control_table, table_name) returns how old a table's verdict is: finalized_until, updated_at, their ages in seconds by the caller's clock, and open_snapshot_lsn when a chunked snapshot holds a silver verdict. The result is Freshness, a TypedDict exported from mssql_cdc. The monitoring guide's alert query joins the control table with the facts and names why each stale verdict is stale: held, silver, no facts, stopped, not advanced, behind.
  • Lab check t11_fanout_cost measures what each start_many stream costs the driver: one Python planning process of about 50 MB per running query (and on a Spark Connect server, a foreachBatch process of about 100 MB), about 10 MB of heap and 20 JVM threads, tens of milliseconds of planning per trigger. ADR 0027 (amendment) and the many-tables guide give about 10 always-on streams per driver as a starting point, and the options past it: a longer trigger interval, availableNow groups, more jobs.

Changed

  • start_many starts each stream once the query before it has started its source (or stopped, up to two minutes). Spark gives the Python process that plans a query's source a fixed 10 s to connect back, and streams started back to back on a busy driver failed at their start with "Python worker failed to connect back" (9 of 20 in lab check t11 on a host other jobs kept busy, none of 20 on an idle one; t11 --back-to-back repeats it).
  • backfill() raises ValueError when the stream's snapshotKeys option (else the unique index) would cut the open snapshot's chunks on other columns than it opened with; before, it raised only when the unique index had changed.
  • snapshotKeys must be a non-empty JSON list of column names, or it is a ValueError; a JSON string was read as one column per letter. Its names match the captured columns ignoring case, as columns does, and a chunked snapshot records them as captured.
  • to_delta(snapshot="full", snapshot_on_switch=True) raises ValueError while a chunked snapshot of the stream is open, as a full bootstrap does: its switch batch would read a whole snapshot next to the chunked one. Before, only bootstrap=True and on_data_loss="resnapshot" runs checked.

Fixed

  • backfill() plans a keyset plan's chunks, and seeks its last chunk's end, after the stream's configured capture instance was disabled (the last step of a switch): the key types come from the table's newest instance, as the snapshot reader's do. Before, it raised "Capture instance ... not found" until captureInstance named the new instance.

0.7.0 - 2026-10-13

State compatibility: offsets and checkpoints unchanged. A new table kind, the snapshot plans table (<target>_snapshot_plans, next to bronze), starts at schema version 0 with its own migrations module, migrations/plans.py; facts migration 13 rewrites the comment of detail (metadata only) the next time a stream opens the facts table. No mssql_cdc.min_version is set.

  • The payload keys that hold key values (keys and plan of 'snapshot_open', chunks of 'snapshot_plan', lo and hi of 'snapshot_chunk') leave only the facts rows this release writes for a chunked snapshot it opens; the plans table holds each payload whole (ADR 0021 amendment 9). Rows written before are read as before.
  • A chunked snapshot opened by an earlier release finishes on this one the way it started: its key values stay in its facts rows and no plans table is written, so jobs of both releases that share those facts can read it. tests/compat finishes each release's open snapshot this way.
  • In the other direction, on a snapshot this release opened, an earlier release's apply_changes applies the waves without range deletes (the rebuild at completion removes those keys), and its backfill() and reconcile() fail with KeyError. Upgrade the jobs that run them together with the stream.

Breaking

  • For type checkers only: SnapshotPlanDetail.chunks, SnapshotChunkDetail.lo and SnapshotChunkDetail.hi are now optional keys, since a facts row written by this release lacks them. At run time nothing changes and mypy still accepts d["chunks"], but pyright in strict mode reports reportTypedDictNotRequiredAccess: read them with .get() or check for the key first.

Added

  • Options user and password: the login is passed to the driver apart from the connection string (mssql-python's UID/PWD keyword arguments, arrow_odbc.connect(user=, password=)). The driver escapes them, so a password with ; or } needs no braces, and Spark redacts an option named password in a query's plan by default, which it never does for connectionString (a Spark Connect server's record of the request is another matter: see Security) (ADR 0034).
  • Option accessToken: a Microsoft Entra ID access token, passed by mssql-python as the SQL_COPT_SS_ACCESS_TOKEN pre-connect attribute (ADR 0034).
  • Option tokenProvider: package.module:function, which the library imports and calls with no arguments each time it opens a connection, on the driver and in every executor task. It returns the token, so a long stream renews it by itself. Only the name crosses to the executors; the module (with azure-identity, for instance, which the library does not depend on) must be installed on every node (ADR 0034).
  • Combinations that leave the login ambiguous fail with a ValueError before anything connects: accessToken with tokenProvider; a token with user, password, or UID/PWD/Authentication/Trusted_Connection in connectionString; user with UID, password with PWD. backend=arrow-odbc refuses a token: arrow-odbc sets no attribute before it connects.
  • SnapshotPlanDetail.chunk_count: the number of chunks of the plan, which a facts row written by this release carries in place of chunks.

Changed

  • A chunked snapshot no longer writes source key values to the facts table. Those values are the key's MIN and MAX at its open and the bounds of every chunk, which with a composite, string or date key are real keys of the table. to_delta(snapshot="chunked") and backfill() now write them to a new table next to bronze, <target>_snapshot_plans: bronze's schema for a name, a sibling directory for a path, so they fall under bronze's access policy. backfill(), apply_changes and reconcile find the table by bronze's name; nothing new needs to be passed. For a snapshot opened by this release, the facts rows leave out keys and plan ('snapshot_open'), chunks ('snapshot_plan', which gains chunk_count) and lo and hi ('snapshot_chunk'). The new table's detail holds each payload whole, so a query that read them from the facts reads them there (ADR 0028 Amendment 4, Tables).

Security

  • The secret need not live in connectionString any more (user/password, accessToken, tokenProvider, above). Spark's default redaction hides the password, accessToken and tokenProvider options in a query's plan, but shows connectionString and user in full. A Spark Connect server (Databricks Connect, standard access mode, serverless) also records each request's options in the Connect tab of its Spark UI and in its event log, redacted only by spark.sql.redaction.string.regex, which is unset by default: there, prefer tokenProvider or a managed identity, or set the pattern the options reference gives.
  • examples/databricks_notebook.py passes the login as user and password from the secret scope, no longer as UID and PWD in connectionString, which Spark shows whole in the streaming query's plan.
  • With the user, password, accessToken or tokenProvider options, a driver error raised while connecting whose message (or any error down its chain) holds one of their values, SQL Server's "Login failed for user '...'" among them, is raised as a ConnectionError with each value replaced by ***. Errors that hold none, and every error of a connection given only connectionString, are raised unchanged, so the retry of transient errors (ADR 0029) still sees them. No option value goes into the facts, the metrics files or a commit's userMetadata.
  • The facts table, often readable by more people than bronze, no longer holds source key values for a chunked snapshot this release opens (see Changed).

0.6.0 - 2026-10-10

The changes are listed under 0.6.0rc3, rc2 and rc1 below; this release adds only the documentation of where it has run. The library ran on Databricks serverless (environment 4, a Spark Connect client) with the fake backend: a stream with availableNow, a chunked bootstrap with backfill(), apply_changes, reconcile and finalization, with the cache API refused by the platform; 24 checks passed (bronze equal to the source, no duplicates, facts counted without persist(), metrics files in a Volume, silver equal to the table). Serverless refuses processingTime triggers. Its SQL Server path was not run there: the validation workspace's serverless compute has no network path to the server.

State compatibility: as 0.6.0rc1 to rc3.

0.6.0rc3 - 2026-10-10

Fixed

  • On Databricks serverless, a stream died on its first batch with rows: InvalidDefinitionException: Java 8 date/time type java.time.LocalDateTime not supported while Spark serialized the query's progress. The uncached sink (0.6.0rc1) observed the batch's first and last commit times as TIMESTAMP_NTZ values, and serverless writes observed metrics with a JSON mapper that has no java.time support. They are now observed as text and parsed back, with the same facts values. Found by running the library on serverless (with the fake backend): every other check passed there, and processingTime triggers are refused by the platform, as documented.

0.6.0rc2 - 2026-10-10

Fixed

  • On PySpark 4.0 and 4.1, Databricks serverless environment 4 included, every read failed with TypeError: to_arrow_schema() got an unexpected keyword argument 'timezone' since 0.1.0 (the stream's and the snapshot's reads), and reconcile() too since 0.2.2: only PySpark 4.2's to_arrow_schema takes timezone; 4.0 and 4.1's take timestamp_utc, whose default reads TIMESTAMP columns as UTC instants as well. A new CI job, spark41, runs the tests that start no JVM and the source's engine tests on PySpark 4.1.1.

0.6.0rc1 - 2026-10-09

State compatibility: no migration; offsets, checkpoint layout, table schemas and the metrics files' names and layout are unchanged, and the state every released wheel wrote also resumes through a Spark Connect session. Where the platform refuses caching:

  • a micro-batch's bronze commit has rows to updates null in its userMetadata, and a batch with no rows right after one with rows writes an empty commit (so does a run's first without a facts table);
  • a backfill() wave's commit has rows, high_lsn, read_seconds and read_mb null for each chunk. WaveChunk.rows is now int | None (ADR 0021 amendment 8).

The facts table holds those values. A 0.5 release that finds such a wave commit after a crash writes NULL counts for its chunks.

Added

  • Spark Connect: the library runs from a Spark Connect client. It is tested against a local Spark Connect server with Delta Connect and the fake backend, in a new CI job, connect (ADR 0033). Databricks serverless and Databricks Connect are Spark Connect clients too, but neither has run it yet. On serverless, pass trigger={"availableNow": True}. The network path from serverless to SQL Server is yours to provide. The calls that talk to SQL Server themselves (bootstrap, backfill(), reconcile()...) and the track() listener run in the client process.
  • On a platform that refuses the DataFrame cache API (Databricks serverless), delta_sink and backfill() now append each batch or wave as they read it instead of failing on persist(). observe() counts the rows on the way, so the facts table gets the same values. The source is read once, except for a batch with rows right after one without: the sink asks isEmpty() before that append, which reads the batch's first partition with rows up to its first row (an empty batch whole, which then writes no commit). For a run's first batch, the batch before it is the facts table's last. A Spark Connect server can answer the append before its observed metrics are in; the sink then counts the batch with one more read, and backfill() counts the wave's chunks in bronze. Where the platform caches, nothing changes. The code tells the two apart by trying persist(), never by platform name. Tested on classic Spark with the cache API made to raise, and through a local Spark Connect server that refuses it in its foreachBatch worker; not run on serverless yet (ADR 0032).
  • metricsPath can be a URI that pyarrow.fs opens (s3://, gs://, abfss://, hdfs://, file://), for platforms with no shared local or FUSE path (EMR, Dataproc, Spark Connect). The metrics and event files are written, listed, read and removed through pyarrow.fs.FileSystem.from_uri, with credentials from pyarrow's defaults on each node or options in the URI query (?region=...). Local and FUSE paths work as before, and so do the layout and file names. Each file is written whole: under a temporary name and moved, or, on a filesystem that cannot move, in one upload under its own name. Tested with file:// URIs and a filesystem without rename, not against a real object store (ADR 0014, Amendment 6).

Changed

  • reconcile() compares counts only (Tier 1) where the platform refuses localCheckpoint, instead of failing: it logs a warning and every bucket has hashed false.
  • to_delta no longer raises ValueError for a URI metricsPath. The reader raises it when the query starts, and only for a URI pyarrow.fs cannot open, such as dbfs:/. On Databricks, keep the metrics in a Volume: pyarrow does not use Unity Catalog's storage credentials.

Fixed

  • A bootstrap's or re-snapshot's facts row could record 0 rows when another writer (the stream, auto compaction) committed to bronze between the snapshot reading the table's latest version and its history. The snapshot's own commit is now selected by version, as are a micro-batch's and a backfill() wave's where the platform refuses to cache.
  • mssql_cdc.spark.get_spark() raised CANNOT_CONFIGURE_SPARK_CONNECT_MASTER when SPARK_REMOTE was set. It now returns a session on that Spark Connect server.
  • Lab checks t5 and t7 failed through Spark Connect because they read numInputRows from progress, which a Connect progress leaves out. They now sum it from the sources.

0.5.0 - 2026-10-09

The same code as 0.5.0rc1; its changes are listed under it below. Measured against a production SQL Server 2016 from Databricks (DBR 18.2, dedicated, numPartitions 4, chunk_rows 1,000,000, the default target_wave_seconds), next to a running stream, with the same checks as for 0.2.0 (every planned chunk read once, no duplicate key, silver equal to the source, reconcile matching every bucket):

  • a 34-million-row table: from the open to completion in 586 s instead of 943 s with 0.2.0 (3 waves instead of 9; 87,000 rows/s during the waves instead of 58,000);
  • a 100-million-row table: 4,427 s instead of 5,096 s (17 waves instead of 27); this read is bound by the link, so the fixed cost per wave weighed less.

A full bootstrap of the same tables takes 325 s and 3,364 s, so a chunked snapshot now costs 1.8 and 1.3 times its wall time (2.9 and 1.5 times with 0.2.0), in exchange for being read next to the stream over as long as it takes.

State compatibility: unchanged from 0.4.1 (as 0.5.0rc1).

0.5.0rc1 - 2026-10-08

State compatibility: unchanged from 0.4.1. Offsets, checkpoint layout, table schemas, facts events and payloads, and the userMetadata of bronze commits keep their shape; no migration.

Added

  • backfill(target_wave_seconds=None): how long a chunked snapshot's wave should take. A wave takes whole rounds of numPartitions chunks: as many rounds as the target (300 s by default) holds at the pace of the last wave read (an earlier call's, from its facts rows), and no more than what is left of max_seconds. 0 reads one round per wave, as before. The plan's chunks, their order and chunk_rows do not change (ADR 0028 amendment 3).

Changed

  • backfill() reads the next wave while a background thread writes the previous wave's bronze commit and facts rows. At most one wave reads while one commits. Bronze appends stay in wave order, facts rows still follow their commit, and crash recovery is unchanged. A call opens one more connection to SQL Server (for commit times) and can hold two waves' rows cached at once. A wave of more chunks than numPartitions still reads numPartitions at a time.
  • By default a wave now takes more than numPartitions chunks once a wave's pace is known (from the second wave of a call, or from the first after an earlier call). Pass target_wave_seconds=0 to keep the 0.4 behaviour. The Databricks benchmark against a production source follows the release candidate.
  • While a chunked snapshot is open, apply_changes reads only the waves that arrived since its last call. Delta now skips the files of the earlier waves, the files of the whole snapshot silver was rebuilt from, and the changes older than the first new wave's stamp. Results are unchanged; what still grows with the waves is the MERGE over silver (ADR 0019 amendment 3).
  • A full snapshot of a composite or non-integer key cuts its partitions with keyset seeks instead of NTILE. The row count comes from sys.sp_spaceused, the ranges take NTILE's sizes, and each bound is a seek of its range's keys from the bound before (as in a chunked snapshot's keyset plan), under the snapshot's isolationLevel. NTILE read and sorted or spooled every key of the table before the first row was read. On 40,000 rows in four ranges, each seek reads at most 10,002 keys, with no sort and no spool. With an exact count the partitions are NTILE's; with a stale one every row is still read once. A single integer key keeps its MIN..MAX ranges (ADR 0016 amendment 4).
  • The default backend sizes each Arrow batch by the widest row of the batch before (the longest text or binary value) instead of its mean. Every read starts with one row, and each fetch takes at most twice the previous batch's rows. A table of (max) values therefore no longer builds gigabyte batches from a large first fetch or from one long value among short ones. A narrow table reaches arrowBatchSize from its 15th fetch on. MssqlPythonBackend takes max_bytes_per_batch, as ArrowOdbcBackend does (ADR 0003 amendment 5).
  • Lab: t8 --concurrency times each backend over 1, 2 and 4 partitions, each with a connection of its own. arrow-odbc's concurrent fetch gains 19% to 48% at 1 and 2 partitions and nothing at 4; the backend still turns it off (ADR 0003 amendment 4).
  • Tests: the fake-vs-SQL Server parity test gives each scripted transaction a commit time of its own. sp_cdc_cleanup_change_table lowers a low water mark to the first cdc.lsn_time_mapping entry sharing its tran_end_time (documented; datetime, 1/300 s), so two commits within one tick made the test flaky. The library is not affected: cleanup deletes exactly below the min_lsn it reports, so the retention guard stays sound.

0.4.1 - 2026-10-08

State compatibility: unchanged from 0.4.0. Code moved and tests added only: offsets, checkpoint layout, table schemas, facts events and payloads are as in 0.4.0; no migration.

Changed

  • mssql_cdc.client is a package of small private modules (protocols, validators, T-SQL builders, backends, SqlCdcClient, chunk planning). Code moved only: every name imports from mssql_cdc.client as before, and the exceptions and the mssql_cdc.client logger keep their names. Tracebacks show the new file paths.
  • mssql_cdc.pipeline is a package too (CdcStream and to_delta, whole snapshots, recovery, chunked snapshots, the snapshot mode lock), and the facts event rows are written and read through one module, mssql_cdc.events: their names, row builders and readers. Code moved only: every name imports from mssql_cdc.pipeline as before (mssql_cdc.sink.write_event too), the mssql_cdc.pipeline logger keeps its name, and every facts row and query result is the same.
  • Both packages declare __all__: from mssql_cdc.client import * and from mssql_cdc.pipeline import * bind the public names the modules bound (Lsn, and Offset, SnapshotMode and the other mssql_cdc.types names in pipeline), no longer their stdlib imports.
  • Tests: tests/test_properties.py checks properties over inputs Hypothesis generates (it joins the dev group, pinned by uv.lock): LSN round trips, canonical form and order; chunk plans tiling random key distributions; each key's latest image in silver whatever the order of its change rows; the spellings a boolean option takes. Behaviour is unchanged.
  • Tests: one mutation-testing pass (mutmut, configured in pyproject.toml and run through uv run --with, not a dependency) over the LSN helpers, chunk planning, validators and the pure silver and reconcile helpers, and tests for the gaps it found; scores and the equivalent mutants are in the development guide. Behaviour is unchanged.

0.4.0 - 2026-10-07

State compatibility: unchanged from 0.3.0. Types only: offsets, checkpoint layout, table schemas and payload keys and formats are as in 0.3.0, now described by types. A 'data_skipped' row's detail (from, to, certain, reason), shipped since 0.2.0, joins the payloads ADR 0021 lists as state (amendment 7). A chunked 'snapshot_open' row writes keys and plan later in its JSON object; readers take keys by name.

Added

  • Backend and CdcClient are typing.Protocols, runtime_checkable, exported from mssql_cdc and listed in the API reference (ADR 0030). Your own backend satisfies Backend without inheriting (batches, scalar, close), and isinstance checks it. Subclassing still works and inherits the methods that have a body. A method later added to a protocol is a break for a class that implements it without inheriting, listed under "Breaking".
  • Lsn, a NewType of str for an LSN in its canonical form (0x and 20 uppercase hex digits). lsn.normalize, lsn.from_int and the client's methods return it. Parameters still take any str, and at run time it is the same str.
  • mssql_cdc.payloads: TypedDicts for the JSON a later release reads back, plain dicts at run time, for Python that reads the facts table (d: SnapshotChunkDetail = json.loads(row["detail"])): SnapshotOpenDetail, SnapshotPlanDetail, SnapshotChunkDetail, SnapshotCompletionDetail, DataSkippedDetail, BatchDetail and WaveMetadata (a wave commit's userMetadata), exported from mssql_cdc, with the nested WaveChunk, SnapshotExtent (IntExtent or KeysetExtent) and SnapshotKind. Listed in the facts table's reference next to detail (ADR 0021 amendment 7).
  • SourceOptions, exported from mssql_cdc: a TypedDict of every source option with a string value. Annotate an options dict with it and a type checker flags a misspelt name. stream(), reconcile(), apply_changes(options=) and start_many() take a SourceOptions or any mapping (they took dict), so every call that type-checked before still does.
  • SparkSessionLike and StreamingQueryLike, exported from mssql_cdc: a classic or a Spark Connect session or query (Databricks Connect, serverless compute), which does not subclass the classic one. Every spark and query parameter takes them, so a Connect caller's type checker accepts register, stream, start_many, apply_changes, reconcile and the finalization functions (ADR 0031 amendment 1).

Changed

  • Typing: every function, method and attribute of the package is annotated; mypy checks src in strict mode, and pyright --verifytypes scores the public API 100% type-complete (76% before). CI's lint job fails on a mypy error or a lower score (ADR 0031). A type checker now sees the types of spark parameters, options, the clients' methods and the helpers' results, which it saw as unknown. Behaviour is unchanged.
  • Typing: CdcClient.max_lsn() returns Lsn | None, as it did at run time (NULL on a database capture has not written to yet), and WaveChunk.high_lsn is str | None (null when the chunk's task could not read max_lsn). Both types are new in this release.
  • Typing: pyright joins the dev group, pinned by uv.lock; make lint and scripts/lab.ps1 lint run its --verifytypes as CI does.

Deprecated

  • Nothing. Backend and CdcClient were ABCs in 0.3; they are now the protocols themselves, not new classes beside them, so class MyBackend(Backend) keeps inheriting the method bodies and an incomplete subclass still fails to instantiate. There is no separate ABC to phase out (ADR 0030).

0.3.0 - 2026-10-07

State compatibility: offsets and checkpoints unchanged. Facts migration 12 rewrites the comment of detail (metadata only), the next time a stream opens the facts table; a micro-batch row's detail gains the key warnings, an added payload (ADR 0021 amendment 6). Rows written before, and batches without warnings, keep detail NULL. No new event value, no other payload key changed.

Breaking

  • Options, flags and modes are keyword-only (ADR 0021 amendment 5). A positional call now raises TypeError before anything runs. To migrate, pass the same values by name:
  • stream(...).to_delta(target, app_id, checkpoint, facts_table=None, *, ...): everything after facts_table (trigger, query_name, bootstrap, on_data_loss, resnapshot_interval_days, snapshot_on_switch, snapshot) by keyword, e.g. to_delta("bronze.orders", "orders-v1", ckpt, "ops.facts", bootstrap=True).
  • stream(...).snapshot(target, *, resnapshot=False, ...): snapshot(t, True) becomes snapshot(t, resnapshot=True).
  • apply_changes(spark, bronze, target, *, capture_instance=None, keys=None, ...): apply_changes(spark, b, s, "dbo_orders", ["order_id"], ...) becomes apply_changes(spark, b, s, capture_instance="dbo_orders", keys=["order_id"], ...).
  • reconcile(spark, options, silver, *, keys=None, ...).
  • granularity of finalization.advance(), finalization.track(), finalization.candidate() and finalization.FinalizationListener(spark, run_id, control_table, table_name, *, granularity="hour"): advance(spark, control, table, end, "day") becomes advance(spark, control, table, end, granularity="day").
  • timeout of await_all() and of FinalizationListener.join(), and source_index of finalization.end_offset_from_progress(): await_all(queries, 600) becomes await_all(queries, timeout=600).
  • sink.delta_sink(target, app_id, facts_table=None, *, metrics_path=None) and spark.get_spark(app_name, master, *, delta=True).
  • For type checkers only: snapshot(), seed(), backfill(), apply_changes(), reconcile() and finalization.end_offset_from_progress() now return TypedDicts instead of dict. Code annotated to take their result as dict should take the new type or Mapping[str, Any]. At run time they are the same dicts.
  • For type checkers only: backfill(isolation=) now takes an Isolation and finalization.candidate(granularity=) a Granularity, where they took any str. A str that holds a valid value, a job parameter say, needs a check or typing.cast(Isolation, value) to type-check. The Literal names the canonical spelling ("readCommitted"); at run time both still match ignoring case.

Added

  • mssql_cdc.types, exported from mssql_cdc:
  • Literal aliases for the mode parameters: SnapshotMode, OnDataLoss, Isolation, Granularity, and BackfillState for backfill()'s state.
  • TypedDicts for the results, plain dicts at run time: Offset (of snapshot(), seed() and finalization.end_offset_from_progress()), BackfillStatus, ApplyResult, ReconcileResult.

The signatures use them, so a type checker flags a misspelt mode or result key. A wrong mode still raises ValueError naming the allowed values. - A micro-batch row of the facts table (event NULL) now carries the warnings the reader logged on the driver in detail, as JSON {"warnings": [...]}, only when there are any: columns it does not read (computed columns, or columns a newer capture instance captures that columns leaves out), unknown options, and the UTC offset it converts commit times with on a server older than SQL Server 2022. Before, they were only in the driver's worker log. A restart whose first batch replays one already in the facts puts them in the next batch's row. Needs metricsPath, like the other metrics (ADR 0023 amendment 6).

Changed

  • finalization.advance() and finalization.candidate() take the offset as any mapping (Mapping[str, Any]), so an Offset and a parsed progress offset both type-check.

0.2.2 - 2026-10-07

State compatibility: unchanged from 0.2.1 (offsets keep their format; no migration, no new event value).

Fixed

  • With a named source time zone that has daylight saving (sourceTimeZone, or auto on SQL Server 2022), commits in the hour a fall-back repeats now keep commit order. SQL Server's AT TIME ZONE reads a repeated time with the offset before the change, so commits made after the clock went back got commit times an hour early, behind the commits before them, and their rows could land in periods finalized_until had already declared final. Those commits now get the offset after the change (ADR 0008 amendment 4).
  • Before SQL Server 2022 (sourceTimeZone=auto, fixed-offset fallback), the driver now reads the server's UTC offset again for each new batch instead of once per run, and sends it with the batch. A long run follows a daylight-saving change from its next batch, and a batch's end offset and rows use the same offset. A warning is logged when the offset changes. On Spark 4.0/4.1 an idle poll now sends max_lsn alone; before, it also read the commit time on every poll.
  • reconcile() no longer reports stream lag as an integrity failure. A difference explained by a change the stream has not read yet is now IN_FLIGHT, not MISMATCH, MISSING_TARGET, MISSING_SOURCE or RECORD_DIFF. Once the source is read, it reads the keys in the change table from bronze's position up to sys.fn_cdc_get_max_lsn() read then, through the capture instances the stream reads. This is the change table the stream already reads, so no new grant is needed. Only a commit the capture job has not harvested yet stays unseen (ADR 0028 amendment 2).
  • reconcile() now finds a change to a key whose value is NULL (the join is null-safe), so that key's difference is IN_FLIGHT instead of RECORD_DIFF or MISSING_*. The report's key now names it, {"code":null}, instead of {}.

0.2.1 - 2026-10-06

State compatibility: unchanged from 0.2.0 (no migration, no new event value).

Fixed

  • pyarrow 19 or later: with pyarrow 18 imported before mssql_python, the Python interpreter crashes on Windows (access violation). The install pins in the Databricks page, the docs home and the example notebook say 0.2.0, not 0.2.0rc1.

Changed

  • Tests and CI: logic the streaming engine is not needed for is tested in plain unit tests (specced mocks, pure functions), the test session starts Delta only for the tests that take it, and -m "not spark and not sqlserver" runs the JVM-free tests alone. CI runs the unit suite in three shards next to the integration and lab jobs instead of before them: about 13 minutes instead of 48. A parity test runs one scripted change history on the fake and on SQL Server 2022. tests/compat/0.2.0 holds the state the 0.2.0 wheel wrote.
  • CI: a release tag no longer starts its own ci run; the release commit's run on main is the one the pypi approval waits for (docs/RELEASING.md). zizmor treats a workflow a tag triggers as publishing, and ci restores caches.
  • CI: a security workflow checks the supply chain on every pull request, every push to main and weekly. pip-audit checks every package uv.lock pins against the known vulnerabilities; zizmor audits the workflows and fails on a finding of medium severity or higher; a job installs the lowest version of each direct dependency the package declares (uv sync --resolution lowest-direct, Python 3.10) and runs the tests that start no JVM; the OpenSSF Scorecard runs from main and publishes its results.

0.2.0 - 2026-10-06

The first final release of the 0.2 line: the same code as 0.2.0rc2. Its changes are listed under 0.2.0rc2 and 0.2.0rc1 below. Both release candidates were validated against a production SQL Server 2016 from Databricks (DBR 18.2): chunked snapshots next to a running stream, silver applied wave by wave and reconcile matching every bucket, on a 34-million and a 100-million-row table with 0.2.0rc1 and again on the 34-million-row one with 0.2.0rc2.

State compatibility: offsets and checkpoints unchanged since 0.1.0. Tables written by 0.1.0 migrate the next time the library opens them, through the migrations listed in the State compatibility lines of 0.2.0rc1 and 0.2.0rc2; tests/compat/0.1.0 resumes such state with this release.

0.2.0rc2 - 2026-10-06

State compatibility: offsets and checkpoints unchanged. Control migration 4 rewrites the comment of finalized_until (metadata only), the next time finalization.advance or apply_changes opens the control table. A migration interrupted between its change and its version stamp now completes on the next open instead of failing every writer (add_columns skips the columns a table already has), and one that loses to another job's concurrent commit runs again. A release still writes a table a newer one migrated, with a WARNING, so jobs that share a table can upgrade or roll back one at a time; a release that knows fewer migrations than the table property mssql_cdc.min_version names refuses it, and no migration sets that property yet (ADR 0013 amendment, ADR 0021). Facts migration 10 rewrites the comments of event, lost_from_ts, lost_to_ts, detail, batch_id and the table (metadata only), and event gains the value 'data_skipped'; facts migration 11 rewrites those of event, lost_from_ts and detail again, for the possible skips a task records (metadata only) (ADR 0018 amendment 3). The state contract also covers the silver and reconcile schemas, the facts event values and the keys of a snapshot row's detail and of a backfill wave's userMetadata, keys that are only ever added (ADR 0021 amendment 4); tests/compat/<version> holds the state each release's wheel wrote, and the tests resume it (amendment 3). A bronze or silver table created with a column name Delta refuses without column mapping gets it, which adds the columnMapping feature to its Delta protocol (older readers and writers refuse the table); existing tables are untouched. A restarted stream's inferred schema leaves out computed columns; an existing bronze table keeps such a column, and the rows appended after the upgrade hold NULL in it, as its change rows always did.

Added

  • lockTimeoutMs source option (off by default): SET LOCK_TIMEOUT before every read of the source table (snapshot, chunked-snapshot planning, reconcile), so a read blocked by a writer fails with error 1222 and Spark retries the task instead of hanging (ADR 0029).
  • The stream's driver-side calls (initialOffset, latestOffset, reportLatestOffset, partitions, prepareForTriggerAvailableNow) retry a transient SQL Server error (SQLSTATE 08xxx, HYT00, HYT01, 40001) on a new connection, up to 3 times within about 14 s, with a WARNING each time; DataLossError, SchemaChangedError, ValueError and PermissionError are never retried (ADR 0029).
  • apply_changes takes the capture instance from the options' captureInstance when capture_instance is omitted (one given still wins).
  • is_data_loss(exc) and is_schema_changed(exc) recognise DataLossError and SchemaChangedError, also inside the StreamingQueryException that awaitTermination() raises or await_all returns.
  • Bronze and silver tables are created with delta.columnMapping.mode = 'name' when a captured column name holds a character Delta refuses otherwise (a space or ,;{}()=, as in [Unit Price]); such tables used to fail on their first write. Other Delta features (deletion vectors, clustering, OPTIMIZE, type widening) are documented as opt-ins under Tables > Table properties (ADR 0012 amendment).
  • When a newer capture instance adds such a column to a bronze or silver table created without column mapping, the stream and apply_changes fail before writing with a SchemaChangedError that names the column, says column mapping is off on the table and gives the ALTER TABLE ... SET TBLPROPERTIES ('delta.columnMapping.mode' = 'name', 'delta.minReaderVersion' = '2', 'delta.minWriterVersion' = '5') that enables it (an upgrade of the table's Delta protocol, which its readers must support), instead of Delta's own error midway; the library does not enable it (ADR 0012 amendment).
  • With failOnDataLoss=false, a batch whose planning skips changes CDC cleanup purged writes a data_skipped facts row: the batch's batch_id, rows 0, min_lsn = max_lsn the min_lsn it resumed at, the LSNs skipped in detail (JSON {from, to, certain}, certain true) and the gap in lost_from_ts and lost_to_ts, as on a resnapshot row. It needs a facts table and metricsPath (to_delta sets it) (ADR 0018 amendment 3).
  • A task that finds, after reading its range, that CDC cleanup ran meanwhile (the executor-side retention guard, failOnDataLoss=false) writes a data_skipped facts row too, the loss possible rather than certain: detail {from, to, certain: false, reason}, min_lsn = max_lsn the min_lsn it found, and lost_from_ts/lost_to_ts from the batch's start offset to that min_lsn. It rides in the partition's metrics file, so a retried task or a replayed batch writes it once; it used to be only logged (ADR 0018 amendment 3).
  • A compatibility test against the state released versions wrote: tests/compat/<version> holds a checkpoint with two generations and the bronze, silver, facts and control tables that version's wheel wrote (0.1.0 from its PyPI wheel), and tests/compat/test_compat.py resumes each with the current code; every release adds its own (docs/RELEASING.md).
  • backfill() returns state: done, running, waiting_headroom, waiting_metrics or no_snapshot, so a loop can tell a pause worth waiting out from a snapshot that does not exist (a wrong app_id or target, or a stream that bootstraps with snapshot="full"); the bootstrap guide's loop branches on it.
  • apply_changes returns bronze_found, and logs a warning naming the table when bronze does not exist (before the stream's first batch, or a wrong name). It also warns, once per table, when called without facts_table (its verdict is then held), when bronze has no _command_id, and when the facts table has no row for bronze's name.
  • FinalizationListener.last_error and failures: the error of the last attempt (None once one succeeds) and how many failed in a row. The first failed verdict of a streak is logged at ERROR with its traceback, then one WARNING at most every 10 minutes.
  • INFO logs for the bootstrap, each snapshot and each backfill() wave (target, LSN, rows, seconds, chunks done), and a WARNING with the lost range when to_delta decides to re-snapshot.
  • finalization.finalized_until is in the API reference, and so public.
  • SECURITY.md: how to report a vulnerability privately, and what is in scope.
  • Documentation: alerts on NULL metrics, on the verdict's freshness (control table) and on capture itself from the SQL Server side (sys.dm_cdc_log_scan_sessions, sys.dm_cdc_errors, the capture job, with a monitoring login); retention for the facts table's micro-batch rows and compaction of bronze; the NULLs CDC stores in change rows (computed columns, LOB types); quoting a password in the connection string, Entra ID authentication and Spark's redaction of connectionString; the access policy that the key values in a chunked snapshot's facts rows and bronze history need; the Microsoft license of the default driver's binaries, and how to install without them; the driver's Python process per stream in start_many; what a Spark Connect client must reach; the source-query cost of the default trigger; minute granularity across a daylight-saving fall-back; the tested platform scope, local Spark 4.2 and Databricks classic compute on a single node, with metricsPath on a path every node sees (without one, as on EMR or Dataproc by default, the metric columns stay NULL and schema change events do not reach the facts); Databricks serverless, unsupported until tested. The site says it documents main (ADR 0003 amendment 3).

Breaking

  • failOnDataLoss and includeCommandId accept only true/false, 1/0, yes/no or y/n (any case), and raise ValueError naming the option otherwise. Values such as on, off or a typo like ture used to read as false, which turned the retention guard off.
  • maxCommitsPerBatch, numPartitions and arrowBatchSize below 1, or not an integer, raise ValueError, and so does a negative connectTimeout. maxCommitsPerBatch=0 used to mean unlimited without a word: omit the option instead.
  • A source column named like a metadata column or one the sink adds (_start_lsn, _chunk, ..., in any case) raises ValueError when the stream loads, instead of being dropped, failing later on a duplicate column, or read as the chunk number: leave it out with columns.
  • to_delta raises ValueError for snapshot_on_switch=True together with snapshot="chunked" (the switch snapshot reads the whole table inside a batch): drop snapshot_on_switch, or use snapshot="full".
  • to_delta raises ValueError for a metricsPath that is a URI (Python wrote it as a local directory named after the scheme, and every metric came out NULL): use a local or FUSE path, such as /Volumes/..., that every node sees. The reader refuses one too.
  • to_delta raises ValueError for on_data_loss="resnapshot" with failOnDataLoss=false (a purge skipped while the query runs would hide the gap from the next run's check): remove failOnDataLoss=false.
  • The inferred schema leaves out computed columns, which SQL Server CDC stores as NULL in every change row, so snapshot rows no longer carry values every later change nulls; load() logs a warning naming them (sys.columns.is_computed, which the reader already sees: no new grant). A query that selects one fails with UNRESOLVED_COLUMN, and a new bronze table lacks it: list it in columns to keep it, NULL in every row, snapshot and seed rows included (ADR 0007 amendment).

Changed

  • reconcile() no longer compares computed columns, also one listed in columns (ADR 0007 amendment).
  • A micro-batch is cut into at most numPartitions ranges of about 50,000 change rows or more, so a batch of fewer than about 100,000 change rows is read in one partition, with one connection, instead of one per core (ADR 0015 amendment).
  • Only a batch's last partition measures where the stream is (retention watermark, max_lsn, capture lag, end commit time); the other partitions skip those four queries (ADR 0014 amendment 5, ADR 0020 amendment 2).
  • The planner reads each capture instance's captured columns (and which are computed) once per query run, and again before checking types whenever a batch holds DDL.
  • stream() and the reader log a WARNING naming any option they do not know, so a misspelt name no longer leaves its default in force silently.
  • The reader logs a WARNING with the file when it cannot write a metrics file, instead of turning the facts' metrics NULL silently.
  • Planning sends fewer queries: each range's start comes from the same query as the split points, and after the first resolution the client lists only the table's own capture instances, falling back to the whole database when that misses.
  • to_delta names the query after the sink's app_id (<app_id>.g<n> in generation n) when query_name is not given, and so does start_many, whose queries kept the generation-less name after a re-snapshot.
  • backfill(isolation=None) reads and plans with the stream's isolationLevel option instead of READ COMMITTED; isolation takes "snapshot" or "readCommitted" in any case and is checked before anything is read.
  • is_final takes an aware datetime in UTC instead of raising TypeError.
  • advance and apply_changes retry a control-table MERGE, and every table's migrations, that lose to a concurrent commit for up to a minute, with backoff, instead of the tracker's two retries.
  • With failOnDataLoss=false, skipping purged changes logs a WARNING naming the capture instance and the LSN range.
  • An idle stream sends one max_lsn query per poll: reportLatestOffset reuses what latestOffset read, and the commit count is skipped when nothing is new.
  • The default backend caps Arrow batches near 64 MiB, as arrow-odbc does, so LOB-heavy rows read in smaller batches; the first batch of each read holds 64 rows, so it stays small before the row width is known.
  • With sourceTimeZone=auto before SQL Server 2022, the driver ships the offset it read to the executors, so a run applies one offset to its offsets and its rows, and logs a WARNING each time it takes that fallback (ADR 0008 amendment 3).
  • The sink logs a warning, once per run, when a batch that read rows found no metrics file.
  • Error messages: "capture instance not found" ends with the driver's own error, and the change-table permission hint also works for logins whose language is not English: on an error naming the change table, the client asks SQL Server (HAS_PERMS_BY_NAME) whether the login may read it, whatever the language and the driver.
  • The Databricks example installs the released version (==0.2.0rc1) instead of the repository's moving main, quotes UID and PWD in braces, and computes the verdict's epoch in UTC.
  • Packaging and CI: pyarrow 18 or later; classifiers for Python 3.10 to 3.13, which the weekly CI run tests; the release workflow refuses a tag whose commit is not on main or whose version has no CHANGELOG.md heading; docker compose binds SQL Server to 127.0.0.1, where .env.example and the lab scripts now point (localhost could resolve to ::1 first), and .env.example ships no password (set MSSQL_SA_PASSWORD); every lab result records the server's @@VERSION.

Fixed

  • backfill() completes a chunked snapshot whose 'snapshot_open' row has no kind (written by an unreleased build that put it in mode), taking generation 0 as a bootstrap and a later one as a re-snapshot, instead of failing with KeyError once its last chunk is in.
  • A checkpoint deleted or rewound while app_id stayed the same made Delta skip every write up to the old batch id while the query ran on and the verdict advanced: with a facts_table, the run's first batch now raises ValueError saying so.
  • on_data_loss="resnapshot" with a checkpoint that Python and Spark resolve differently (a schemeless /mnt/... path on Databricks classic, an HDFS default file system, a Spark Connect client) found no offsets and re-snapshotted on every run: when the facts table holds batches of the stream, the pre-flight now raises ValueError.
  • A migration re-run after a crash between its change and its version stamp, or by a job that lost the race to stamp it, appended duplicate columns and failed.
  • apply_changes failed on a bronze written with includeCommandId=false; it now orders by (_start_lsn, _seqval, _operation) there.
  • Capture instance names with non-ASCII letters (dbo_CafĂ©) failed every call; any name without ] or a control character, up to 100 characters, is accepted. The validators no longer accept a trailing newline.
  • backfill(isolation="SNAPSHOT") passed the reader's check but failed while planning.
  • A stream on a database capture had not written to yet (sys.fn_cdc_get_max_lsn() NULL) failed with TypeError in latestOffset: a NULL max_lsn is now the zero LSN, as the pipeline already took it. startingLsn=latest on a capture instance capture has not reached yet (a quiet database just after the enable) started below the instance's first LSN and failed its first batch with a false DataLossError: it starts just before that LSN, as a snapshot is stamped, and asks to retry when neither is known.
  • A snapshot's own commit is found among every commit after it, not only the last five.
  • A table path holding a backtick is escaped in SQL.
  • A stream whose first batch created bronze while backfill() created it too failed with DELTA_PROTOCOL_CHANGED; a CREATE that loses the race to another writer now takes the table that writer made, for every table the library creates.
  • The Databricks example's epoch was off by the driver's UTC offset on a driver not set to UTC.

0.2.0rc1 - 2026-10-04

State compatibility: offsets and checkpoints unchanged. Existing tables migrate the next time a stream, finalization.advance or apply_changes opens them: bronze migration 2 adds _snapshot and _chunk (NULL on existing rows, whose snapshot is still found by their _start_lsn) and rewrites the comment of _start_lsn; facts migration 7 rewrites the comments of rows, event and detail, and facts migration 8 those of event and detail and the table comment again, and facts migration 9 that of event (the facts schema version is now 9; no column is added); control migration 2 adds open_snapshot_lsn and snapshot_wave, and control migration 3 rewrites the comment of open_snapshot_lsn. New table kind reconcile (the report of reconcile()), with no migrations (ADR 0013).

Added

  • Chunked snapshots read next to the running stream, for tables the link cannot read within the CDC retention: to_delta(..., snapshot="chunked") (with bootstrap=True or on_data_loss="resnapshot"; needs facts_table) records the snapshot's LSN S and the key's extent and starts the stream at S without reading the table, and stream(...).backfill(target, *, app_id, facts_table, chunk_rows, max_waves, max_seconds, min_headroom_hours, isolation), called repeatedly in its own task, reads them in waves of numPartitions chunks, each stamped with max_lsn before its read, one Delta commit per wave. The first call plans every chunk and records the plan in a snapshot_plan facts row that later calls reuse (another chunk_rows is ignored with a warning), so the chunks are fixed ranges and chunks_total is exact. One integer key is counted per slice of a fixed grid in one server-side GROUP BY, dense slices counted again on a finer grid, and the slices packed into chunks of at most chunk_rows rows whatever the skew; any other key gets keyset bounds chunk_rows rows apart. The last chunk ends just above the MAX recorded at the open, so the rows inserted since come from the stream alone (for a keyset plan, at the first key after it, looked for again before each wave while there is none). New facts events snapshot_open, snapshot_plan and snapshot_chunk (last on the final chunk); the bootstrap or resnapshot row comes after the last chunk. A crash between a wave's append and its facts rows reruns the wave without appending twice, rebuilding its facts rows from the commit or, once log cleanup dropped it, from the chunks bronze holds, whatever the rerun's numPartitions; a loss while one is open opens a newer one (ADR 0028).
  • Snapshot reader options snapshotChunks (read only the given key ranges, numbered in _chunk, with a chunk-<i>.json metrics file each), snapshotKeys (the columns those ranges apply to, by default the unique index) and isolationLevel=snapshot (SNAPSHOT isolation where the database allows it; never NOLOCK).
  • apply_changes applies an open chunked snapshot, bootstrap or re-snapshot, wave by wave (control columns open_snapshot_lsn, snapshot_wave): a chunk row never brings back a key deleted after its stamp. With one integer, date or timestamp (datetime2, datetime, smalldatetime) key that the chunks are cut on, each wave also deletes the silver keys of its chunks' ranges that the chunks lack and whose image is older than the chunk's stamp, so the keys a re-snapshot's purged gap deleted leave silver wave by wave; keys in a datetime2(7) bound's microsecond, which Spark cannot place, are left to the rebuild. At the snapshot's completion row silver is rebuilt by absence; its verdict is held while a snapshot is open (ADR 0019 amended).
  • reconcile(spark, options, silver, keys=None, *, bronze, control_table, facts_table=None, bucket_rows, sample, report_table, seed): compares silver with its SQL Server table by row count and key sum per bucket of an integer or date key (one count for other keys), then row by row through a SHA-256 of the captured columns on the mismatched buckets and a sample, read in ranges of keys, and classifies keys as MISSING_TARGET, MISSING_SOURCE, RECORD_DIFF or IN_FLIGHT (against silver's applied_lsn in control_table). With facts_table it checks the newest chunked snapshot's chunks against bronze (CHUNK_TILING, CHUNK_ROWS, CHUNK_STAMP). The report goes to an optional report_table; a "Validation" guide covers it.

  • Lab check t10 (lab/checks/t10_chunked_snapshot.py): a chunked bootstrap next to the running stream under a continuous writer (inserts, updates, deletes, key updates) and a transaction holding a range's locks, and with --resnapshot a chunked re-snapshot after a forced cleanup purged a gap with deletes, whose keys leave silver wave by wave. Passes on SQL Server 2022 CU27; runs in the CI lab job.

  • stream(...).seed(target, df, as_of): seed the target from a copy of the table you already have, for tables too big to snapshot within the CDC retention. as_of is an LSN or the UTC time the copy started, mapped on the server's clock (sys.fn_cdc_map_time_to_lsn, through the new CdcClient.time_to_lsn). The copy is written as snapshot rows with a bootstrap facts event, and to_delta(bootstrap=True) starts from it; a rerun returns it, also after cleanup passed as_of. Options allow_missing_columns and reseed (ADR 0025).

  • finalization.track(spark, query, control_table, table_name): continuous-mode finalization. A StreamingQueryListener advances finalized_until after every batch of a running query (a processingTime trigger or the default), from a worker thread, off the listener bus, retrying a failed MERGE twice; join() waits for the last verdict after the query stops (ADR 0026).
  • start_many, await_all and stop_all (mssql_cdc.fanout): one to_delta stream per capture instance from {ci} templates for target, app_id and checkpoint, with per-table option overrides, a shared facts table created before the first start, and failures isolated per query; a "Many tables" guide covers sizing, one job versus one job per table, failure isolation and Databricks (ADR 0027).
  • CI runs the integration tests that read through the backend (14 of them) a second time with backend=arrow-odbc (ODBC Driver 18 installed in the job): the arrow-odbc backend is now tested, except for the heartbeat, facts metrics, re-snapshot, silver, type changes and dropped columns (ADR 0003 Amendment 2).
  • Lab check t9 (lab/checks/t9_capture_instance_switch.py): a table switched to a new capture instance with a new column under a continuous writer, running sql/switch_capture_instance.sql as written with a least-privilege login. Passes on SQL Server 2022 CU27 and 2017 CU31; runs in the CI lab job (ADR 0023 amended, which also records that the production SQL Server 2016 SP3 change tables have __$command_id).

Breaking

  • apply_changes advances silver's finalized_until only when given facts_table: a chunked snapshot shows in the facts alone until its first wave, and an emptied table's re-snapshot only there, so without them silver could claim periods it does not hold. A stream without a facts table can pass any name where no table exists.

Changed

  • to_delta(bootstrap=True) with either snapshot mode, and snapshot(), return the S of a chunked snapshot of the target, open or complete, instead of reading the table again; seed() refuses a target that holds one. A recovery that opened a chunked re-snapshot and stopped before writing its state opens a newer one, a generation on, once cleanup has passed the first.
  • One snapshot mode per run, locked while a snapshot is open: with a facts table, a full snapshot taken by to_delta now writes a snapshot_open facts row (detail mode full, kind bootstrap or resnapshot; one per run, at its own LSN) before it reads the table, closed by its bootstrap or resnapshot row. to_delta (with bootstrap=True or on_data_loss="resnapshot"), snapshot() and backfill() raise ValueError while a snapshot of the stream in the other mode is open, and seed() while one in either mode is, saying how to finish it; the mode may change between runs. A full one stops counting once CDC cleanup passes its LSN, as it can then never complete: a full re-snapshot that outlived the retention can be retried with snapshot="chunked" in the same generation. snapshot() takes new keyword arguments app_id and facts_table for the check (ADR 0028 amended).
  • backfill(isolation="snapshot") plans the chunks under SNAPSHOT isolation too: the counts and seeks of the first call and the search for the first key after MAX before each wave no longer wait for a writer's locks under READ COMMITTED. plan_chunks, last_bound and the client's key_buckets, key_range and key_bound take an optional isolation (ADR 0028 amended).
  • The file-backed fake gives an update's 3 and 4 rows one __$seqval and __$command_id, as SQL Server does, so the unit tests rank them by __$operation alone.
  • A snapshot in bronze is named by its _snapshot column, never by the largest _start_lsn of its rows: snapshot(), to_delta(bootstrap=True), the re-snapshot's reuse check, seed()'s reruns and silver's rebuild point read coalesce(_snapshot, _start_lsn) of whole snapshots, and the facts' completion rows (ADR 0016 amendment 3, ADR 0025 amended).
  • apply_changes: operation 3 deletes its own key, outranked by the 4 of the same key and commit, instead of being ignored, so a key update SQL Server records as 3 and 4 cannot leave the old key in silver.
  • The bootstrap guide sizes a snapshot by MB/s (1.8 to 5.9 MB/s measured against a production source), not a fixed rows-per-second figure: rows per second vary with the row width.
  • Snapshot key bounds are bound as text and CAST server-side for both backends (binary as hex): the same SQL shape and the same seeks (ADR 0016 amended).
  • docker-compose.yml takes MSSQL_IMAGE to run the lab on another SQL Server version (e.g. mcr.microsoft.com/mssql/server:2017-latest).

Fixed

  • The arrow-odbc backend now reads what mssql-python reads:
  • it adds the ODBC Driver 18 keyword when the connection string names no driver, and honours connectTimeout;
  • text is UTF-16 both ways;
  • (max)/text/xml/image values up to 64 KiB (longer ones fail the read);
  • datetime2 is fetched in microseconds (no overflow past 2262);
  • datetimeoffset is converted to its UTC instant;
  • close() releases the connection.
  • The integration tests retry enabling CDC while the SQL Server Agent is still starting (error 14258).

0.1.0 - 2026-10-01

State compatibility: first release: offsets v1 contract (ADR 0002), checkpoint generations (ADR 0018), facts migrations 1-6 (6: the detail column of the source's events), bronze migration 1 (the comments of _capture_instance and _command_id), control migration 1, and the silver table kind with no migrations yet (ADR 0013).

Added

  • Streaming source format("mssql_cdc") on the Python DataSource V2 API, registered with register(spark) (ADR 0001). Offsets are the last processed commit LSN as hex plus its commit time (ADR 0002); every batch is a prefix of the commit history and ends on a commit boundary.
  • Trigger.AvailableNow and admission control (maxCommitsPerBatch) on Spark 4.2, and on DBR 18.2 through its backport; a legacy reader for older Spark.
  • Reads of the change table cdc.<ci>_CT itself, for __$command_id, with only SELECT on it beyond what the CDC functions need (ADR 0009).
  • Retention guard on the driver and again on the executors after each read: DataLossError unless failOnDataLoss=false.
  • Captured columns inferred with sys.sp_cdc_get_captured_columns, columns to override (ADR 0007).
  • sourceTimeZone=auto: the server's time zone from CURRENT_TIMEZONE_ID(), or its current UTC offset before SQL Server 2022 (ADR 0008); every time column is a TIMESTAMP_NTZ in UTC.
  • numPartitions=auto: the compute's cores (ADR 0011), with commit-aligned ranges balanced by the change table's rows (ADR 0015).
  • Drivers: mssql-python by default, installed with the package and fetching straight into Arrow, and arrow-odbc (untested, the [arrow-odbc] extra) (ADR 0003); a file-backed fake (backend=fake) for engine tests (ADR 0006). mssql-python is imported only when it opens a connection, so the other backends work without its system libraries.
  • Capture instances match ignoring case, as SQL Server's default collation does: the exact name first, and two names that differ only in case raise an error naming both. _capture_instance keeps the name as the options give it (ADR 0016, amendment 2).
  • Delta sink sink.delta_sink(): idempotent appends (txnAppId/txnVersion) and per-batch facts (row counts, LSN and commit-time ranges, timings) in the commit's userMetadata and a facts table.
  • Network and read metrics per partition in the facts, folded from metricsPath, one directory per stream (stream() puts an explicit one's files under <metricsPath>/<app_id>) (ADR 0014), and the retention headroom (ADR 0017).
  • Capture lag and ingestion lag in the facts (source_max_commit_ts, capture_lag_seconds, ingestion_lag_seconds), and max_lsn with its commit time in the query progress on every trigger (reportLatestOffset) (ADR 0020).
  • The batch's end offset in the facts (end_lsn, end_commit_ts, facts migration 5): the retention headroom and the ingestion lag are measured from it, not from the batch's last change. Every micro-batch writes a facts row, those that read no rows included (rows = 0), so facts stop arriving only when the stream or CDC capture stops (ADR 0014 amendment 3, ADRs 0017 and 0020 amended).
  • Completeness signal finalized_until in a control table, advanced after the data and never backwards: finalization.candidate(), advance(), is_final(), end_offset_from_progress() (ADR 0004, ADR 0005).
  • Control, facts and bronze tables created typed and commented through the DeltaTable API (ADR 0012), with append-only schema migrations per table kind (ADR 0013).
  • stream(spark, options).to_delta(...): source and sink from one set of options.
  • Bootstrap snapshot stamped with the max_lsn recorded before the read: stream(...).snapshot(target), to_delta(bootstrap=True) and format("mssql_cdc_snapshot") (ADR 0016). Its partitions are uniform ranges of a single integer key, else NTILE tiles of the rows by the whole key, for composite and non-integer keys (ADR 0016 amendment).
  • Automatic re-snapshot after CDC data loss, to_delta(on_data_loss="resnapshot"), into a new checkpoint generation, with the loss recorded in the facts (ADR 0018).
  • Silver helper apply_changes: keeps a current-state table equal to the source from the bronze change log with MERGE, rebuilds it from a newer snapshot (bootstrap or re-snapshot), keeps its position in the control table (applied_lsn, snapshot_lsn, control migration 1) and propagates finalized_until (ADR 0019).
  • Schema changes on the source and a second capture instance (ADR 0023): every batch checks the DDL SQL Server recorded inside it (sys.sp_cdc_get_ddl_history); a captured column whose new type the query's no longer holds fails the batch before it reads (SchemaChangedError, exported), other DDL goes on with a schema_change facts row, and schemaChangePolicy=fail fails on any DDL. The stream follows a newer capture instance of its table at that instance's start LSN, within one batch, and writes a capture_instance_switched facts row; the schema is the union of both instances' columns. Bronze appends use mergeSchema, and a type bronze cannot take fails with a message about delta.enableTypeWidening. to_delta(snapshot_on_switch=True) snapshots the table after the switch. Snapshots read NULL for a dropped column and are found under any instance of the table; a DataLossError caused by an old instance disabled too early says so, and a PermissionError names the new change table's grant. sql/switch_capture_instance.sql documents the DBA's procedure.
  • sql/heartbeat.sql, an optional Agent job that keeps max_lsn moving on a quiet database (ADR 0010).
  • The lab: a Faker OLTP workload and checks t1-t8 against SQL Server 2022, Spark and Delta (LAB.md).
  • Inline type hints (py.typed).
  • import mssql_cdc without PySpark raises an ImportError that says to run on a Spark platform, which ships its own, or to install the [spark] extra.