TableOptimizer

deltalake.table.TableOptimizer

TableOptimizer(table: DeltaTable)

API for table optimization commands.

compact

compact(partition_filters: FilterConjunctionType | None = None, target_size: int | None = None, max_concurrent_tasks: int | None = None, max_spill_size: int | None = None, max_temp_directory_size: int | None = None, min_commit_interval: int | timedelta | None = None, writer_properties: WriterProperties | None = None, *args: Any, commit_properties: CommitProperties | None = None, post_commithook_properties: PostCommitHookProperties | None = None) -> dict[str, Any]

Compacts small files to reduce read overhead.

This operation is eventually idempotent; repeated runs on an unchanged table converge to a stable layout, but a rewritten file that lands under the target size can still be merged again by a later run.

Compaction keeps file order within each partition.

The target size is approximate.

If this operation runs with any operation other than append, it fails.

Parameters:

Name	Type	Description	Default
partition_filters	FilterConjunctionType | None	partition filters used to match files	None
target_size	int | None	approximate target file size in bytes. If not provided, this uses delta.targetFileSize or 100MB.	None
max_concurrent_tasks	int | None	the maximum number of concurrent tasks to use for file compaction. Defaults to number of CPUs. More concurrent tasks can make compaction faster, but will also use more memory.	None
max_spill_size	int | None	the maximum number of bytes allowed in memory before spilling to disk. If not specified, uses DataFusion's default.	None
max_temp_directory_size	int | None	the maximum disk space for temporary spill files. If not specified, uses DataFusion's default.	None
min_commit_interval	int | timedelta | None	minimum interval in seconds or as timedeltas before a new commit is created. Interval is useful for long running executions. Set to 0 or timedelta(0), if you want a commit per partition.	None
writer_properties	WriterProperties | None	Pass writer properties to the Rust parquet writer.	None
commit_properties	CommitProperties | None	properties of the transaction commit. If None, default values are used.	None
post_commithook_properties	PostCommitHookProperties | None	properties for the post commit hook. If None, default values are used.	None

Returns:

Type	Description
dict[str, Any]	optimize metrics

Example

Use a timedelta object to specify the seconds, minutes or hours of the interval.

from deltalake import DeltaTable, write_deltalake
from datetime import timedelta
import pyarrow as pa

write_deltalake("tmp", pa.table({"x": [1], "y": [4]}))
write_deltalake("tmp", pa.table({"x": [2], "y": [5]}), mode="append")

dt = DeltaTable("tmp")
time_delta = timedelta(minutes=10)
dt.optimize.compact(min_commit_interval=time_delta)
{'numFilesAdded': 1, 'numFilesRemoved': 2, 'filesAdded': ..., 'filesRemoved': ..., 'partitionsOptimized': 1, 'numBatches': 2, 'totalConsideredFiles': 2, 'totalFilesSkipped': 0, 'plannerStrategy': 'preserveLocality', 'preservedStableOrder': True, 'preserveInsertionOrder': True, 'maxBinSpanFiles': 2}

z_order

z_order(columns: Iterable[str], partition_filters: FilterConjunctionType | None = None, target_size: int | None = None, max_concurrent_tasks: int | None = None, max_spill_size: int | None = None, max_temp_directory_size: int | None = None, min_commit_interval: int | timedelta | None = None, writer_properties: WriterProperties | None = None, *args: Any, commit_properties: CommitProperties | None = None, post_commithook_properties: PostCommitHookProperties | None = None) -> dict[str, Any]

Reorders the data using a Z-order curve to improve data skipping.

This also performs compaction, so the same parameters as compact() apply. Z order rewrites file order.

Parameters:

Name	Type	Description	Default
columns	Iterable[str]	the columns to use for Z-ordering. There must be at least one column.	required
partition_filters	FilterConjunctionType | None	partition filters used to match files	None
target_size	int | None	approximate target file size in bytes. If not provided, this uses delta.targetFileSize or 100MB.	None
max_concurrent_tasks	int | None	the maximum number of concurrent tasks to use for file compaction. Defaults to number of CPUs. More concurrent tasks can make compaction faster, but will also use more memory.	None
max_spill_size	int | None	the maximum number of bytes allowed in memory before spilling to disk. If not specified, uses DataFusion's default.	None
max_temp_directory_size	int | None	the maximum disk space for temporary spill files. If not specified, uses DataFusion's default.	None
min_commit_interval	int | timedelta | None	minimum interval in seconds or as timedeltas before a new commit is created. Interval is useful for long running executions. Set to 0 or timedelta(0), if you want a commit per partition.	None
writer_properties	WriterProperties | None	Pass writer properties to the Rust parquet writer.	None
commit_properties	CommitProperties | None	properties of the transaction commit. If None, default values are used.	None
post_commithook_properties	PostCommitHookProperties | None	properties for the post commit hook. If None, default values are used.	None

Returns:

Type	Description
dict[str, Any]	optimize metrics

Example

Use a timedelta object to specify the seconds, minutes or hours of the interval.

from deltalake import DeltaTable, write_deltalake
from datetime import timedelta
import pyarrow as pa

write_deltalake("tmp", pa.table({"x": [1], "y": [4]}))
write_deltalake("tmp", pa.table({"x": [2], "y": [5]}), mode="append")

dt = DeltaTable("tmp")
time_delta = timedelta(minutes=10)
dt.optimize.z_order(["x"], min_commit_interval=time_delta)
{'numFilesAdded': 1, 'numFilesRemoved': 2, 'filesAdded': ..., 'filesRemoved': ..., 'partitionsOptimized': 0, 'numBatches': 1, 'totalConsideredFiles': 2, 'totalFilesSkipped': 0, 'plannerStrategy': 'zOrder', 'preservedStableOrder': False, 'preserveInsertionOrder': False, 'maxBinSpanFiles': 2}