Storage

The delta filesystem handler for the pyarrow engine writer.

deltalake.fs.DeltaStorageHandler

DeltaStorageHandler(table_uri: str, options: Optional[Dict[str, str]] = None, known_sizes: Optional[Dict[str, int]] = None)

Bases: FileSystemHandler

DeltaStorageHandler is a concrete implementations of a PyArrow FileSystemHandler.

copy_file

copy_file(src: str, dst: str) -> None

Copy a file.

If the destination exists and is a directory, an error is returned. Otherwise, it is replaced.

create_dir

create_dir(path: str, recursive: bool = True) -> None

Create a directory and subdirectories.

This function succeeds if the directory already exists.

delete_dir

delete_dir(path: str) -> None

Delete a directory and its contents, recursively.

delete_dir_contents

delete_dir_contents(path: str, *, accept_root_dir: bool = False, missing_dir_ok: bool = False) -> None

Delete a directory's contents, recursively.

Like delete_dir, but doesn't delete the directory itself.

delete_file

delete_file(path: str) -> None

Delete a file.

delete_root_dir_contents

delete_root_dir_contents() -> None

Delete the root directory contents, recursively.

get_file_info

get_file_info(paths: List[str]) -> List[FileInfo]

Get info for the given files.

A non-existing or unreachable file returns a FileStat object and has a FileType of value NotFound. An exception indicates a truly exceptional condition (low-level I/O error, etc.).

get_file_info_selector

get_file_info_selector(selector: FileSelector) -> List[FileInfo]

Get info for the files defined by FileSelector.

Parameters:

Name	Type	Description	Default
selector	FileSelector	FileSelector object	required

Returns:

Type	Description
List[FileInfo]	list of file info objects

move

move(src: str, dest: str) -> None

Move / rename a file or directory.

If the destination exists: - if it is a non-empty directory, an error is returned - otherwise, if it has the same type as the source, it is replaced - otherwise, behavior is unspecified (implementation-dependent).

normalize_path

normalize_path(path: str) -> str

Normalize filesystem path.

open_input_file

open_input_file(path: str) -> pa.PythonFile

Open an input file for random access reading.

Parameters:

Name	Type	Description	Default
path	str	The source to open for reading.	required

Returns:

Type	Description
PythonFile	NativeFile

open_input_stream

open_input_stream(path: str) -> pa.PythonFile

Open an input stream for sequential reading.

Parameters:

Name	Type	Description	Default
path	str	The source to open for reading.	required

Returns:

Type	Description
PythonFile	NativeFile

open_output_stream

open_output_stream(path: str, metadata: Optional[Dict[str, str]] = None) -> pa.PythonFile

Open an output stream for sequential writing.

If the target already exists, existing data is truncated.

Parameters:

Name	Type	Description	Default
path	str	The source to open for writing.	required
metadata	Optional[Dict[str, str]]	If not None, a mapping of string keys to string values.	None

Returns:

Type	Description
PythonFile	NativeFile