io.delta.standalone

Interface OptimisticTransaction

public interface OptimisticTransaction

Used to perform a set of reads in a transaction and then commit a set of updates to the state of the log. All reads from the DeltaLog MUST go through this instance rather than directly to the DeltaLog otherwise they will not be checked for logical conflicts with concurrent updates.

This class is not thread-safe.

Method Summary

Modifier and Type	Method and Description
<T extends Action> CommitResult	commit(Iterable<T> actions, Operation op, String engineInfo) Modifies the state of the log by adding a new commit that is based on a read at the table's latest version as of this transaction's instantiation.
DeltaScan	markFilesAsRead(Expression readPredicate) Mark files matched by the readPredicate as read by this transaction.
Metadata	metadata()
long	readVersion()
void	readWholeTable() Mark the entire table as tainted (i.e.
long	txnVersion(String id)
void	updateMetadata(Metadata metadata) Records an update to the metadata that should be committed with this transaction.

Method Detail

commit

<T extends Action> CommitResult commit(Iterable<T> actions,
                                       Operation op,
                                       String engineInfo)

Modifies the state of the log by adding a new commit that is based on a read at the table's latest version as of this transaction's instantiation. In the case of a conflict with a concurrent writer this method will throw an exception.

Note: any AddFile with an absolute path within the table path will be updated to have a relative path (based off of the table path). Because of this, be sure to generate all RemoveFiles using AddFiles read from the Delta Log (do not use the AddFiles created pre-commit.)

Type Parameters:

T - A derived class of Action. This allows, for example, both a List<Action> and a List<AddFile> to be accepted.

Parameters:

actions - Set of actions to commit.

op - Details of operation that is performing this transactional commit.

engineInfo - String used to identify the writer engine. It should resemble "{engineName}/{engineVersion}", with dashes in place of whitespace. For example, "Flink-Connector/1.1.0".

Returns:

a CommitResult, wrapping the table version that was committed.

markFilesAsRead

DeltaScan markFilesAsRead(Expression readPredicate)

Mark files matched by the readPredicate as read by this transaction.

Please note filtering is only supported on partition columns, thus the files matched may be a superset of the files in the Delta table that satisfy readPredicate. Users should use DeltaScan.getResidualPredicate() to check for any unapplied portion of the input predicate.

Internally, readPredicate and the matched readFiles will be used to determine if logical conflicts between this transaction and previously-committed transactions can be resolved (i.e. no error thrown).

For example:

• This transaction TXN1 reads partition 'date=2021-09-08' to perform an UPDATE and tries to commit at the next table version N.
• After TXN1 starts, another transaction TXN2 reads partition 'date=2021-09-07' and commits first at table version N (with no other metadata changes).
• TXN1 sees that another commit won, and needs to know whether to commit at version N+1 or fail. Using the readPredicates and resultant readFiles, TXN1 can see that none of its read files were changed by TXN2. Thus there are no logical conflicts and TXN1 can commit at table version N+1.

Parameters:

readPredicate - Predicate used to determine which files were read.

Returns:

a DeltaScan containing the list of files matching the pushed portion of the readPredicate.

updateMetadata

void updateMetadata(Metadata metadata)

Records an update to the metadata that should be committed with this transaction.

Use Metadata.copyBuilder() to build a new Metadata instance based on the current table metadata. For example:

 Metadata newMetadata = optimisticTransaction.metadata().copyBuilder()
     .schema(newSchema)
     .build();
 optimisticTransaction.updateMetadata(newMetadata);
 

IMPORTANT: It is the responsibility of the caller to ensure that files currently present in the table are still valid under the new metadata.

Parameters:

metadata - The new metadata for the delta table.

readWholeTable

void readWholeTable()

Mark the entire table as tainted (i.e. read) by this transaction.

txnVersion

long txnVersion(String id)

Parameters:

id - transaction id

Returns:

the latest version that has committed for the idempotent transaction with given id.

metadata

Metadata metadata()

Returns:

the metadata for this transaction. The metadata refers to the metadata of the table's latest version as of this transaction's instantiation unless updated during the transaction.

readVersion

long readVersion()

Returns:

The table version that this transaction is reading from.