Appending to and overwriting a Delta Lake table
This section explains how to append to an exising Delta table and how to overwrite a Delta table.
Delta Lake append transactions
Suppose you have a Delta table with the following contents:
+-------+----------+
| num | letter |
|-------+----------|
| 1 | a |
| 2 | b |
| 3 | c |
+-------+----------+
Append two additional rows of data to the table:
let table = open_table("tmp/some-table").await?;
DeltaOps(table).write(RecordBatch::try_new(
Arc::new(Schema::new(vec![
Field::new("num", DataType::Int32, false),
Field::new("letter", DataType::Utf8, false),
])),
vec![
Arc::new(Int32Array::from(vec![8, 9])),
Arc::new(StringArray::from(vec![
"dd", "ee"
])),
])).with_save_mode(SaveMode::Append).await?;
Here are the updated contents of the Delta table:
+-------+----------+
| num | letter |
|-------+----------|
| 1 | a |
| 2 | b |
| 3 | c |
| 8 | dd |
| 9 | ee |
+-------+----------+
Now let's see how to perform an overwrite transaction.
Delta Lake overwrite transactions
Now let's see how to overwrite the exisitng Delta table.
let table = open_table("tmp/some-table").await?;
DeltaOps(table).write(RecordBatch::try_new(
Arc::new(Schema::new(vec![
Field::new("num", DataType::Int32, false),
Field::new("letter", DataType::Utf8, false),
])),
vec![
Arc::new(Int32Array::from(vec![1, 2, 3])),
Arc::new(StringArray::from(vec![
"a", "b", "c",
])),
])).with_save_mode(SaveMode::Overwrite).await?;
Here are the contents of the Delta table after the overwrite operation:
+-------+----------+
| num | letter |
|-------+----------|
| 11 | aa |
| 22 | bb |
+-------+----------+
Overwriting just performs a logical delete. It doesn't physically remove the previous data from storage. Time travel back to the previous version to confirm that the old version of the table is still accessable.