Schema
Schema and field
Schemas, fields, and data types are provided in the deltalake.schema
submodule.
deltalake.Schema
Bases: deltalake._internal.StructType
A Delta Lake schema
Create using a list of :class:Field
:
Schema([Field("x", "integer"), Field("y", "string")]) Schema([Field(x, PrimitiveType("integer"), nullable=True), Field(y, PrimitiveType("string"), nullable=True)])
Or create from a PyArrow schema:
from arro3.core import DateType, Schema as ArrowSchema Schema.from_pyarrow(ArrowSchema({"x": DateType.int32(), "y": DateType.string()})) Schema([Field(x, PrimitiveType("integer"), nullable=True), Field(y, PrimitiveType("string"), nullable=True)])
invariants
invariants: list[tuple[str, str]] = <attribute 'invariants' of 'deltalake._internal.Schema' objects>
from_arrow
staticmethod
Create a Schema from a schema that implements Arrow C Data Interface.
Will raise TypeError
if one of the Arrow type is not a primitive type.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
type |
ArrowSchemaExportable
|
an object that is |
required |
Returns:
Type | Description |
---|---|
Schema
|
a Schema |
from_json
staticmethod
Create a new Schema from a JSON string.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
json |
str
|
a JSON string |
required |
to_arrow
method descriptor
Return equivalent arro3 schema
Parameters:
Name | Type | Description | Default |
---|---|---|---|
as_large_types |
bool
|
get schema with all variable size types (list, binary, string) as large variants (with int64 indices). This is for compatibility with systems like Polars that only support the large versions of Arrow types. |
False
|
Returns:
Type | Description |
---|---|
Schema
|
an arro3 Schema |
to_json
method descriptor
deltalake.Field
from_arrow
staticmethod
Create a Field from an object with an ArrowSchemaExportable
field
Note: This currently doesn't preserve field metadata.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
field |
ArrowSchemaExportable
|
a Field object that is |
required |
Returns:
Type | Description |
---|---|
Field
|
a Field |
from_json
staticmethod
Create a Field from a JSON string.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
json |
str
|
the JSON string. |
required |
Returns:
Type | Description |
---|---|
Field
|
Field |
to_arrow
method descriptor
Convert to an equivalent arro3 field Note: This currently doesn't preserve field metadata.
Returns:
Type | Description |
---|---|
Field
|
a arro3 Field |
Data types
deltalake.schema.PrimitiveType
from_arrow
staticmethod
Create a PrimitiveType from an ArrowSchemaExportable
datatype
Will raise TypeError
if the arrow type is not a primitive type.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
type |
ArrowSchemaExportable
|
an object that is |
required |
Returns:
Type | Description |
---|---|
PrimitiveType
|
a PrimitiveType |
from_json
staticmethod
Create a PrimitiveType from a JSON string
The JSON representation for a primitive type is just a quoted string: PrimitiveType.from_json('"integer"')
Parameters:
Name | Type | Description | Default |
---|---|---|---|
json |
str
|
a JSON string |
required |
Returns:
Type | Description |
---|---|
PrimitiveType
|
a PrimitiveType type |
deltalake.schema.ArrayType
contains_null
element_type
from_arrow
staticmethod
Create an ArrayType from an ArrowSchemaExportable
datatype.
Will raise TypeError
if a different arrow DataType is provided.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
type |
ArrowSchemaExportable
|
an object that is |
required |
Returns:
Type | Description |
---|---|
ArrayType
|
an ArrayType |
from_json
staticmethod
Create an ArrayType from a JSON string
Parameters:
Name | Type | Description | Default |
---|---|---|---|
json |
str
|
a JSON string |
required |
Returns:
Type | Description |
---|---|
ArrayType
|
an ArrayType |
deltalake.schema.MapType
value_contains_null
value_contains_null: bool = <attribute 'value_contains_null' of 'deltalake._internal.MapType' objects>
from_arrow
staticmethod
Create a MapType from an ArrowSchemaExportable
datatype
Will raise TypeError
if passed a different type.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
type |
ArrowSchemaExportable
|
an object that is |
required |
Returns:
Type | Description |
---|---|
MapType
|
a MapType |
from_json
staticmethod
Create a MapType from a JSON string
Parameters:
Name | Type | Description | Default |
---|---|---|---|
json |
str
|
a JSON string |
required |
Returns:
Type | Description |
---|---|
MapType
|
an ArrayType |
Example
The JSON representation for a map type is an object with type
(set to map
),
keyType
, valueType
, and valueContainsNull
:
deltalake.schema.StructType
type
The string "struct"
from_arrow
staticmethod
Create a new StructType from an ArrowSchemaExportable
datatype
Will raise TypeError
if a different data type is provided.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
type |
ArrowSchemaExportable
|
a struct type object that is |
required |
Returns:
Type | Description |
---|---|
StructType
|
a StructType |
from_json
staticmethod
Create a new StructType from a JSON string.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
json |
str
|
a JSON string |
required |
Returns:
Type | Description |
---|---|
StructType
|
a StructType |
to_arrow
method descriptor
Get the equivalent arro3 DataType (arro3.core.DataType)