io.delta.kernel.defaults.engine

Class DefaultFileSystemClient

Object

io.delta.kernel.defaults.engine.DefaultFileSystemClient

All Implemented Interfaces:

io.delta.kernel.engine.FileSystemClient

public class DefaultFileSystemClient
extends Object
implements io.delta.kernel.engine.FileSystemClient

Default implementation of FileSystemClient based on Hadoop APIs. It takes a Hadoop Configuration object to interact with the file system. The following optional configurations can be set to customize the behavior of the client:

• io.delta.kernel.logStore.<scheme>.impl - The class name of the custom LogStore implementation to use for operations on storage systems with the specified scheme. For example, to use a custom LogStore for S3 storage objects:

  
       <property>
         <name>io.delta.kernel.logStore.s3.impl</name>
         <value>com.example.S3LogStore</value>
       </property>
       

  If not set, the default LogStore implementation for the scheme will be used.
• delta.enableFastS3AListFrom - Set to true to enable fast listing functionality when using a LogStore created for S3 storage objects.

The above list of options is not exhaustive. For a complete list of options, refer to the specific implementation of FileSystem.

Constructor Summary

Constructors

Constructor and Description
DefaultFileSystemClient(org.apache.hadoop.conf.Configuration hadoopConf) Create an instance of the default FileSystemClient implementation.

Method Summary

Modifier and Type	Method and Description
io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.FileStatus>	listFrom(String filePath)
boolean	mkdirs(String path)
io.delta.kernel.utils.CloseableIterator<java.io.ByteArrayInputStream>	readFiles(io.delta.kernel.utils.CloseableIterator<io.delta.kernel.engine.FileReadRequest> readRequests)
String	resolvePath(String path)

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DefaultFileSystemClient

public DefaultFileSystemClient(org.apache.hadoop.conf.Configuration hadoopConf)

Create an instance of the default FileSystemClient implementation.

Parameters:

hadoopConf - Configuration to use. List of options to customize the behavior of the client can be found in the class documentation.

Method Detail

listFrom

public io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.FileStatus> listFrom(String filePath)
                                                                                   throws java.io.IOException

Specified by:

listFrom in interface io.delta.kernel.engine.FileSystemClient

Throws:

java.io.IOException

resolvePath

public String resolvePath(String path)
                   throws java.io.IOException

Specified by:

resolvePath in interface io.delta.kernel.engine.FileSystemClient

Throws:

java.io.IOException

readFiles

public io.delta.kernel.utils.CloseableIterator<java.io.ByteArrayInputStream> readFiles(io.delta.kernel.utils.CloseableIterator<io.delta.kernel.engine.FileReadRequest> readRequests)

Specified by:

readFiles in interface io.delta.kernel.engine.FileSystemClient

mkdirs

public boolean mkdirs(String path)
               throws java.io.IOException

Specified by:

mkdirs in interface io.delta.kernel.engine.FileSystemClient

Throws:

java.io.IOException