roboto.fs.object_store.s3#

Module Contents#

class roboto.fs.object_store.s3.ProgressCallbackInvoker(callback)#

Bases: boto3.s3.transfer.BaseSubscriber

Invoke a provided callback via a subscriber.

Taken from boto/boto3

on_progress(bytes_transferred, **kwargs)#

Callback to be invoked when progress is made on transfer

This callback can be useful for:

  • Recording and displaying progress

Parameters:

  • future (s3transfer.futures.TransferFuture) – The TransferFuture representing the requested transfer.

  • bytes_transferred (int) – The number of bytes transferred for that invocation of the callback. Note that a negative amount can be provided, which usually indicates that an in-progress request needed to be retried and thus progress was rewound.

class roboto.fs.object_store.s3.S3Store(s3_client, transfer_config=None)#

Bases: roboto.fs.object_store.object_store.ObjectStore

Base class for protocol classes.

Protocol classes are defined as:

class Proto(Protocol):
    def meth(self) -> int:
        ...

Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing).

For example:

class C:
    def meth(self) -> int:
        return 0

def func(x: Proto) -> int:
    return x.meth()

func(C())  # Passes static type check

See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as:

class GenProto[T](Protocol):
    def meth(self) -> T:
        ...
Parameters:

  • s3_client (botocore.client.BaseClient)

  • transfer_config (Optional[boto3.s3.transfer.TransferConfig])

classmethod create(credential_provider, **kwargs)#

Factory function to assemble an S3Store with refreshable credentials.

Parameters:

credential_provider (roboto.fs.object_store.object_store.CredentialProvider)

Return type:

S3Store

get(source_uri, destination, on_progress=None)#

Downloads a file from a cloud URI to a local path.

Parent directories of the destination path are created automatically if they don’t exist.

Parameters:

  • source_uri (str) – Full URI (e.g., ‘s3://my-bucket/folder/data.csv’)

  • destination (pathlib.Path) – Local path where the file should be saved.

  • on_progress (Optional[roboto.fs.object_store.object_store.OnProgress]) – Optional callback to be periodically called with the number of bytes downloaded.

Return type:

roboto.fs.object_store.object_store.FutureLike[None]

put(source, destination_uri, on_progress=None)#

Uploads a local file to a specific cloud URI.

Parameters:

  • source (pathlib.Path) – Local path to the file.

  • destination_uri (str) – Full URI (e.g., ‘s3://my-bucket/folder/data.csv’)

  • on_progress (Optional[roboto.fs.object_store.object_store.OnProgress]) – Optional callback to be periodically called with the number of bytes uploaded.

Return type:

roboto.fs.object_store.object_store.FutureLike[None]

On this page