roboto.domain.files.file_service#

Module Contents#

class roboto.domain.files.file_service.DynamicCallbackSubscriber(on_done_cb=None, on_progress_cb=None, on_queued_cb=None)#

Bases: boto3.s3.transfer.BaseSubscriber

The base subscriber class

It is recommended that all subscriber implementations subclass and then override the subscription methods (i.e. on_{subsribe_type}() methods).

Parameters:

  • on_done_cb (Optional[Callable[[], None]])

  • on_progress_cb (Optional[Callable[[Any], None]])

  • on_queued_cb (Optional[Callable[[Any], None]])

on_done(future, **kwargs)#

Callback to be invoked once a transfer is done

This callback can be useful for:

  • Recording and displaying whether the transfer succeeded or failed using future.result()

  • Running some task after the transfer completed like changing the last modified time of a downloaded file.

Parameters:

future (s3transfer.futures.TransferFuture) – The TransferFuture representing the requested transfer.

on_progress(future, bytes_transferred, **kwargs)#

Callback to be invoked when progress is made on transfer

This callback can be useful for:

  • Recording and displaying progress

Parameters:

  • future (s3transfer.futures.TransferFuture) – The TransferFuture representing the requested transfer.

  • bytes_transferred (int) – The number of bytes transferred for that invocation of the callback. Note that a negative amount can be provided, which usually indicates that an in-progress request needed to be retried and thus progress was rewound.

on_queued(future, **kwargs)#

Callback to be invoked when transfer request gets queued

This callback can be useful for:

  • Keeping track of how many transfers have been requested

  • Providing the expected transfer size through future.meta.provide_transfer_size() so a HeadObject would not need to be made for copies and downloads.

Parameters:

future (s3transfer.futures.TransferFuture) – The TransferFuture representing the requested transfer.

class roboto.domain.files.file_service.FileService(roboto_client)#
Parameters:

roboto_client (roboto.http.RobotoClient)

download_files(file_generator, credential_provider, progress_monitor_factory=NoopProgressMonitorFactory(), max_concurrency=8)#
Parameters:
Return type:

None

upload_file(local_path, bucket, key, credential_provider, tags=None, progress_monitor_factory=NoopProgressMonitorFactory())#
Parameters:
Return type:

None

upload_files(bucket, file_generator, credential_provider, tags=None, progress_monitor_factory=NoopProgressMonitorFactory(), max_concurrency=20)#
Parameters:
Return type:

None

upload_many_files(file_generator, credential_provider, on_file_complete=None, tags=None, progress_monitor=NoopProgressMonitor(), max_concurrency=20)#
Parameters:
roboto.domain.files.file_service.MANY_FILES = 100#
roboto.domain.files.file_service.logger#
On this page