API reference

CloudStore.Object(store, key; credentials=nothing, kwargs...)

A remote object in an S3 bucket or Azure Blob container.

Constructing an Object from a store and key sends a metadata request. Upload and list operations also return Object values. The fields include store, credentials, key, size, eTag, and provider-specific properties.

PrefetchedDownloadStream{T <: Object} <: IO
PrefetchedDownloadStream(args...; kwargs...) -> PrefetchedDownloadStream{T <: Object}

A buffered, read-only, in-memory IO stream that fetches chunks from remote cloud Object.

Data is downloaded to two internal buffers. Once you start reading from the first buffer, a secondary buffer will begin to be "prefetched" by multiple background tasks. Once you read past the first buffer, the two buffers are switched and new round of prefetching begins.

To control memory usage and speed, the user can change two parameters when constructing the stream: prefetch_multipart_size and prefetch_size. prefetch_multipart_size is the max size of any individual GET request in bytes (default 8.000 MiB), prefetch_size is the size of a buffer that stores the fetched bytes and which is iterated when we consume/read the IO (default 32.000 MiB).

The number of spawned tasks is also governed by these two parameters, with approximately prefetch_size / prefetch_multipart_size tasks spawned for performing the GET requests (defaults to 4 if those fields aren't specified) + 1 task is spawned to coordinate the prefetching process. Number of spawned tasks is upper-bounded by the size of the input and the number of threads available (see the internal _ndownload_tasks helper function).

Reading from this stream is not thread-safe.

Arguments

• store::AbstractStore: The S3 Bucket / Azure Container object
• key::String: S3 key / Azure blob resource name
• prefetch_size::Int=DEFAULT_PREFETCH_SIZE: The size of each of the two internal prefetch buffers in bytes

Keywords

• credentials::Union{CloudCredentials, Nothing}=nothing: Credentials object used in HTTP requests
• prefetch_multipart_size::Int=DEFAULT_PREFETCH_MULTIPART_SIZE: The size of each individual GET request in bytes
• kwargs...: HTTP keyword arguments are forwarded to underlying HTTP requests,

Examples

# Get an IO stream for a remote CSV file `test.csv` living in your S3 bucket
io = PrefetchedDownloadStream(my_bucket, "test.csv"; credentials)

# Integrates with TranscodingStreams; HTTP keyword arguments are forwarded to underlying HTTP requests
using CodecZlib
io = GzipDecompressorStream(
    PrefetchedDownloadStream(my_bucket, "test.csv.gz"; credentials, retries=5)
)

# Up to 8 concurrent download tasks, each fetching 2MiB into 16MiB prefetch buffer.
io = PrefetchedDownloadStream(
    my_bucket, "test.csv", 16*1024*1024; credentials, prefetch_multipart_size=2*1024*1024)
)

This is an experimental API.

MultipartUploadStream <: IO
MultipartUploadStream(args...; kwargs...) -> MultipartUploadStream

An in-memory IO stream that uploads chunks to a URL in blob storage.

For every data chunk we call write(io, data;) to write it to a channel. We spawn one task per chunk to read data from this channel and uploads it as a distinct part to blob storage to the same remote object. We expect the chunks to be written in order. For cases where there is no need to upload data in parts or the data size is too small, put can be used instead.

Arguments

• store::AbstractStore: The S3 Bucket / Azure Container object
• key::String: S3 key / Azure blob resource name

Keywords

• credentials::Union{CloudCredentials, Nothing}=nothing: Credentials object used in HTTP requests
• concurrent_writes_to_channel::Int=(4 * Threads.nthreads()): represents the max number of chunks in flight. Defaults to 4 times the number of threads. We use this value to initialize a semaphore to perform throttling in case the writing to the channel is much faster to uploading to blob storage, i.e. write will block as a result of this limit being reached.
• kwargs...: HTTP keyword arguments are forwarded to underlying HTTP requests,

Examples

# Get an IO stream for a remote CSV file `test.csv` living in your S3 bucket
io = MultipartUploadStream(bucket, "test.csv"; credentials)

# Write a chunk of data (Vector{UInt8}) to the stream
write(io, data;)

# Wait for all chunks to be uploaded
wait(io)

# Close the stream
close(io; credentials)

# Alternative syntax that encapsulates all these steps
MultipartUploadStream(bucket, "test.csv"; credentials) do io
    write(io, data;)
end

Note on upload size

Some cloud storage providers might have a lower limit on the size of the uploaded object.
For example it seems that S3 requires at minimum an upload of 5MB:
https://github.com/minio/minio/issues/11076.
We haven't found a similar setting for Azure.
For such cases where the size of the data is too small, one can use `put`

CloudStore.abort(io::MultipartUploadStream; kwargs...)

Wait for active part requests, then abort the multipart upload. S3 removes the uploaded parts immediately. Azure has no matching abort request and removes uncommitted blocks after its service retention period.