API reference

CloudStore.Object(store, key; credentials=nothing, kwargs...)

A remote object in an S3 bucket or Azure Blob container.

Constructing an Object from a store and key sends a metadata request. Upload and list operations also return Object values. The fields include store, credentials, key, size, eTag, and provider-specific properties.

copyto! and PrefetchedDownloadStream use the size and strong ETag to validate each range. If the ETag is missing, they refresh metadata before reading.

CloudStore.ObjectBytes(object::CloudStore.Object; blocksize=64 * 1024, kwargs...)

A read-only AbstractVector{UInt8} over one version of a remote object. Indexing, view, and copyto! can read selected bytes without downloading the whole object. Range requests use the object's size and strong ETag; changed, short, or malformed responses throw. Missing ETag metadata is refreshed once when constructing the array. The array contains stored bytes, without decompression.

Scalar reads share one buffer of at most blocksize bytes. Contiguous views share that buffer and the same object version. Bulk copyto! fetches the requested range unless it is already buffered. Contiguous strided destinations receive directly; other destination types may use a temporary vector. A failed copy may leave partial bytes in the destination. copy and ordinary range indexing return owned vectors. Materializing the entire array requires object-sized output memory.

Reads from multiple tasks are supported: scalar reads serialize while filling the shared buffer, and independent bulk copies can run concurrently. The array owns no open connection and does not require closing. Retained views keep the shared buffer alive. Additional keywords are forwarded to each range request; do not mutate any objects passed as keyword arguments while the array is in use.

PrefetchedDownloadStream(object::CloudStore.Object, prefetch_size=32 * 1024^2; prefetch_multipart_size=8 * 1024^2, prefetch_concurrency=Threads.nthreads(), kwargs...)
PrefetchedDownloadStream(store, key, prefetch_size=32 * 1024^2; credentials=nothing, prefetch_multipart_size=8 * 1024^2, prefetch_concurrency=Threads.nthreads(), kwargs...)

A read-only IO over a remote object that downloads ahead of the reader. Up to two buffers of prefetch_size bytes alternate: while you read one, background tasks fill the other with range requests of at most prefetch_multipart_size bytes. prefetch_concurrency is a positive integer limiting concurrent range requests; it defaults to Threads.nthreads() and is capped by the number of ranges in one buffer. Raising it can overlap more network waits without increasing the buffer sizes, but may use more connections and HTTP/TLS memory. Increasing prefetch_size allows more ranges per buffer and uses more memory.

Every range must match the object's size and strong ETag, so a changed object raises an error. The stream returns the stored bytes; wrap it in a decompressor such as CodecZlib.GzipDecompressorStream for gzip objects. Do not read from several tasks at once. Other keyword arguments are passed to the HTTP requests.

io = PrefetchedDownloadStream(bucket, "data.csv.gz"; credentials)
text = read(GzipDecompressorStream(io), String)

MultipartUploadStream(store, key; credentials=nothing, concurrent_writes_to_channel=4 * Threads.nthreads(), kwargs...)
MultipartUploadStream(f, store, key; kwargs...)

An IO that uploads each write(io, bytes::Vector{UInt8}) as the next part of a multipart upload. Parts upload in background tasks; when concurrent_writes_to_channel parts are in flight, write waits. Do not modify bytes after passing it to write.

close(io) waits for every part, completes the upload, and returns the object's ETag. CloudStore.abort(io) discards the upload instead. If a part or the completion fails, close aborts the upload and rethrows. The do-block form calls close after f(io) returns, or aborts if f throws.

headers and contentType describe the object, as for CloudStore.put, and apply to close too: write conditions such as If-None-Match: * and If-Match are checked when the upload completes. On S3 every part except the last must be at least 5 MiB; use CloudStore.put for smaller objects. Other keyword arguments are passed to the HTTP requests. This API is experimental.

CloudStore.abort(io::MultipartUploadStream; kwargs...)

Wait for active part requests, then abort the multipart upload. S3 removes the uploaded parts immediately. Azure has no matching abort request and removes uncommitted blocks after its service retention period.