Object operations

CloudStore accepts three forms for most object operations:

a store handle and object key;
a CloudStore.Object;
an S3 or Azure object URL.

The examples below use the generic CloudStore interface. The CloudStore.S3 and CloudStore.Blobs namespaces provide matching methods.

Upload

Upload bytes, text, an open IO, or a file path:

obj = CloudStore.put(bucket, "reports/today.csv", codeunits("a,b\n1,2\n"))
obj = CloudStore.put(bucket, "reports/today.csv", IOBuffer(data))
obj = CloudStore.put(bucket, "reports/today.csv", "/tmp/today.csv")

The return value is an Object. It contains the store, key, byte size, and ETag.

CloudStore starts a multipart upload above multipartThreshold. Control the transfer with partSize and batchSize:

obj = CloudStore.put(
    bucket,
    "large.bin",
    data;
    multipartThreshold=8 * 1024^2,
    partSize=8 * 1024^2,
    batchSize=8,
)

Set allowMultipart=false to force one request. Set compress=true to gzip the uploaded bytes. CloudStore does not add .gz to the key, and the returned Object has the compressed size.

When allowMultipart=true, partSize and batchSize must be positive, including for small uploads. CloudStore checks them before reading the input or starting a request. With allowMultipart=false, these multipart size options are ignored.

Download

Download into a new byte vector:

data = CloudStore.get(bucket, "reports/today.csv")

Download into an existing destination:

buffer = Vector{UInt8}(undef, expected_size)
CloudStore.get(bucket, "large.bin", buffer)

CloudStore.get(bucket, "large.bin", "/tmp/large.bin")

open("/tmp/large.bin", "w") do io
    CloudStore.get(bucket, "large.bin", io)
end

A download first requests the object's first partSize bytes as a range (at most the destination buffer's length; 256 KiB, or partSize if smaller, for a file or stream destination). The response gives the object's size and ETag, so a smaller object needs no other request; the rest of a larger object downloads in parallel parts of partSize bytes, which start as soon as the first response's headers arrive when the result is a byte vector. Up to batchSize workers fetch these later parts; each starts its next range as soon as the preceding range finishes. A slow range does not hold up the other workers. Without a destination buffer, HTTP grows the first part's buffer as its bytes arrive, so passing a buffer avoids that copy. A destination buffer shorter than multipartThreshold, or objectMaxSize below it, makes a single GET without a range instead. Set decompress=true to gunzip the downloaded bytes.

Each later range response must match the first one: the requested bytes, the object size, and the strong ETag. If the object changes during the download or a response is incomplete, get throws instead of returning mixed or partial data. The destination may already hold some of the bytes. The body of an error response, the provider's message, never reaches the destination. S3 and Azure Blob Storage send strong ETags; for other endpoints, pass allowMultipart=false to make a single GET. Conditional headers you pass, such as If-Match, and query parameters in an object URL, such as versionId, apply to every request.

Metadata and listing

Read response headers without downloading the object:

headers = CloudStore.head(bucket, "reports/today.csv")
content_length = parse(Int, headers["Content-Length"])

List every object, or limit the result to a prefix:

objects = CloudStore.list(bucket)
reports = CloudStore.list(bucket; prefix="reports/")

CloudStore follows provider pagination until it has the complete result. maxKeys controls the page size. It does not limit the total returned object count.

Use an Object

An upload or listing returns CloudStore.Object values. You can use an object as the source for later operations:

data = CloudStore.get(obj)
headers = CloudStore.head(obj)
CloudStore.delete(obj)

Constructing an object by key sends a metadata request:

obj = CloudStore.Object(bucket, "reports/today.csv")

Read selected bytes

CloudStore.ObjectBytes presents an object as a read-only byte vector. Indexing fetches one small block, and bulk copies fetch only the requested range. Contiguous views share one buffer and the object's version. Every request uses the same validation as copyto! from an Object.

bytes = CloudStore.ObjectBytes(obj) # 64 KiB buffer by default
footer = bytes[end-7:end]          # owned Vector{UInt8}
portion = view(bytes, 101:200)     # no request until bytes are read
copyto!(destination, 1, portion, 1, length(portion))

Consumers that accept an AbstractVector{UInt8} can use this array directly. For example, install ZipArchives separately to list a remote archive and read one entry:

using CloudStore, ZipArchives

obj = CloudStore.Object(bucket, "archive.zip")
archive = ZipArchives.ZipReader(CloudStore.ObjectBytes(obj))
names = ZipArchives.zip_names(archive)
text = ZipArchives.zip_readentry(archive, "reports/today.csv", String)

ZipArchives still owns the ZIP directory and decompression. Reading one entry normally downloads the directory, the selected entry, and a small amount of buffered neighboring data. Its output size is separate from the byte view's blocksize bound. Calling copy on the whole byte view downloads and allocates the whole object.

Views keep the shared buffer alive; no close is needed. Scalar reads from multiple tasks share a lock. Independent bulk copies can overlap. Already buffered bytes remain from the original object version; a later request fails if that version was replaced.

Delete

Delete one object:

CloudStore.delete(bucket, "reports/today.csv")

CloudStore does not create or delete S3 buckets or Azure containers. Use the provider control plane for store lifecycle operations.

Pass request options

Extra keyword arguments pass to the signed CloudBase HTTP request. This supports options such as timeouts, retries, and explicit credentials. Treat provider-specific headers and query parameters with care because they can change request signing.