Streaming transfers

Use a stream when the complete object should not be in memory at one time.

Prefetched downloads

CloudStore.PrefetchedDownloadStream reads ranges in parallel into up to two buffers. Construct it from an Object:

obj = CloudStore.Object(bucket, "large.csv")
io = CloudStore.PrefetchedDownloadStream(obj)
try
    while !eof(io)
        row = readline(io)
        # Process the row.
    end
finally
    close(io)
end

Tune the memory, request size, and request concurrency when needed:

io = CloudStore.PrefetchedDownloadStream(
    obj,
    16 * 1024^2;
    prefetch_multipart_size=2 * 1024^2,
    prefetch_concurrency=8,
)

The first positional size is the size of each in-memory prefetch buffer; prefetch_multipart_size is the maximum size of each range request. The positive integer prefetch_concurrency limits concurrent range requests and defaults to Threads.nthreads(). It is capped by the number of ranges in one buffer, so the example permits at most eight requests at a time, even when Julia has fewer threads.

More concurrent requests can overlap network waits, but may use more connections and HTTP/TLS memory. They do not increase the prefetch buffer sizes. Increasing the buffer size permits more ranges at once and increases buffer memory. Small objects need fewer requests and smaller buffers. The stream is read-only and is not thread-safe.

Prefetched streams and copyto!(dest, doff, object, soff, n) check every range response against the object's size and strong ETag, and throw if the object changed. An Object without an ETag is refreshed with a metadata request first; copyto! does this on every call. The stream returns the stored bytes; wrap it in a decompressor such as CodecZlib.GzipDecompressorStream for gzip objects.

Multipart uploads

CloudStore.MultipartUploadStream sends each written byte vector as one part. The do-block form waits for all parts and completes the upload:

CloudStore.MultipartUploadStream(bucket, "generated.bin") do io
    for chunk in chunks
        write(io, chunk)
    end
end

Each chunk must be a Vector{UInt8}. Except for the final part, Amazon S3 requires parts to meet its minimum part size. Use CloudStore.put for small objects.

Use the manual form only when you need direct lifecycle control:

io = CloudStore.MultipartUploadStream(bucket, "generated.bin")
write(io, first_chunk)
write(io, second_chunk)
wait(io)
close(io)

Keep chunks in object order. concurrent_writes_to_channel limits the number of uploads in flight and applies backpressure to write.

Default buffered transfers

With compress=false, CloudStore.put uploads byte vectors and the readable part of an IOBuffer without copying them. Multipart parts are views into the input. Keep that storage unchanged until the call returns, including retries. Files, other IO inputs, and compressed uploads use extra buffers. On HTTP 1, views over non-Array storage (such as string bytes) are copied first.

Uncompressed byte-vector uploads keep at most batchSize part requests in flight. Each finished request starts another part without waiting for the other requests.

File and compressed multipart uploads read parts into at most batchSize reusable buffers. A buffer is refilled only after every upload in its batch, including retries, has finished. Other IO inputs allocate a new buffer for each part.

CloudStore.get(store, key, destination) accepts a byte vector or a writable byte view. Multipart downloads give each concurrent range request its own part of the destination. File and IO outputs use bounded part buffers and write each part in order while later parts download. copyto! receives directly into a contiguous destination.

CloudStore gives HTTP a private HTTP.Headers collection for each request and passes copyheaders=false when the installed HTTP version honors it. Caller headers stay unchanged, and concurrent parts never share headers. A copyheaders=true keyword from the caller still takes precedence.

The lowest-allocation path needs an HTTP version that supports copyheaders and a CloudBase version that signs buffered payloads without copying them. bench/transfer_allocations.jl --check measures client allocations against local MinIO and Azurite services, including one forced retry per data request.