Object Storage (Softlayer SWIFT) Synchronization Time?

0
votes

Re. SWIFT cluster synchronization time.

I need to know if there's a deterministic way to check whether a HEAD/GET request for a Dynamic Large Object (DLO) will report accurate Content-Length.

My uploader sends files in XX MB chunks and wraps them with a DLO manifest. I'm seeing several instances where a client issuing a HEAD or GET request for a DLO object, receives a "short byte count" in the Content-Length field (typically short by 1 or 2 chunks).

Further, some of these instances can persists for several minutes and even HOURS. In all instances, the Content-Length eventually reports the correct value, so it doesn't look like an uploader bug, it looks like SLOS requires time to synchronize/settle down.

For background info (ignore LightSync), see also: http://www.cs.yale.edu/homes/zhai-ennan/titcheu16sync.pdf

So I need to know if my understanding is grounded in reality, and what mechanisms I should be considering to ensure that the SL Object Store reports an accurate Content-Length, to prevent a situation where a client downloads "corrupt" files (files that are missing a chunk). Is there any SWIFT REST API which provides that guarantee?

Thanks!

1
ibm-cloud-infrastructure

1 Answers

1
votes

The description is working as expected but ‘HOURS’ is definitely too long to be eventually consistent. I recommend to submit a ticket, in order to get further assistance with this behavior. There isn’t a way to deterministically report the new, updated size due to the nature of the eventual consistency.

Perhaps you can try with this header "X-Newest", it could be expensive with DLO depending on the number of segments involved, see: