Working with multipart uploads
With the Hitachi API for Amazon S3, you can perform operations to create an individual object by uploading the object data in multiple parts. This process is called multipart upload.
This section of the Help provides general information about working with multipart uploads.
About multipart uploads
Multipart upload is the process of creating an object by breaking the object data into parts and uploading the parts to HCP individually. The result of a multipart upload is a single object that behaves the same as objects for which the data was stored by means of a single PUT object request. An object created by means of a multipart upload is called a multipart object.
Multipart uploads facilitate storing large objects. With a multipart upload:
- You can upload multiple parts of the object data to HCP concurrently, thereby speeding up the time it takes to store the whole object.
- You don't need to know the full size of the object data before you start uploading the data to HCP. Thus, multipart uploads support storing streaming data in real time.
- You can store an object over time. By spacing the uploads of the individual parts over time, you can reduce the use of bandwidth when other high-bandwidth operations are in progress.
- You can avoid the need to repeat a large upload operation when the connection is lost while the upload is in progress. Because each part you upload can be small, the time required to repeat the upload of a part can be short.
Creating an object by multipart upload
- Initiate the multipart upload.
- Upload the object data parts.
To upload a part, you use either of these operations:
- Upload the part using data from outside HCP.
- Upload the part using data copied from another object already stored in HCP
- Complete the multipart upload.
You can also perform these operations related to multipart uploads:
- Abort a multipart upload
- List the parts of an in-progress multipart upload
- For any given bucket, list the multipart uploads that are currently in progress in that bucket
Considerations for working with multipart uploads
The following considerations apply to working with multipart uploads.
Support for multipart uploads
Whether a bucket supports multipart uploads depends on a bucket configuration setting. The tenant administrator determines the default for this setting for new buckets. This setting is not visible through the S3 compatible API.
If a bucket is empty, the tenant administrator can change it from supporting multipart upload to not supporting multipart upload. Whether the tenant administrator can change a bucket from not supporting multipart upload to supporting multipart upload depends on which other access protocols are enabled for the bucket.
To learn whether the bucket you're working with supports multipart upload, contact your tenant administrator.
Part size and count
The parts you upload for a multipart upload can be any size up to and including five gigabytes. However, the minimum size for the parts you include in a complete multipart upload request, except the last part, is one megabyte. The last part can be smaller than one megabyte.
A multipart upload can have at most ten thousand parts. Therefore, the maximum size for the object resulting from a multipart upload is five terabytes (ten thousand times five gigabytes).
Automatic abort of multipart uploads
After initiating a multipart upload, the user who initiated it can upload parts at any time until the multipart upload is completed or aborted. Completing a multipart upload creates a multipart object from the uploaded parts. Aborting a multipart upload causes the uploaded parts to be deleted. After a multipart upload is aborted, no more parts can be uploaded for it, and it cannot be completed.
If a multipart upload is never completed or aborted, the disjointed parts can remain in HCP indefinitely. Even though the multipart upload is incomplete, these parts count toward the storage used by the bucket where they were uploaded.
To prevent parts of multipart uploads from remaining in HCP indefinitely, the tenant administrator can set the maximum amount of time for which a multipart upload can remain incomplete before the multipart upload is automatically aborted. This time is counted from the time the multipart upload was initiated.
Each bucket has its own automatic abort time for multipart uploads. This time is not visible through the S3 compatible API. By default, the automatic abort time for new buckets is 30 days.
You can change the automatic abort time for a namespace at any time. When you change the time, the new time applies to all subsequent multipart uploads as well as to multipart uploads that are already in progress.
For example, if a user initiates a multipart upload on September 9th while the automatic abort time is set to 30 days, that multipart upload will be automatically aborted on October 9th. If, before October 9th, you change the automatic abort time to 35 days, the date on which the multipart upload will be automatically aborted changes to October 14th. If you change the automatic abort time to an amount of time that has already elapsed since the multipart upload was initiated, the multipart upload is immediately aborted.
x-amz-abort-date
header returned in response to an S3 compatible request to list the parts of a multipart upload is the date on which the multipart upload will be automatically aborted, as determined by the current automatic abort time setting.Client timeouts
Uploading a part of a multipart upload by copying a large amount of data from another object can take some time. If a client times out because an upload part copy operation is taking too long, HCP continues the operation in the background.
Because the connection to the client is broken, HCP cannot report the completion of the upload part copy operation to the client. To see whether the operation has finished successfully, use a GET request with the multipart upload ID to list the parts of the multipart upload. If the operation finished successfully, the part appears in the list. If the operation is still in progress or has failed, the part does not appear in the list.
If upload part copy operations are causing a client to time out, consider increasing the client timeout interval.
Creation date and time for multipart objects
The creation date and time for the object resulting from a multipart upload is the time when the multipart upload was completed, regardless of the amount of time that has elapsed since the multipart upload was initiated.
Retention setting for multipart objects
The retention setting for the object resulting from a multipart upload is calculated based on the time when the multipart upload was completed, regardless of the amount of time that has elapsed since the multipart upload was initiated.
For example, suppose the bucket default retention setting for new objects is 10 days. If a multipart upload in the bucket is completed on September 9th, the resulting object will be under retention until September 19th, regardless of when the multipart upload was initiated.
Retention is not set for the individual parts of an in-progress multipart upload. Replaced parts and parts not used in completed multipart uploads are automatically deleted regardless of the retention setting for the completed multipart object.
Multipart uploads and versioning
If versioning is enabled for a bucket, you can use multipart uploads to store new versions of objects in the bucket. However, HCP does not check whether versioning is enabled until you submit the request to complete a multipart upload. If versioning is disabled and an object with the same name already exists when you submit the complete request, the request fails with a 409 (Conflict) status code.
If versioning is disabled for a bucket and you are concerned about object conflicts, before you initiate a multipart upload for an object, you can check the bucket for an existing object with the same name.
Multiple multipart uploads for objects with the same name
Regardless of the bucket versioning status, you can initiate multiple concurrent multipart uploads with the same object name in a single bucket. These multipart uploads are distinguished by upload ID. Because you specify both the upload ID and the object name in each request to upload a part, each part you upload applies to only one of the multipart uploads.
If versioning is enabled for the bucket, each multipart upload with the same object name, when completed, creates a separate version of the object. The order of the versions is determined by the order in which the multipart uploads are completed. The oldest version is the one resulting from the multipart upload that was completed first. The most recent version is the one resulting from the multipart upload that was completed last.
If versioning is disabled for the bucket, the object stored is the one resulting from the multipart upload that was completed first. Each subsequent completion of a multipart upload for the same object results in a 409 (Conflict) status code, and the resulting object is not stored. The parts of a multipart upload that cannot be completed remain in HCP until the multipart upload is aborted.
Multipart uploads for objects under retention or on hold
You cannot store a new version of an object that's under retention or on hold. However, HCP does not check the retention or hold status of the existing object until you submit the request to complete the multipart upload. If the object is under retention or on hold when you submit the complete request, the request fails with a 403 (Forbidden) status code.
Before you initiate a multipart upload to store a new version of an object, you should use the REST API to check the retention and hold status of the existing object. Because this status can change, you may also want to check the retention and hold status of the existing object periodically while the multipart upload is in progress. If the object is put under retention or on hold while the multipart upload is in progress, you can abort the multipart upload.
Multipart uploads and replicated buckets
All requests related to a given multipart upload must be sent to the same HCP system. That is, even if the bucket on which you initiate the multipart upload is replicated on other HCP systems, requests to upload parts for, list parts for, complete, or abort that multipart upload must be sent to the system on which you initiated the multipart upload.
Initiating a multipart upload
You use the HTTP POST method with the uploads query parameter to initiate a multipart upload in a bucket.
To initiate a multipart upload, you must be an authenticated user. Additionally, you need write permission for the target bucket.
When you initiate a multipart upload, you specify a name for the object you're creating.
In initiate the multipart upload request, you can specify custom metadata to be added to the object when the multipart upload is completed. To do this, you use x-amz-meta- headers.
In the initiate multipart upload request, you can specify an ACL to be added to the object when the multipart upload is completed. To do this, you need to use ACL headers. You cannot use an ACL request body when initiating a multipart upload.
If the ACL you specify in a request to initiate a multipart upload is invalid, HCP returns a 400 (Bad Request) or 501 (Not Implemented) status code and does not initiate the multipart upload.
You cannot change the custom metadata or ACL for a multipart object until the multipart upload used to create the object has been completed.
When you complete a multipart upload, you become the owner of the resulting object.
Depending on whether the bucket name is included in the hostname in the S3 compatible request, a request to initiate a multipart upload has either of these formats:
- With the bucket name included in the hostname:
POST /object-name?uploads HTTP/1.1
- With the bucket name following the hostname:
POST /bucket-name/object-name?uploads HTTP/1.1
The uploads query parameter is case sensitive.
The list below describes the headers you can use in a request to initiate a multipart upload.
Authorization
Specifies user credentials or requests anonymous access.
Date
Specifies the date and time when the request is being made according to the requester. Normally, this is the current date and time.
The date and time must always be specified using Greenwich Mean Time (GMT).
To specify the date and time, use this format:
DDD, dd MMM yyyy HH:mm:ss (+0000|GMT)
In this format:
DDD
The three-letter abbreviation for the day of the week, with an uppercase first letter (for example, Mon).
dd
The two-digit day of the month.
MMM
The three-letter abbreviation for the month, with an uppercase first letter (for example, Feb).
yyyy
The four-digit year.
HH
The hour on a 24-hour clock.
mm
The number of minutes.
ss
The number of seconds.
For example:
Thu, 23 Mar 2017 14:27:05 +0000
All S3 compatible requests must include either a
Date
header or anx-amz-date
header. If a request includes both headers, HCP uses the date and time in thex-amz-date
header.Host
Specifies the hostname for the request. The host name identifies either a tenant or a bucket.
For a tenant, use this format:
tenant-name.hcp-domain-name
For a bucket, use this format:
bucket-name.tenant-name.hcp-domain-name
x-amz-date
Specifies the date and time at which the request is being made according to the requester. Normally, this is the current date and time.
For the valid values for this header, see the description of the
Date
header above.
x-amz-acl
Adds a canned ACL to the bucket.
This header is used only to add a canned ACL to a bucket. If you’re using individual
x-amz-grant-
headers to add the ACL, thex-amz-acl
header is invalid.x-amz-grant-full-control
Grants full control over the bucket to one or more specified grantees.
If you’re using a canned ACL to add an ACL to a bucket, the
x-amz-grant-full-control
header is invalid.x-amz-grant-read
Grants the browse and read data access permissions for the bucket to one or more specified grantees.
If you’re using a canned ACL to add an ACL to a bucket, the
x-amz-grant-read
header is invalid.x-amz-grant-read-acp
Grants the read ACL data access permission for the bucket to one or more specified grantees.
If you’re using a canned ACL to add an ACL to a bucket, the
x-amz-grant-read-acp
header is invalid.x-amz-grant-write
Grants the write and delete data access permissions for the bucket to one or more specified grantees.
If you’re using a canned ACL to add an ACL to a bucket, the
x-amz-grant-write
header is invalid.x-amz-grant-write-acp
Grants the write ACL data access permission for the bucket to one or more specified grantees.
If you’re using a canned ACL to add an ACL to a bucket, the
x-amz-grant-write-acp
header is invalid.x-amz-meta-
Adds custom metadata to the resulting object when the multipart upload is completed.
x-hcp-pretty-print
Optionally, requests that the XML response body be formatted for readability. Valid values are:
true
Format the XML response body for readability.
false
Do not apply any special formatting to the XML response body.
The default is false.
The values
true
andfalse
are not case sensitive.
The list below describes the headers returned in response to a successful request to initiate a multipart upload.
Content-Type
Specifies the Internet media type of the response body. For a request to initiate a multipart upload, the value of this header is always
application/xml;charset=UTF-8
.Date
The date and time when HCP responded to the request, in Greenwich Mean Time (GMT). The date and time are returned in this format:
DDD dd MMM yyyy HH:mm:ss GMT
For example:
Fri, 18 Sep 2020 14:27:05 GMT
Transfer-Encoding
Indicates that HCP could not determine the size of the response body before formulating the response. For a request to list the buckets you own, the value of this header is always
chunked
.x-amz-abort-date
Specifies the date and time when the multipart upload will be automatically aborted if it has not already been completed. The date and time are expressed in this format:
DDD, dd MMM yyyy hh:mm:ss GMT
For example:
Sat, 25 Mar 2017 17:19:26 GMT
x-amz-server-side-encryption
Specifies whether objects stored in HCP are encrypted. Possible values are:
- If objects are encrypted,
AES256
- If objects are not encrypted,
None
This header is returned only if the request headers include
x-amz-server-side-encryption
.- If objects are encrypted,
HCP returns information about the multipart upload that results from a successful initiate request in an XML response body.
The list below describes the XML elements in the response body returned in response to a request to initiate a multipart upload. The elements are listed in alphabetical order.
Bucket
Child of the
InitiateMultipartUploadResult
element.The
Bucket
element specifies the name of the bucket where the multipart upload was initiated.InitiateMultipartUploadResult
Root element.
Key
Child of the
InitiateMultipartUploadResult
element.The
Key
element specifies the name of the object for which the multipart upload was initiated.UploadId
Child of the
InitiateMultipartUploadResult
element.The
UploadId
element specifies the upload ID for the multipart upload initiated by the request.Use this ID in conjunction with the object name to identify the multipart upload in:
- Requests to upload parts for the multipart upload
- Requests to upload parts for the multipart upload by copying
- Requests to complete the multipart upload
- Requests to abort the multipart upload
- Requests to list the parts that have been uploaded for the multipart upload
- Responses to requests to list in-progress multipart uploads
The table below describes HTTP status codes that can be returned in response to a request to initiate a multipart upload.
Code | Meaning | Description |
200 | OK | HCP successfully initiated the multipart upload. |
400 | Bad Request |
Possible reasons include:
|
403 | Forbidden |
Possible reasons include:
|
404 | Not Found | The specified bucket does not exist. |
500 | Internal Server Error |
An internal error occurred. If this error persists, contact your tenant administrator. |
501 | Not Implemented | The request includes the x-amz-acl header with an invalid value. |
503 | Service Unavailable |
HCP is temporarily unable to handle the request, probably due to system overload, maintenance, or upgrade. Try the request again, gradually increasing the delay between each successive attempt. If this error persists, contact your tenant administrator. |
Here’s a sample POST request that initiates a multipart upload for an object named acctg/RulesAndRegulations.pdf in the finance bucket.
Request with s3curl command line
./s3curl.pl --id=lgreen --post -- -k "https://finance.europe.hcp.example.com/acctg/RulesAndRegulations.pdf?uploads" -H "x-hcp-pretty-print: true"
Request headers
POST /acctg/RulesAndRegulations.pdf?uploads HTTP/1.1 Host: finance.europe.hcp.example.com Date: Fri, 07 February 2020 17:19:26 +0000 Authorization: AWS bGdyZWVu:u0eqPDwPw1SR7kdZn87dnGACzaM= x-hcp-pretty-print: true
Response headers
HTTP/1.1 200 OK Date: Fri, 07 February 2020 17:19:26 GMT x-amz-server-side-encryption: None x-amz-abort-date: Sat, 25 Mar 2017 17:19:26 GMT Content-Type: application/xml;charset=UTF-8 Transfer-Encoding: chunked
Response body
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Bucket>finance</Bucket> <Key>acctg/RulesAndRegulations.pdf</Key> <UploadId>94837746087105</UploadId> </InitiateMultipartUploadResult>
Uploading a part of a multipart upload
You use the HTTP PUT method with the uploadId and partNumber query parameters to upload a part of a multipart upload. The bucket where you upload the part must be the same bucket as the one where the multipart upload was initiated.
To upload a part, you must be an authenticated user. You need to use the same user account as the account used in the request to initiate the multipart upload. Additionally, you need write permission for the bucket where you're uploading the part.
For a request to upload a part, the request body consists of the data in a specified file.
You use the uploadId query parameter to specify the upload ID of the multipart upload for which you're uploading the part. If the upload ID you specify doesn't match the upload ID of an in-progress multipart upload or isn't associated with the object specified in the request, HCP returns a 404 (Not Found) status code and does not upload the part.
When you upload a part, you use the partNumber query parameter to specify the part number of the data you're uploading. The part number must be an integer in the range 1 (one) through 10,000.
The parts of the data for an object are ordered, but the part numbers do not need to start at one and do not need to be consecutive numbers. For example, if you're uploading the data for an object in three parts, you could number the parts 1, 2, and 3, but you could also number them 2, 7, and 19.
You can upload the parts of a multipart upload in any order. However, in the request to complete the multipart upload, you need to list the parts in ascending numeric order.
You can upload multiple parts of a multipart upload concurrently.
If a request to upload a part includes any custom metadata (x-amz-meta-) headers, HCP returns a 400 (Bad Request) status code and does not upload the part.
HCP ignores ACL headers in requests to upload a part.
Depending on whether the bucket name is included in the hostname in the S3 compatible request, a request to upload a part of a multipart upload has either of these formats:
- With the bucket name included in the hostname:
PUT /object-name?uploadId=upload-id&partNumber=part-number HTTP/1.1
- With the bucket name following the hostname:
PUT /bucket-name/object-name?uploadId=upload-id&partNumber=part-number HTTP/1.1
The uploadId and partNumber query parameters are not case sensitive.
The list below describes the headers you can use in a request to upload a part of a multipart upload.
Authorization
Specifies user credentials or requests anonymous access.
Content-Length
Specifies the size, in bytes, of the data being uploaded.
Date
Specifies the date and time when the request is being made according to the requester. Normally, this is the current date and time.
The date and time must always be specified using Greenwich Mean Time (GMT).
To specify the date and time, use this format:
DDD, dd MMM yyyy HH:mm:ss (+0000|GMT)
In this format:
DDD
The three-letter abbreviation for the day of the week, with an uppercase first letter (for example, Mon).
dd
The two-digit day of the month.
MMM
The three-letter abbreviation for the month, with an uppercase first letter (for example, Feb).
yyyy
The four-digit year.
HH
The hour on a 24-hour clock.
mm
The number of minutes.
ss
The number of seconds.
For example:
Thu, 23 Mar 2017 14:27:05 +0000
All S3 compatible requests must include either a
Date
header or anx-amz-date
header. If a request includes both headers, HCP uses the date and time in thex-amz-date
header.Host
Specifies the hostname for the request. The host name identifies either a tenant or a bucket.
For a tenant, use this format:
tenant-name.hcp-domain-name
For a bucket, use this format:
bucket-name.tenant-name.hcp-domain-name
x-amz-date
Specifies the date and time at which the request is being made according to the requester. Normally, this is the current date and time.
For the valid values for this header, see the description of the
Date
header above.
Content-MD5
Directs HCP to check the integrity of the data it receives by comparing a Base64-encoded MD5 hash of that data to the value specified by this header. The valid value for this header is the Base64-encoded MD5 hash of the data in the request body.
Content-Type
Specifies the Internet media type of the data being uploaded. Valid values are Internet media types (for example,
text/plain
,application/xml
, orimage/jpeg
).Expect
Tells the application not to send the request body if the request headers are rejected. The only valid value is
100-continue
. This value is not case sensitive.
The list below describes the headers returned in response to a successful request to upload a part of a multipart upload.
Content-length
Specifies the size, in bytes, of the response body. In response to a successful request to upload a part of a multipart upload, the value of this header is always
0
(zero).Date
The date and time when HCP responded to the request, in Greenwich Mean Time (GMT). The date and time are returned in this format:
DDD dd MMM yyyy HH:mm:ss GMT
For example:
Fri, 18 Sep 2020 14:27:05 GMT
ETag
Specifies the ETag for the object.
ETags are useful for making object-level operations conditional based on the object content. Operations that can be made conditional are checking the existence of an object, copying an object, and retrieving an object.
x-amz-server-side-encryption
Specifies whether objects stored in HCP are encrypted. Possible values are:
- If objects are encrypted,
AES256
- If objects are not encrypted,
None
This header is returned only if the request headers include
x-amz-server-side-encryption
.- If objects are encrypted,
x-amz-version-id
Specifies the version ID of the object. This header is returned only while versioning is enabled for the bucket.
The table below describes HTTP status codes that can be returned in response to a request to upload a part of a multipart upload.
Code | Meaning | Description |
200 | OK | HCP successfully uploaded the part. |
400 | Bad Request |
Possible reasons include:
|
403 | Forbidden |
Possible reasons include:
|
404 | Not Found |
One of these:
|
411 | Length Required | Either the request does not include a Content-Length header, or the request includes a Content-Length header with no value. |
413 | Request Entity Too Large | The part you are trying to upload is too big for the amount of space left in the bucket. |
500 | Internal Server Error |
An internal error occurred. If this error persists, contact your tenant administrator. |
503 | Service Unavailable |
HCP is temporarily unable to handle the request, probably due to system overload, maintenance, or upgrade. Try the request again, gradually increasing the delay between each successive attempt. If this error persists, contact your tenant administrator. |
Here’s a sample PUT request that uploads a part of the multipart upload for an object named acctg/RulesAndRegulations.pdf in the finance bucket. The data for the part is in a file named /file-parts-temp/RulesAndRegs-Acctg-1.
The example shows the response headers HCP returns while versioning is enabled for the bucket and while versioning is disabled for the bucket.
Request with s3curl command line
./s3curl.pl --id=lgreen --put=/file-parts-temp/RulesAndRegs-Acctg-1 -- -k "https://finance.europe.hcp.example.com/acctg/RulesAndRegulations.pdf ?uploadId=94837746087105&partNumber=1"
Request headers
PUT /acctg/RulesAndRegulations.pdf?uploadId=94837746087105&partNumber=1 HTTP/1.1 Host: finance.europe.hcp.example.com Date: Fri, 07 February 2020 17:19:26 +0000 Authorization: AWS bGdyZWVu:oVRZgtbngViycDs+I2p2NP5Y3zY=
Response headers with versioning enabled
HTTP/1.1 200 OK Date: Fri, 16 Dec 2016 15:29:28 GMT ETag: "4a01f8acf3353b0a7e1c03a0e34f63e4" x-amz-version-id: 94841738807809 x-amz-server-side-encryption: None Content-Length: 0
Response headers with versioning disabled
HTTP/1.1 200 OK Date: Fri, 07 February 2020 17:19:26 GMT ETag: "4a01f8acf3353b0a7e1c03a0e34f63e4" x-amz-server-side-encryption: None Content-Length: 0
Uploading a part of a multipart object by copying
You can upload a part of a multipart upload by copying all or part of the data for an existing object. To do this, you use the HTTP PUT method with the uploadId and partNumber query parameters and the x-amz-copy-source header. The bucket where you upload the part must be the same bucket as the one where the multipart upload was initiated.
You use the x-amz-copy-source header to specify the object you're copying (the source object) and the bucket that contains that object (the source bucket). The source bucket can be the same bucket as the one to which you're uploading the part. Alternatively, the source bucket can be a different bucket that's owned by the same tenant.
To upload a part by copying, you must be an authenticated user. You need to use the same user account as the account used in the request to initiate the multipart upload. Additionally, you need write permission for the bucket where you're uploading the part and read permission for the source bucket or for the source object itself.
You use the uploadId query parameter to specify the upload ID of the multipart upload for which you're uploading the part. If the upload ID you specify doesn't match the upload ID of an in-progress multipart upload or isn't associated with the object specified in the request, HCP returns a 404 (Not Found) status code and does not upload the part.
When you upload a part by copying, you use the partNumber query parameter to specify the part number of the data you're uploading. The part number must be an integer in the range 1 (one) through 10,000.
The parts of the data for an object are ordered, but the part numbers do not need to start at one and do not need to be consecutive numbers. For example, if you're uploading the data for an object in three parts, you could number the parts 1, 2, and 3, but you could also number them 2, 7, and 19.
You can upload the parts of a multipart upload in any order. However, in the request to complete the multipart upload, you need to list the parts in ascending numeric order.
You can upload multiple parts of a multipart upload concurrently.
If a request to upload a part by copying includes any custom metadata (x-amz-meta-) headers, HCP returns a 400 (Bad Request) status code and does not upload the part.
HCP ignores ACL headers in requests to upload a part by copying.
Depending on whether the bucket name is included in the hostname in the S3 compatible request, for a request to upload a part of a multipart upload by copying has either of these formats:
- With the bucket name included in the hostname:
PUT /object-name?uploadId=upload-id&partNumber=part-number HTTP/1.1
- With the bucket name following the hostname:
PUT /bucket-name/object-name?uploadId=upload-id&partNumber=part-number HTTP/1.1
The uploadId and partNumber query parameters are not case sensitive.
The table below describes the headers you can use in a request to upload a part of a multipart upload by copying.
Authorization
Specifies user credentials or requests anonymous access.
Date
Specifies the date and time when the request is being made according to the requester. Normally, this is the current date and time.
The date and time must always be specified using Greenwich Mean Time (GMT).
To specify the date and time, use this format:
DDD, dd MMM yyyy HH:mm:ss (+0000|GMT)
In this format:
DDD
The three-letter abbreviation for the day of the week, with an uppercase first letter (for example, Mon).
dd
The two-digit day of the month.
MMM
The three-letter abbreviation for the month, with an uppercase first letter (for example, Feb).
yyyy
The four-digit year.
HH
The hour on a 24-hour clock.
mm
The number of minutes.
ss
The number of seconds.
For example:
Thu, 23 Mar 2017 14:27:05 +0000
All S3 compatible requests must include either a
Date
header or anx-amz-date
header. If a request includes both headers, HCP uses the date and time in thex-amz-date
header.Host
Specifies the hostname for the request. The host name identifies either a tenant or a bucket.
For a tenant, use this format:
tenant-name.hcp-domain-name
For a bucket, use this format:
bucket-name.tenant-name.hcp-domain-name
x-amz-copy-source
Specifies the source bucket and object or object version, in this format:
/bucket-name/source-object-name[?versionId=source-object-version-id]
The initial forward slash (/) is required.
Valid values for source-object-version-id are the IDs of versions of the source object specified in the request.
The versionId query parameter is not case sensitive.
If you include the versionId query parameter in the x-amz-copy-source header with an invalid value while versioning is enabled, HCP returns a 404 (Not Found) status code and does not perform the upload part copy operation.
If you include the versionId query parameter in the
x-amz-copy-source
header while versioning is disabled, the parameter is ignored, and the current version of the specified object is used as the source for the upload part copy operation.x-amz-date
Specifies the date and time at which the request is being made according to the requester. Normally, this is the current date and time.
For the valid values for this header, see the description of the
Date
header above.
Content-Type
Specifies the Internet media type of the data being copied. Valid values are Internet media types (for example,
text/plain
,application/xml
, orimage/jpeg
).x-amz-copy-source-if-match
Specifies one or more values for comparison with the ETag of the specified source object or object version. If the ETag matches one of the specified values, HCP continues processing the request. If the ETag doesn’t match any of the specified values, HCP returns a 412 (Precondition Failed) status code and does not upload the part.
To specify the values for this header, use this format:
"value"[, "value"]...
In this format, each value can be any string of one or more characters and must be enclosed in double quotation marks (").
Alternatively, you can specify a single asterisk (*) as the value for the
x-amz-copy-source-if-match
header. All ETags match an asterisk in anx-amz-copy-source-if-match
header.x-amz-copy-source-if-modified-since
Specifies a date and time, in Greenwich Mean Time (GMT), for comparison with the date and time the specified source object or object version was last modified. If the modification date and time is later than the specified date and time, HCP continues processing the request. If the modification date and time is equal to or earlier than the specified date and time, HCP returns a 412 (Precondition Failed) status code and does not upload the part.
To specify the date and time for this header, use one of these formats:
- DDD,ddMMMyyyyHH:mm:ss(+0000|GMT)
For example: Tue, 07 Feb 2017 14:27:05 +0000
- DDDD,dd-MMM-yyyyHH:mm:ss(+0000|GMT)
For example: Tuesday, 07-Feb-17 14:27:05 +0000
- DDDMMMdHH:mm:ssyyyy
For example: Tue Feb 7 14:27:05 2017
- DDD,ddMMMyyyyHH:mm:ss(+0000|GMT)
x-amz-copy-source-if-none-match
Specifies one or more values for comparison with the ETag of the specified source object or object version. If the ETag doesn’t match any of the specified values, HCP continues processing the request. If the ETag matches any of the specified values, HCP returns a 412 (Precondition Failed) status code and does not upload the part.
To specify the values for this header, use this format:
"value"[, "value"]...
In this format, each value can be any string of one or more characters and must be enclosed in double quotation marks (").
Alternatively, you can specify a single asterisk (*) as the value for the
x-amz-copy-source-if-match
header. No ETags match an asterisk in anx-amz-copy-source-if-match
header.x-amz-copy-source-if-unmodified-since
Specifies a date and time, in Greenwich Mean Time (GMT), for comparison with the date and time the specified source object or object version was last modified. If the modification date and time is equal to or earlier than the specified date and time, HCP continues processing the request. If the modification date and time is later than the specified date and time, HCP returns a 412 (Precondition Failed) status code and does not upload the part.
For valid values for this header, see the description of the
x-amz-copy-source-if-modified-since
header above.x-amz-copy-source-range
To upload a part by copying only part of the data for the source object, you use the
x-amz-copy-source-range
request header. The value of thex-amz-copy-source-range
header is the range of bytes you want to copy. The first byte of the data for an object is in position 0 (zero), so a range of one to five specifies the second through sixth bytes, not the first through fifth.To specify a byte range in an x-amz-copy-source-range header, use this format:
bytes=start-position-end-position
These considerations apply to x-amz-copy-source-range header values:
- If you specify a valid range in which the start position is less than the size of the source object data, HCP uploads the part and returns a 200 (OK) status code.
- If you specify a valid range in which the start position is greater than or equal to the size of the source object data, HCP returns a 400 (Bad Request) status code and does not upload a part.
- If you specify an invalid range (for example, a range in which the start position is greater than the end position, HCP returns a 400 (Bad Request) status code and does not upload a part.
The list below describes the headers returned in response to a successful request to upload a part of a multipart upload by copying.
Content-Type
Specifies the Internet media type of the response body. For a request to upload a part by copying, the value of this header is always
application/xml;charset=UTF-8
.Date
The date and time when HCP responded to the request, in Greenwich Mean Time (GMT). The date and time are returned in this format:
DDD dd MMM yyyy HH:mm:ss GMT
For example:
Fri, 18 Sep 2020 14:27:05 GMT
ETag
Specifies the ETag for the object.
ETags are useful for making object-level operations conditional based on the object content. Operations that can be made conditional are checking the existence of an object, copying an object, and retrieving an object.
Transfer-Encoding
Indicates that HCP could not determine the size of the response body before formulating the response. For a request to list the buckets you own, the value of this header is always
chunked
.x-amz-copy-source-version-id
Specifies the version ID of the source object. This header is returned only while versioning is enabled for the source bucket.
x-amz-server-side-encryption
Specifies whether objects stored in HCP are encrypted. Possible values are:
- If objects are encrypted,
AES256
- If objects are not encrypted,
None
This header is returned only if the request headers include
x-amz-server-side-encryption
.- If objects are encrypted,
x-amz-version-id
Specifies the version ID of the object. This header is returned only while versioning is enabled for the bucket.
HCP returns information about the part of a multipart upload that results from a successful upload part copy request in an XML response body.
The list below describes the XML elements in the response body returned in response to a request to upload a part by copying. The elements are listed in alphabetical order.
CopyPartResult
Root element.
ETag
Child of the
CopyPartResult
element.The
ETag
element specifies the ETag for the uploaded part.LastModified
Child of the
CopyPartResult
element.The
LastModified
element specifies the date and time when the uploaded part was last modified, in Greenwich Mean Time (GMT). The date and time are expressed in this format:yyyy-MM-ddTHH:mm:ss.SSSZ
For example:
2020-02-18T19:46:03.856Z
The table below describes HTTP status codes that can be returned in response to a request to upload a part of a multipart upload by copying.
Code | Meaning | Description |
200 | OK | HCP successfully uploaded the part. |
400 | Bad Request |
Possible reasons include:
|
403 | Forbidden |
Possible reasons include:
|
404 | Not Found |
One of these:
|
409 | Conflict | The specified multipart upload was completed or aborted while the upload part copy operation was in progress. |
412 | Precondition Failed |
One of these:
|
500 | Internal Server Error |
An internal error occurred. If this error persists, contact your tenant administrator. |
503 | Service Unavailable |
Possible reasons include:
Try the request again, gradually increasing the delay between each successive attempt. If this error persists, contact your tenant administrator. |
Here’s a sample PUT request that conditionally uploads a part of the multipart upload for an object named acctg/RulesAndRegulations.pdf in the finance bucket by copying the data from an existing object named acctg/AcctgRR-Summary in the same bucket.
For this example, assume that part 6 of the multipart upload has already been uploaded and has an ETag of 7914d874df2c1d55cfab4fa82088ff56. The request directs HCP to create a new part 6 from the copied data only if the ETag of the copied data is different from the ETag of the data previously uploaded for part 6.
The example shows the response headers HCP returns while versioning is enabled for the bucket.
Request with s3curl command line
./s3curl.pl --id=lgreen --copysrc=finance/acctg/AcctgRR-Summary -- -k "https://finance.europe.hcp.example.com/acctg/RulesAndRegulations.pdf ?uploadId=94837746087105&partNumber=6" -H "x-amz-copy-source-if-none-match:7914d874df2c1d55cfab4fa82088ff56" -H "x-hcp-pretty-print: true"
Request headers
PUT /acctg/RulesAndRegulations.pdf?uploadId=94837746087105&partNumber=6 HTTP/1.1 Host: finance.europe.hcp.example.com Date: Fri, 07 February 2020 17:19:26 +0000 Authorization: AWS bGdyZWVu:PBZDJM2WbzEIX4jFBO4Crah4GMQ= x-amz-copy-source: finance/acctg/AcctgRR-Summary x-amz-copy-source-if-none-match:7914d874df2c1d55cfab4fa82088ff56 x-hcp-pretty-print: true
Response headers
HTTP/1.1 200 OK Date: Fri, 07 February 2020 17:19:26 GMT ETag: "f79eac0151d6b62226986e721c89a2f1" x-amz-version-id: 94870864774145 x-amz-copy-source-version-id: 94842948673409 Content-Type: application/xml;charset=UTF-8 x-amz-server-side-encryption: None Transfer-Encoding: chunked
Response body
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <CopyPartResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <LastModified>2017-02-23T17:19:26.000Z</LastModified> <ETag>"f79eac0151d6b62226986e721c89a2f1"</ETag> </CopyPartResult>
Here’s a sample PUT request that uploads a part of the multipart upload for an object named acctg/RulesAndRegulations.pdf in the finance bucket by copying part of the data for an existing object named AcctgBestPractices.doc in the same bucket.
The example shows the response headers HCP returns while versioning is enabled for the bucket.
Request with s3curl command line
./s3curl.pl --id=lgreen --copysrc=finance/AcctgBestPractices.doc -- -k "https://finance.europe.hcp.example.com/acctg/RulesAndRegulations.pdf ?uploadId=94837746087105&partNumber=5" -H "x-amz-copy-source-range: bytes=2800000-9499999" -H "x-hcp-pretty-print: true"
Request headers
PUT /acctg/RulesAndRegulations.pdf?uploadId=94837746087105&partNumber=5 HTTP/1.1 Host: finance.europe.hcp.example.com Date: Fri, 07 February 2020 17:19:26 +0000 Authorization: AWS bGdyZWVu:a0wC/DuV+w2Agq6FagnIe6cShaU= x-amz-copy-source: finance/AcctgBestPractices.doc x-amz-copy-source-range: bytes=2800000-9499999 x-hcp-pretty-print: true
Response headers
HTTP/1.1 200 OK Date: Fri, 07 February 2020 17:19:26 GMT ETag: "bbe438b2a0376f08dc37867a82ea87e6" x-amz-version-id: 94871138333377 x-amz-copy-source-version-id: 94860356828929 Content-Type: application/xml;charset=UTF-8 x-amz-server-side-encryption: None Transfer-Encoding: chunked
Response body
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <CopyPartResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <LastModified>2017-02-23T17:19:26.000Z</LastModified> <ETag>"bbe438b2a0376f08dc37867a82ea87e6"</ETag> </CopyPartResult>
Listing the parts of a multipart upload
You use the HTTP GET method with the uploadId query parameter to list the parts that have been uploaded for an in-progress multipart upload. An in-progress multipart upload is one that has been initiated but not yet completed or aborted. You cannot list the parts of a completed or aborted multipart upload.
To list the parts of a multipart upload, you must be an authenticated user. You need to use either the same user account as the account used in the request to initiate the multipart upload or the user account for the owner of the bucket where the multipart upload was initiated. Additionally, you need read permission for the bucket.
You use the uploadId query parameter to specify the upload ID of the multipart upload for which you're requesting the part listing. If the upload ID you specify doesn't match the upload ID of an in-progress multipart upload or isn't associated with the object specified in the request, HCP returns a 404 (Not Found) status code and does not return a part listing.
A part listing is returned in an XML response body. The parts are listed in numeric order. For each part, the listing contains the part number, the date and time when the part was uploaded, the ETag for the part, and the size of the part.
A part listing does not include parts that are in the process of being uploaded or parts that have been replaced.
Depending on whether the bucket name is included in the hostname in the S3 compatible request, a request to list the parts of a multipart upload has either of these formats:
- With the bucket name included in the hostname:
GET /object-name?uploadId=upload-id[&query-parameters]HTTP/1.1
- With the bucket name following the hostname:
GET /bucket-name/object-name?uploadId=upload-id[&query-parameters]HTTP/1.1
The uploadId query parameter is case sensitive.
query-parameters can be none, one, or more of:
encoding-type
You use the encoding-type query parameter to request the S3 compatible API to encode the response. You can also use this query parameter to specify the encoding method to use.
The key for an object can contain any Unicode character. Some characters, such as those with an ASCII value from 0 to 10, cannot be parsed by XML 1.0 parsers. For these characters, you can add the encoding-type query parameter to request the S3 compatible API to encode the keys in the response.
The encoding-type parameter name is case sensitive.
max-parts
By default, a part listing for a multipart upload can include at most one thousand parts. However, you can use the max-parts query parameter to specify a smaller maximum number of parts to be included in the returned listing. If more than the maximum number of parts satisfy the criteria for a request, you can use query parameters to retrieve the parts in groups.
Valid values for the max-parts query parameter are integers in the range zero through one thousand. If you specify an integer greater than one thousand, HCP returns a 400 (Invalid Argument) status code and does not return a part listing.
The max-parts parameter name is case sensitive.
part-number-marker
By default, a part listing starts with the lowest-numbered part of the multipart upload. You can use the part-number-marker query parameter to request a listing that starts with a part with a higher number. When you include this query parameter in the request for a part listing, the returned listing starts with the part with the next higher number than the value you specified for the parameter.
Valid values for the part-number-marker query parameter are integers in the range 0 (zero) through 10,000.
The part-number-marker parameter name is case sensitive.
The list below describes the headers you can use in a request to list the parts of a multipart upload.
Authorization
Specifies user credentials or requests anonymous access.
Date
Specifies the date and time when the request is being made according to the requester. Normally, this is the current date and time.
The date and time must always be specified using Greenwich Mean Time (GMT).
To specify the date and time, use this format:
DDD, dd MMM yyyy HH:mm:ss (+0000|GMT)
In this format:
DDD
The three-letter abbreviation for the day of the week, with an uppercase first letter (for example, Mon).
dd
The two-digit day of the month.
MMM
The three-letter abbreviation for the month, with an uppercase first letter (for example, Feb).
yyyy
The four-digit year.
HH
The hour on a 24-hour clock.
mm
The number of minutes.
ss
The number of seconds.
For example:
Thu, 23 Mar 2017 14:27:05 +0000
All S3 compatible requests must include either a
Date
header or anx-amz-date
header. If a request includes both headers, HCP uses the date and time in thex-amz-date
header.Host
Specifies the hostname for the request. The host name identifies either a tenant or a bucket.
For a tenant, use this format:
tenant-name.hcp-domain-name
For a bucket, use this format:
bucket-name.tenant-name.hcp-domain-name
x-amz-date
Specifies the date and time at which the request is being made according to the requester. Normally, this is the current date and time.
For the valid values for this header, see the description of the
Date
header above.
x-hcp-pretty-print
Optionally, requests that the XML response body be formatted for readability. Valid values are:
true
Format the XML response body for readability.
false
Do not apply any special formatting to the XML response body.
The default is false.
The values
true
andfalse
are not case sensitive.
The list below describes the headers returned in response to a successful request to list the parts of a multipart upload.
Content-Type
Specifies the Internet media type of the response body. For a request to list the parts of a multipart upload, the value of this header is always
application/xml;charset=UTF-8
.Date
The date and time when HCP responded to the request, in Greenwich Mean Time (GMT). The date and time are returned in this format:
DDD dd MMM yyyy HH:mm:ss GMT
For example:
Fri, 18 Sep 2020 14:27:05 GMT
Transfer-Encoding
Indicates that HCP could not determine the size of the response body before formulating the response. For a request to list the buckets you own, the value of this header is always
chunked
.x-amz-abort-date
Specifies the date and time when the multipart upload will be automatically aborted if it has not already been completed. The date and time are expressed in this format:
DDD, ddMMMyyyyhh:mm:ss GMT
For example:
Sat, 25 Mar 2017 17:19:26 GMT
HCP returns the list of parts of a multipart upload in an XML response body.
The list below describes the XML elements in the response body returned in response to a request to list the parts of a multipart upload. The elements are listed in alphabetical order.
Bucket
Child of the
ListPartsResult
element.The
Bucket
element specifies the name of the bucket where the multipart upload is being performed.DisplayName
Child of the
Initiator
orOwner
element.If the multipart upload initiator or object owner is identified by an HCP user account, the value of the
DisplayName
element is the user name for that account. If the multipart upload initiator or object owner is identified by an AD user account, the value of theDisplayName
element is the user name of that account followed by an at sign (@) and the AD domain name.ETag
Child of the
Part
element.The
ETag
element specifies the ETag of the applicable part.ID
Child of the
Initiator
orOwner
element.If the multipart upload initiator or object owner is identified by an HCP user account, the value of the ID element is the user ID for that account. If the multipart upload initiator or object owner is identified by an AD user account, the value of the ID element is the SID for that account.
Initiator
Child of the
ListPartsResult
element and container for theDisplayName
andID
elements.The
Initiator
element identifies the user who initiated the multipart upload.IsTruncated
Child of the
ListPartsResult
element.The
IsTruncated
element indicates whether the returned part listing includes the last part that satisfies the request criteria. Possible values are:true
The returned listing includes the last part that satisfies the request criteria.
false
The returned listing does not include the last part that satisfies the request criteria.
Key
Child of the
ListPartsResult
element.The
Key
element specifies the name of the object being created by the multipart upload.LastModified
Child of the
Part
element.The
LastModified
element specifies the date and time when the applicable part was last modified, in Greenwich Mean Time (GMT).The date and time are expressed in this format:
yyyy-MM-ddTHH:mm:ss.SSSZ
For example:
2020-02-18T19:46:03.856Z
ListPartsResult
Root element.
MaxParts
Child of the
ListPartsResult
element.The
MaxParts
element specifies the value of the max-parts query parameter included in the request. If the request did not include the max-parts query parameter, the value of theMaxParts
element is 1000.NextPartNumberMarker
Child of the
ListPartsResult
element.The
NextPartNumberMarker
element specifies the number of the last part included in the returned part listing. If the returned listing doesn't include any parts, the value of theNextPartNumberMarker
element is0
(zero).If the returned listing is truncated, you can use the value of the
NextPartNumberMarker
element as the value of the part-number-marker query parameter in a new request to retrieve the next set of parts that satisfy the request criteria.Owner
Child of the
ListPartsResult
element and container for theDisplayName
and ID elements.The
Owner
element identifies the user who will own the object created by the multipart upload.Part
Child of the
ListPartsResult
element and container for the elements that describe a part.The response body contains one
Part
element for each part that has been uploaded for the multipart upload.PartNumber
Child of the
Part
element.The
PartNumber
element specifies the part number of the applicable part.PartNumberMarker
Child of the
ListPartsResult
element.The
PartNumberMarker
element specifies the value of the part-number-marker query parameter included in the request. If the request did not include the part-number-marker query parameter, the value of thePartNumberMarker
element is0
(zero).Size
Child of the
ListPartsResult
element.The value of the
StorageClass
element is alwaysSTANDARD
.UploadId
Child of the
ListPartsResult
element.The
UploadId
element specifies the upload ID of the multipart upload.
The table below describes HTTP status codes that can be returned in response to a request to list the parts of a multipart upload.
Code | Meaning | Description |
200 | OK | HCP successfully listed the parts of the multipart upload. |
400 | Bad Request | Possible reasons include that a query parameter has an invalid value. |
403 | Forbidden |
Possible reasons include:
|
404 | Not Found |
One of these:
|
500 | Internal Server Error |
An internal error occurred. If this error persists, contact your tenant administrator. |
503 | Service Unavailable |
HCP is temporarily unable to handle the request, probably due to system overload, maintenance, or upgrade. Try the request again, gradually increasing the delay between each successive attempt. If this error persists, contact your tenant administrator. |
Here's a sample GET request for a listing of the parts of an in-progress multipart upload for an object named acctg/RulesAndRegulations.pdf in the finance bucket. The request uses these query parameters in addition to uploadId:
part-number-marker=1
Starts the list with the first part with a part number greater than one
max-parts=3
Lists a maximum of three parts
The example assumes that parts one, two, three, five, and six have already been uploaded.
Request with s3curl command line
./s3curl.pl --id=lgreen -- -k "https://finance.europe.hcp.example.com/acctg/RulesAndRegulations.pdf ?uploadId=94837746087105&part-number-marker=1&max-parts=3" -H "x-hcp-pretty-print: true"
Request headers
GET /acctg/RulesAndRegulations.pdf?uploadId=94837746087105 &part-number-marker=1&max-parts=3 HTTP/1.1 Host: finance.europe.hcp.example.com Date: Fri, 07 February 2020 17:19:26 +0000 Authorization: AWS bGdyZWVu:Ey256+39Nz19jRw7chPIOJwD8m4= x-hcp-pretty-print: true
Response headers
HTTP/1.1 200 OK Date: Fri, 07 February 2020 17:19:26 GMT x-amz-abort-date: Sat, 25 Feb 2017 17:19:26 GMT Content-Type: application/xml;charset=UTF-8 Transfer-Encoding: chunked
Response body
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <ListPartsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Bucket>finance</Bucket> <Key>acctg/RulesAndRegulations.pdf</Key> <UploadId>94837746087105</UploadId> <Initiator> <ID>835be4b1-8f84-407b-8084-b9329beadf9b</ID> <DisplayName>lgreen</DisplayName> </Initiator> <Owner> <ID>835be4b1-8f84-407b-8084-b9329beadf9b</ID> <DisplayName>lgreen</DisplayName> </Owner> <StorageClass>STANDARD</StorageClass> <PartNumberMarker>1</PartNumberMarker> <NextPartNumberMarker>5</NextPartNumberMarker> <MaxParts>3</MaxParts> <IsTruncated>true</IsTruncated> <Part> <PartNumber>2</PartNumber> <LastModified>2017-02-23T17:19:26.000Z</LastModified> <ETag>"97ede335e69d08bb3cfbbd974bae68e4"</ETag> <Size>6800000</Size> </Part> <Part> <PartNumber>3</PartNumber> <LastModified>2017-02-23T17:19:26.000Z</LastModified> <ETag>"a39ebe49948a821cab2c4d7d04f9b1c3"</ETag> <Size>6800000</Size> </Part> <Part> <PartNumber>5</PartNumber> <LastModified>2017-02-23T17:19:26.000Z</LastModified> <ETag>"bbe438b2a0376f08dc37867a82ea87e6"</ETag> <Size>6700000</Size> </Part> </ListPartsResult>
Completing a multipart upload
You use the HTTP POST method with the uploadId query parameter to complete a multipart upload. You must use the same user account as the account used in the request to initiate the multipart upload. Additionally, you need write permission for the bucket where the multipart upload was initiated.
You use the uploadId query parameter to specify the upload ID of the multipart upload you are completing. If the upload ID you specify does not match the upload ID of an in-progress multipart upload or is not associated with the object specified in the request, the upload does not complete and HCP returns a 404 (Not Found) status code.
To specify the list of parts to use when completing a multipart upload, you use an XML request body. You identify each part by its part number and the ETag
returned when the part was uploaded. The list of parts must be in ascending order by part number.
Only the parts listed in the request to complete a multipart upload are used as data for the resulting object. Parts that were uploaded for the multipart upload but that are not listed in the request are automatically deleted after the request is processed.
A multipart upload must have at least one part to complete.
When you complete a multipart upload, you become the owner of the resulting object.
If a request to complete a multipart upload includes any custom metadata x-amz-meta-
headers, HCP returns a 400 (Bad Request) status code and does not complete the multipart upload.
HCP ignores ACL headers in requests to complete a multipart upload.
Depending on whether the bucket name is included in the hostname in the S3 compatible request, a request to complete a multipart upload has either of these formats:
- With the bucket name included in the hostname:
POST /object-name?uploadId=upload-id HTTP/1.1
- With the bucket name following the hostname:
POST /bucket-name/object-name?uploadId=upload-id HTTP/1.1
The uploadId query parameter is case sensitive.
The following list describes the required headers to use in a request to complete a multipart upload.
Authorization
Specifies user credentials or requests anonymous access.
Content-Length
Specifies the size, in bytes, of the request body.
Content-Type
Specifies the Internet media type of the request body. The only valid values are
application/xml
andtext/xml
.Date
Specifies the date and time when the request is being made according to the requester. Normally, this is the current date and time.
The date and time must always be specified using Greenwich Mean Time (GMT).
To specify the date and time, use this format:
DDD, dd MMM yyyy HH:mm:ss (+0000|GMT)
In this format:
DDD
The three-letter abbreviation for the day of the week, with an uppercase first letter (for example, Mon).
dd
The two-digit day of the month.
MMM
The three-letter abbreviation for the month, with an uppercase first letter (for example, Feb).
yyyy
The four-digit year.
HH
The hour on a 24-hour clock.
mm
The number of minutes.
ss
The number of seconds.
For example:
Thu, 23 Mar 2017 14:27:05 +0000
All S3 compatible requests must include either a
Date
header or anx-amz-date
header. If a request includes both headers, HCP uses the date and time in thex-amz-date
header.Host
Specifies the hostname for the request. The host name identifies either a tenant or a bucket.
For a tenant, use this format:
tenant-name.hcp-domain-name
For a bucket, use this format:
bucket-name.tenant-name.hcp-domain-name
x-amz-date
Specifies the date and time at which the request is being made according to the requester. Normally, this is the current date and time.
For the valid values for this header, see the description of the
Date
header above.
If-Match
Specifies a value for comparison with the ETag of the most recent version of the object specified in the request. If the specified value matches the Etag, HCP continues processing the request. If the specified value doesn't match the Etag or the specified object doesn't exist, HCP returns an error response body and does not complete the multipart upload.
The value of this header can be any string of one or more characters.
x-hcp-pretty-print
Optionally, requests that the XML response body be formatted for readability. Valid values are:
true
Format the XML response body for readability.
false
Do not apply any special formatting to the XML response body.
The default is false.
The values
true
andfalse
are not case sensitive.x-hcp-labelretentionhold
Specifies whether to place one or more labeled holds on the object and the associated labeled hold IDs. The values passed with this header are in a JSON format string containing one or more id and hold value pairs.
id
Unique label name. The maximum label size is 64 characters.
hold
Either
true
orfalse
.Example
Place two labeled holds on the object.
x-hcp-labelretentionhold [{"id":"UniqueLabelHold-1","hold":true}, {"id":"UniqueLabelHold-2","hold":true}]
x-hcp-retention
Specifies the retention value for the object being stored. This value can be a fixed date, an offset, a retention class, or a special value.
x-hcp-retentionhold
Specifies whether the object is on hold. This value can be either
true
orfalse
.
The next list describes the XML elements in a complete multipart upload request body. The elements are listed in alphabetical order.
CompleteMultipartUpload
Root element. This must be the first element in the complete multipart upload request body.
The
CompleteMultipartUpload
element can optionally include this XML namespace specification:xmlns="http://s3.amazonaws.com/doc/2006-03-01/"
ETag
Child of the
Part
element.The
ETag
element specifies the ETag of the applicable part.Part
Child of the
CompleteMultipartUpload
element and container for the elements that describe a part.The
Part
element identifies a part that has been uploaded for the multipart upload.PartNumber
Child of the
Part
element.The
PartNumber
element specifies the part number of the applicable part.
The list below describes the headers returned in response to a successful request to complete a multipart upload.
Content-Type
Specifies the Internet media type of the response body. For a request to complete a multipart upload, the value of this header is always
application/xml;charset=UTF-8
.Date
The date and time when HCP responded to the request, in Greenwich Mean Time (GMT). The date and time are returned in this format:
DDD dd MMM yyyy HH:mm:ss GMT
For example:
Fri, 18 Sep 2020 14:27:05 GMT
x-amz-server-side-encryption
Specifies whether objects stored in HCP are encrypted. Possible values are:
- If objects are encrypted,
AES256
- If objects are not encrypted,
None
This header is returned only if the request headers include
x-amz-server-side-encryption
.- If objects are encrypted,
x-amz-version-id
Specifies the version ID of the object. This header is returned only while versioning is enabled for the bucket.
HCP returns information about the object that results from a successful complete multipart upload request in an XML response body.
The information is returned in the following format:
<?xml version='1.0' encoding='UTF-8'?> <CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2020-09-01/"> <Location>object-url</Location> <Bucket>bucket-name</Bucket> <Key>object-name</Key> <ETag>"etag"</ETag> </CompleteMultipartUploadResult>
The following list describes the XML elements returned in the response body. The elements are listed in alphabetical order.
Bucket
Child of the
CompleteMultipartUploadResult
element.The
Bucket
element specifies the bucket where the object was created.CompleteMultipartUploadResult
Root element.
ETag
Child of the
CompleteMultipartUploadResult
element.The
ETag
element specifies the ETag for the object.The ETag in the multipart upload complete operation is not an MD5 hash of the object data and therefore is not useful for verifying the data integrity.
Key
Child of the
CompleteMultipartUploadResult
element.The
Key
element specifies the name of the object.Location
Child of the
CompleteMultipartUploadResult
element.The
Location
element specifies the complete URL for the object.
The table below describes HTTP status codes that can be returned in response to a request to complete a multipart upload.
Code | Meaning | Description |
200 | OK |
Possible reasons include:
|
400 | Bad Request |
Possible reasons include:
|
403 | Forbidden |
Possible reasons include:
|
404 | Not Found |
One of these:
|
500 | Internal Server Error |
An internal error occurred. If this error persists, contact your tenant administrator. |
503 | Service Unavailable |
HCP is temporarily unable to handle the request, probably due to system overload, maintenance, or upgrade. Try the request again, gradually increasing the delay between each successive attempt. If this error persists, contact your tenant administrator. |
Here is a sample POST request that completes a multipart upload for an object named acctg/RulesAndRegulations.pdf in the finance bucket. The request body is in a file named Acctg-RulesAndRegulations.xml.
In this example, the retention value of the object is set to five days past the time when the object was stored. Additionally, a labeled retention hold is placed on the object. The user has write and data access permissions on the bucket.
The request is being made while versioning is enabled for the bucket.
Request body
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <CompleteMultipartUpload> <Part> <PartNumber>1</PartNumber> <ETag>4a01f8acf3353b0a7e1c03a0e34f63e4</ETag> </Part> <Part> <PartNumber>2</PartNumber> <ETag>97ede335e69d08bb3cfbbd974bae68e4</ETag> </Part> <Part> <PartNumber>3</PartNumber> <ETag>a39ebe49948a821cab2c4d7d04f9b1c3</ETag> </Part> <Part> <PartNumber>4</PartNumber> <ETag>a7b8192c98da2c13a034980376b6c809</ETag> </Part> <Part> <PartNumber>5</PartNumber> <ETag>bbe438b2a0376f08dc37867a82ea87e6</ETag> </Part> <Part> <PartNumber>6</PartNumber> <ETag>f79eac0151d6b62226986e721c89a2f1</ETag> </Part> </CompleteMultipartUpload>
Request with s3curl command line
./s3curl.pl --id=lgreen --post=Acctg-RulesAndRegulations.xml -- -k "https://finance.europe.hcp.example.com/acctg/RulesAndRegulations.pdf ?uploadId=94837746087105" -H "x-hcp-retention: A+5d" .-H 'x-hcp-labelretentionhold: [{"id":"UniqueLabelHold-1","hold":true}]' -H "x-hcp-pretty-print: true"
Request headers
POST /acctg/RulesAndRegulations.pdf?uploadId=94837746087105 HTTP/1.1 Host: finance.europe.hcp.example.com Date: Fri, 26 June 2020 17:19:26 +0000 Authorization: AWS bGdyZWVu:y4wahRZz+oLpASXkNeBy9LQAJsE= X-HCP-Retention: A+5d X-HCP-LabelRetentionHold: '[{"id":"UniqueLabelHold-1","hold":true}]'
Response headers
HTTP/1.1 200 OK Date: Fri, 26 June 2020 17:19:26 GMT x-amz-server-side-encryption: None x-amz-version-id: 94874564091777 Content-Type: application/xml;charset=ISO-8859-1 ETag: "e879d0c42d032f7b55d2be82825071d6-6" Content-Length: 401
Response body
<?xml version='1.0' encoding='UTF-8'?>
<CompleteMultipartUploadResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Location>https://finance.europe.hcp.example.com/hs3/acctg
/RulesAndRegulations.pdf</Location>
<Bucket>finance</Bucket>
<Key>acctg/RulesAndRegulations.pdf</Key>
<ETag>"e879d0c42d032f7b55d2be82825071d6-6"</ETag>
</CompleteMultipartUploadResult>
Aborting a multipart upload
You use the HTTP DELETE method with the uploadId query parameter to abort a multipart upload.
To abort a multipart upload, you must be an authenticated user. You need to use the same user account as the account used in the request to initiate the multipart upload or the user account for the owner of the bucket where the multipart upload was initiated. Additionally, you need delete permission for the bucket.
You use the uploadId query parameter to specify the upload ID of the multipart upload you're aborting. If the upload ID you specify doesn't match the upload ID of an in-progress multipart upload or isn't associated with the object specified in the request, HCP returns a 404 (Not Found) status code and does not abort the multipart upload.
You can abort any in-progress multipart upload, including multipart uploads that are in the process of being completed. You cannot abort a completed multipart upload.
When you abort a multipart upload, HCP stops uploading any parts that are in the process of being uploaded. You cannot upload additional parts for or list the parts of an aborted multipart upload.
Parts of an aborted upload that were already uploaded or that were in the process of being uploaded when you issued the abort request are automatically deleted.
Depending on whether the bucket name is included in the hostname in the S3 compatible request, a request to abort a multipart upload has either of these formats:
- With the bucket name included in the hostname:
DELETE /object-name?uploadId=upload-id HTTP/1.1
- With the bucket name following the hostname:
DELETE /bucket-name/object-name?uploadId=upload-id HTTP/1.1
The uploadId query parameter is case sensitive.
If the named object exists, HCP deletes it and returns a 204 (No Content) status code, indicating that the request was successful. HCP also returns a 204 status code if the named object does not exist. Because HCP returns a 204 status code for a successful abort of a multipart upload, you cannot tell from the status code alone whether a request to abort a multipart upload had the intended result.
The list below describes the headers you can use in a request to abort a multipart upload.
Authorization
Specifies user credentials or requests anonymous access.
Date
Specifies the date and time when the request is being made according to the requester. Normally, this is the current date and time.
The date and time must always be specified using Greenwich Mean Time (GMT).
To specify the date and time, use this format:
DDD, dd MMM yyyy HH:mm:ss (+0000|GMT)
In this format:
DDD
The three-letter abbreviation for the day of the week, with an uppercase first letter (for example, Mon).
dd
The two-digit day of the month.
MMM
The three-letter abbreviation for the month, with an uppercase first letter (for example, Feb).
yyyy
The four-digit year.
HH
The hour on a 24-hour clock.
mm
The number of minutes.
ss
The number of seconds.
For example:
Thu, 23 Mar 2017 14:27:05 +0000
All S3 compatible requests must include either a
Date
header or anx-amz-date
header. If a request includes both headers, HCP uses the date and time in thex-amz-date
header.Host
Specifies the hostname for the request. The host name identifies either a tenant or a bucket.
For a tenant, use this format:
tenant-name.hcp-domain-name
For a bucket, use this format:
bucket-name.tenant-name.hcp-domain-name
x-amz-date
Specifies the date and time at which the request is being made according to the requester. Normally, this is the current date and time.
For the valid values for this header, see the description of the
Date
header above.
The list below describes the headers returned in response to a successful request to abort a multipart upload.
Date
The date and time when HCP responded to the request, in Greenwich Mean Time (GMT). The date and time are returned in this format:
DDD dd MMM yyyy HH:mm:ss GMT
For example:
Fri, 18 Sep 2020 14:27:05 GMT
The table below describes HTTP status codes that can be returned in response to a request to complete a multipart upload.
Code | Meaning | Description |
204 | No Content | HCP successfully aborted the multipart upload. |
403 | Forbidden |
Possible reasons include:
|
404 | Not Found |
One of these:
|
500 | Internal Server Error |
An internal error occurred. If this error persists, contact your tenant administrator. |
503 | Service Unavailable |
HCP is temporarily unable to handle the request, probably due to system overload, maintenance, or upgrade. Try the request again, gradually increasing the delay between each successive attempt. If this error persists, contact your tenant administrator. |
Here’s a sample DELETE request that aborts a multipart upload for an object named acctg/RulesAndRegulations.pdf in the finance bucket.
Request with s3curl command line
./s3curl.pl --id=lgreen --delete -- -k "https://finance.europe.hcp.example.com/acctg/RulesAndRegulations.pdf ?uploadId=94874755807297"
Request headers
DELETE /acctg/RulesAndRegulations.pdf?uploadId=94874755807297 HTTP/1.1 Host: finance.europe.hcp.example.com Date: Fri, 07 February 2020 17:19:26 +0000 Authorization: AWS bGdyZWVu:KxEygWb4ay0jHtOrIljBE5lGmls=
Response headers
HTTP/1.1 204 No Content Date: Fri, 07 February 2020 17:19:26 GMT