Working with POST object uploads
With the Hitachi API for Amazon S3, you can accept objects uploaded directly to HCP from web browser forms (that is, a POST form request). This process is called a POST object upload.
This section of the Help starts with general information about and considerations for working with POST object uploads.
About POST object uploads
The POST request is an alternative to a PUT request. A POST object upload is the process of accepting objects from an HTTP form sent by a web browser. When a user submits the form, the browser uploads its contents, which can include data such as a text field or a file, to HCP. Any parameters are passed, not within the header as in a PUT request, but as form fields within the body of the request. The web page can be hosted on the same domain as HCP or a different domain. A user must have write access to a bucket to add an object to it. The POST request can upload a security policy with the request using form fields in the multipart/form-data encoded message body, so a bucket administrator can create preset conditions under which the upload is allowed.
Because the request is authenticated without passing data through a secure intermediary node that protects credentials, POST object uploads can reduce latency. Because an object is uploaded in a single operation, an HTTP success response indicates that the entire object has been stored.
POST object upload authentication
A POST object upload request can be authenticated in these ways:
Header authentication
The header can include AWS Signature Version 2 authorization information. Header authentication supports only AWS local user account credentials. Use a header with a value in the format
Authorization: AWS all_users:
for anonymous access, orAuthorization: AWS access-key:signature
for authenticated access.NoteActive Directory, SPNEGO, cookie, and HCP authentication are not supported.In-form authentication
The POST form can contain authentication-related fields. In-form authentication supports only AWS local user account credentials. All authorization-related form fields must be presented for in-form authentication to succeed. If both V2 and V4 authentication form fields are presented, V2 authentication information is used.
Security policy validation
A policy is included in the request. Required if in-form authentication is used. If the policy is missing, HCP returns a status of 400.
For a namespace that does not permit anonymous access (that is, a non-public bucket), either header or in-form authentication is required. (Both can be provided.)
Security policies
The POST security policy is part of the authentication of a POST object upload request. The policy consists of base-64 encoded JSON code which is submitted as a form field in the request.
If a form contains fields that don't meet the policy's requirements, the request is rejected with the status 403.
Here's an example of a POST policy using AWS Signature Version 4 conditions:
{ "expiration": "2020-11-02T123:01:00.000Z", "conditions": [ {"bucket": "Finance"}, ["starts-with", "$key", "user/user1/"], {"acl": "public-read"}, {"success_action_redirect": "http://www.company.com/success"}, {"x-amz-meta-uuid": "14365123651274"}, {"x-amz-credential": "AKIAIOSFODNN7EXAMPLE/20151229/us-east-1/s3/aws4_request"}, {"x-amz-algorithm": "AWS4-HMAC-SHA256"}, {"x-amz-date": "20201102T000000Z" } ] }
A security policy always contains the elements expiration
and conditions
. The example policy uses two condition matching types, exact matching and starts-with matching. The following sections describe these elements.
The element expiration
specifies the expiration date and time of the security policy in ISO8601 GMT date format. For example, 2019-08-10T12:00:00.000Z specifies that the POST policy is not valid after midnight GMT on August 10, 2019.
For AWS Signature Version 4 authentication, if the field x-amz-date
contains an expired date, the policy is considered expired.
The table below describes condition matching types that you can use to specify POST security policy conditions. You must specify at least one condition for each field that you specify. You can create more complex matching criteria by specifying multiple conditions for a given form field. If there are multiple conditions for a field, then all the conditions must match.
Exact Matches
The form field value must match the value specified. This example indicates that the ACL must be set to public-read:
{"acl": "public-read" }
This example is an alternate way to indicate that the ACL must be set to public-read:
[ "eq", "$acl", "public-read" ]
Starts With
The value must start with the specified value. This example indicates that the object key must start with user/user1:
["starts-with", "$key", "user/user1/"]
Matching Any Content
To configure the POST security policy to allow any content within a form field, use starts-with with an empty value (""). This example allows any value for success_action_redirect:
["starts-with", "$success_action_redirect", ""]
Specifying Ranges
For form fields that accept a range, separate the upper and lower limit with a comma. This example allows a file size from 1 to 10 MB:
["content-length-range", 1048579, 10485760]
The conditions in a POST policy are an array of objects, any of which you can use to validate the request. You can use these conditions to restrict what is allowed in the request. For example, the policy conditions in the preceding example require the following conditions:
- The request must specify the bucket name
Finance
. - The object key name must have the prefix
user/user1
. - The object ACL must be set to
public-read
.
Each field that you specify in a form (except x-amz-signature
, AccessKeyId
, file, policy, and field names that have the prefix x-ignore-) must appear in the list of conditions.
All variables within the form are expanded before validating the security policy. Therefore, all condition matching is executed against the expanded form fields. For example, suppose that you want to restrict your object key name to a specific prefix (user/user1
). In this case, you set the key form field to user/user1/${filename}
. Your POST policy should be [ "starts-with", "$key", "user/user1/" ]
(do not enter [ "starts-with", "$key", "user/user1/${filename}" ]
).
The list below describes supported conditions.
acl
Specifies the ACL value that must be used in the submitted form.
bucket
Specifies the acceptable bucket name.
content-length-range
Specifies the minimum and maximum allowable size for the uploaded content.
If the object is compressed and the value of content-encoding is
gzip
, the size limit is applied to the size of the uncompressed object.key
Specifies the acceptable key name or a prefix of the uploaded object.
success_action_ redirect
redirect
Specifies the URL to which the client is redirected upon successful upload.
success_action_status
If you don't specify a value for
success_action_redirect
, this element specifies the status code returned to the client when the upload succeeds.x-amz-algorithm
(AWS Signature Version 4 authentication only field)
Specifies the signing algorithm that must be used during signature calculation. The value is
AWS4-HMAC-SHA256
.x-amz-credential
(AWS Signature Version 4 authentication only field)
Specifies the credentials that you used to calculate the signature. Required if you include a POST policy document with the request. Use this format:
access_key_id/date/aws_region/aws_service/aws4_request
For Amazon S3, the value of
aws_service
iss3
.For example:
AKIAIOSFODNN7EXAMPLE/20191024/us-east-1/s3/aws4_request
x-amz-date
(AWS Signature Version 4 authentication only field)
Specifies a date value in ISO8601 format. Required if you include a POST policy document with the request.
For example:
20191024T000000Z
x-amz-meta-*
User-specified metadata.
x-amz-*
Other x-amz- headers.
x-hcp-retention
Specifies the retention value for the object being stored. This value can be a fixed date, an offset, a retention class, or a special value.
x-hcp-retentionhold
Specifies whether the object is on hold. This value can be either
true
orfalse
.x-ignore-*
Users can specify additional form fields with this header that should be ignored.
The table below describes characters that must be escaped within a security policy.
Escape sequence | Description |
\\ | Backslash |
\b | Backspace |
\f | Form feed |
\n | New line |
\r | Carriage return |
\t | Horizontal tab |
\v | Vertical tab |
\unnnn | All Unicode characters |
Executing a POST object upload
When you execute a POST object upload, you specify a name for the object you're creating.
In the upload request, you can specify custom metadata to be added to the object when the upload is completed. To do this, you use the field x-amz-meta- within the post form.
In the upload request, you can specify an ACL to be added to the object when the upload is completed. To do this, you use the field acl within the post form.
If the ACL you specify in a request to execute an upload is invalid, HCP returns the status code 400 (Bad Request) or 501 (Not Implemented) and does not execute the upload.
When you complete an upload, you become the owner of the resulting object.
The list below describes the headers you can use in a request to execute a POST object upload.
Authorization
Specifies user credentials or requests anonymous access.
Host
Specifies the hostname for the request. The host name identifies either a tenant or a bucket.
For a tenant, use this format:
tenant-name.hcp-domain-name
For a bucket, use this format:
bucket-name.tenant-name.hcp-domain-name
x-amz-server-side-encryption
Requests that the response headers include
x-amz-server-side-encryption
, which indicates whether objects stored in HCP are encrypted. The value of thex-amz-server-side-encryption
request header can be any character string.Data is only encrypted when Encryption for Primary Running Storage is configured.
The content of a POST object upload request contains a form with fields that are passed to HCP as the contents of a Content-Disposition
header. The list below describes the fields you can use in a POST object upload.
AWSAccessKeyId
(AWS Signature Version 2 authentication only field)
Required if policy is sent with request
Specifies the owner of the bucket who grants an anonymous user access for a request that satisfies the constraints in the policy.
A Base64-encoded username for a user account.
Signature
(AWS Signature Version 2 authentication only field)
Required if policy is sent with request
Specifies a value calculated using the secret key and the policy string, as "string to sign."
file
Contains file or text content. Can be used with a
File
or aTextarea
form element.HCP ignores any fields that appear after this field.
You can only upload one file at a time.
key
Specifies the name of the uploaded key.
To use the file name provided by the user, use the variable ${filename}. For example, if the user Greta uploads the file new_hq.jpg and you specify
/user/greta/${filename}
, the key name is/user/greta/new_hq.jpg
.policy
Required if bucket is not publicly writable
Specifies a security policy describing what is permitted in the request. Requests without a security policy are considered anonymous and work only on publicly writable buckets.
x-amz-algorithm
Required if policy is sent with request
(AWS Signature Version 4 authentication only field)
Specifies the signing algorithm that must be used during signature calculation. The value is
AWS4-HMAC-SHA256
.x-amz-date
Required if policy is sent with request
(AWS Signature Version 4 authentication only field)
Specifies the date and time at which the request is being made according to the requester. Normally, this is the current date and time.
x-amz-signature
Required if policy is sent with request
(AWS Signature Version 4 authentication only field)
Specifies a value calculated using the secret key and specific elements of the request, including the policy string, as "string to sign."
x-amz-credential
Required if using V4 authentication and policy is sent with request
(AWS Signature Version 4 authentication only field)
Specifies the credentials that you used to calculate the signature. It provides access key ID and scope information identifying the region and service for which the signature is valid. This should be the same scope you used to calculate the signing key.
Use this format:
access-key-id/date/aws-region/aws-service/aws4_request
For example:
AKIAIOSFODNN7EXAMPLE/20191024/us-east-1/s3/aws4_request
For Amazon S3, the aws-service string is
s3
.
acl
Adds a canned ACL to the resulting object when the upload is completed. The default is private.
Content-Encoding
REST-specific heades. You can use the values in a policy; they are otherwise treated as follows:
If value is
gzip
and file is gzipped, HCP decodes the file.Expires
REST-specific headers. You can use the values in a policy; they are otherwise treated as follows:
If present, HCP returns the value in the response header.
success_action_ redirect
redirect
Specifies a URL to which the client is redirected upon successful upload.
If not specified, HCP returns the empty document type specified in the field
success_action_status
.If HCP cannot interpret the URL, it acts as if the field is not present.
If the upload fails, HCP displays an error and does not redirect the user to a URL.
NoteThe fieldredirect
is deprecated.success_action_status
If you don't specify
success_action_redirect
, this status code is returned to the client when the upload succeeds.Accepts the values
200
,201
, or204
(the default).If set to
200
or204
, HCP returns an empty document with a 200 or 204 status code.If set to
201
, HCP returns an XML document with a 201 status code.If not set or set to an invalid value, HCP returns an empty document with a 204 status code.
NoteSome versions of Adobe Flash Player do not properly handle HTTP responses with an empty body. To support uploads through Adobe Flash, set to201
.x-amz-meta-*
Adds customer metadata, stored as key-value pairs, to the resulting object when the upload is completed.
x-hcp-retention
Specifies the retention value for the object being stored. This value can be a fixed date, an offset, a retention class, or a special value
x-hcp-retentionhold
Specifies whether the object is on hold. This value can be either
true
orfalse
.
The list below describes the headers returned in response to a successful POST object upload request.
Date
The date and time when HCP responded to the request, in Greenwich Mean Time (GMT). The date and time are returned in this format:
DDD dd MMM yyyy HH:mm:ss GMT
For example:
Thu, 23 Mar 2017 14:27:05 GMT
Location
Specifies the URL to which the client is redirected if the upload is successful.
x-amz-server-side-encryption
Specifies whether objects stored in HCP are encrypted. Possible values are:
- If objects are encrypted,
AES256
- If objects are not encrypted,
None
This header is returned only if the request headers include
x-amz-server-side-encryption
.- If objects are encrypted,
x-amz-version-id
Specifies the version ID of the object. This header is returned only while versioning is enabled for the bucket.
If success_action_code
is set to 201, HCP returns information about the object that results from a successful upload request in an XML response body.
The list below describes the XML elements in the response body returned in response to a request to complete a POST object upload. The elements are listed in alphabetical order.
Bucket
Child of the
PostResponse
element.The
Bucket
element specifies the bucket where the object was created.ETag
Child of the
PostResponse
element.The
ETag
or entity tag element specifies the MD5 hash of the object. The element reflects the contents of an object, not its metadata.You can do a conditional GET operation by combining the
ETag
tag with anIf-Modified
tag.Key
Child of the
PostResponse
element.The
Key
element specifies the name of the object.Location
Child of the
PostResponse
element.The
Location
element specifies the complete URL for the object.
The table below describes HTTP status codes that can be returned in response to a request to complete a POST object upload.
Code | Meaning | Description |
200 | OK | HCP successfully processed the request. |
201 | OK | HCP successfully processed the request. Returns an XML document with a 201 status code. Recommended for uploads through Adobe Flash. |
204 | OK | HCP successfully processed the request. (Default success response.) |
400 | Bad Request |
Possible reasons include:
|
403 | Forbidden |
Possible reasons include:
|
500 | Internal Server Error |
An internal error occurred. If this error persists, contact your tenant administrator. |
503 | Service Unavailable |
HCP is temporarily unable to handle the request, probably due to system overload, maintenance, or upgrade. Try the request again, gradually increasing the delay between each successive attempt. If this error persists, contact your tenant administrator. |
Here’s a sample POST object upload request that initiates an object upload for an object named acctg/Checklist.txt in the bucket named finance
with AWS Signature Version 2 in-form authentication.
Request headers and form
<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> </head> <body> <form action="https://finance.europe.hcp.example.com" method="post" enctype="multipart/form-data"> <input type="input" name="key" value="acctg/${filename}" /><br /> <input type="hidden" name="AWSAccessKeyId" value="bGdyZWVu" /> <input type="hidden" name="Signature" value="yAk4cp0LQzNo9HMxXsiM9NNSjVs=" /> <input type="hidden" name="Policy" value='eyJleHBpcmF0aW9uIjoiMjAxOS0xMi0zMFQxMjowMDowMC4wMDBaLdwiY29uZGl0aW9ucyI6W3siYnVja2V0IjogIm5zIn0sWyJzdGFydHMtd2l0aCIsICIka2V5IiwgImRpci9kaXIxLyJdLFsic3RhcnRzLXdpdGgiLCAiJHN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyIsICIiXSx7ImFjbCI6ICJwdWJsaWMtcmVhZCJ9LHsieC1hbXotbWV0YS1jbGFzcyI6ICJidXNpbmVzcyJ9LFsiY29udGVudC1sZW5ndGgtcmFuZ2UiLCAwLCAxMDQ4NTc2MF1dfQ== /> <input type="hidden" name="success_action_status" value="201"/> <input type="hidden" name="acl" value="public-read"/> <input type="hidden" name="x-amz-meta-class" value="business"/> <input type="file" name="file" /> <br /> <!-- The elements after this will be ignored --> <input type="submit" name="submit" value="Upload to HCP" /> </form> </html>
Security policy
{ "expiration": "2019-12-31T12:00:00.000Z", "conditions": [ {"bucket": "finance"}, ["starts-with", "$key", "acctg/"], {"acl": "public-read"}, {"x-amz-meta-class": "business"}, ["content-length-range", 0, 10485760] ] }
Response headers
Cache-Control: no-cache,no-store,must-revalidate Content-Encoding: gzip Content-Security-Policy: default-src 'self'; script-src…elf'; frame-ancestors 'self'; Content-Type: application/xml;charset=utf-8 Date: Thu, 11 Apr 2019 20:10:25 GMT Expires: Thu, 01 Jan 1970 00:00:00 GMT Pragma: no-cache Strict-Transport-Security: max-age=31536000; includeSubDomains Transfer-Encoding: chunked Vary: Origin, Access-Control-Request…Access-Control-Request-Method Vary: Accept-Encoding, User-Agent x-amz-version-id: 99520854223169 X-Content-Type-Options: nosniff X-DNS-Prefetch-Control: off X-Download-Options: noopen X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block
Response body
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<PostResponse>
<Location>https://finance.europe.hcp.example.com/hs3/acctg%2FChecklist.txt</Location>
<Bucket>finance</Bucket>
<Key>acctg/Checklist.txt</Key>
<ETag>"fc7abcc28fdf00ef400d7240546de4b9"</ETag>
</PostResponse>