Skip to main content
Hitachi Vantara Knowledge

Bucket and object properties

Buckets and objects have various properties that affect what you can do with them and what happens when you take action on them. Some of these properties, such as versioning for a bucket, are visible through the S3 compatible API. Others, such as object retention, aren’t visible through the S3 compatible API but still affect that API.

This section of the Help describes bucket and object properties that have an impact on the S3 compatible API.

Bucket names

When you create a bucket, you give it a name. HCP derives the hostname for the bucket from this name. The hostname is used in URLs for access to the bucket.

In English, the name you specify for a bucket must be from three through 63 characters long and can contain only alphanumeric characters and hyphens (-) but cannot start or end with a hyphen. In other languages, because the derived hostname cannot be more than 63 characters long, the name you specify may be limited to fewer than 63 characters.

Bucket names cannot contain white space or special characters other than hyphens.

Bucket names are not case sensitive. However, if you will be accessing a bucket through an interface that does not automatically convert URLs to all lowercase, you should use lowercase for all letters in the bucket name.

Bucket names cannot start with xn-- (that is, the characters x and n followed by two hyphens).

The name you give a bucket must be unique for the tenant for which you create the bucket. Different tenants can have buckets with the same name.

You can reuse bucket names that are not currently in use. So, for example, if you delete a bucket, you can give a new bucket the same name as the deleted bucket had.

Tenant administrators can change the name of a bucket any time after you create the bucket. When the name of a bucket changes, the URL for the bucket may change as well.

Object names

When you create an object, you give it a name. Object names:

  • Can contain any characters except the NULL character (ASCII 0 (zero)). This includes nonprinting characters such as spaces and line breaks.
  • Are case sensitive.
  • Can be up to 4,095 bytes long.
  • Cannot consist of a single period (.) or a single forward slash (/).

.directory-metadata is a reserved name. You cannot create an object with this name.

You cannot directly change the name of an object. However, you can effectively rename an object by making a copy of it with a different name and then deleting the original. For the object with the new name to have the same ACL as the original object, you need to specify the ACL in the copy request.

Forward slashes in object names

When using the S3 compatible API, you can view objects as being in a flat space or in a structured space:

  • In a flat space, objects are not grouped by any sort of structural elements, such as folders (also called directories), and forward slashes (/) in object names are simply part of those names.
  • In a structured space, forward slashes in object names serve as folder separators, and objects can be grouped into folders.

To support the structured view, when you use the S3 compatible API to store an object that has forward slashes in its name, HCP also creates folders and subfolders that conform to the pattern of those forward slashes. For example, if you store an object named quarterly_rpts/Q4_2019, HCP not only stores that object but also creates a folder named quarterly_rpts.

HCP does not create duplicate folders. If, after storing the object named quarterly_rpts/Q4_2019, you store an object named quarterly_rpts/Q3_2019, the single quarterly_rpts folder appears to contain two objects: Q4_2019 and Q3_2019.

Folders provide a method for organizing the objects you store in a bucket. By using meaningful names for the portions of object names from which HCP creates folders, you can more easily manage the contents of the bucket.

You can also use the S3 compatible API to create folders and subfolders by themselves. For example, you could create a folder named mktg with a subfolder named budget_proposals. If you then store an object named mktg/budget_proposals/BudgProp-2020, that object appears to be in the mktg/budget_proposals folder. Folder names follow the same rules as object names.

When you use HCP interfaces other than the S3 compatible API to view or manage objects stored through the S3 compatible API, HCP always treats forward slashes in object names as folder separators. So, for example, in the Namespace Browser, the object you stored as quarterly_rpts/Q4_2019 shows up as an object named Q4_2019 in the quarterly_rpts folder.

Grouping objects into folders not only lets you more easily manage those objects, it can also enhance HCP performance.

Object naming considerations

The following considerations apply to object names.

Names ending with a forward slash

A forward slash (/) at the end of a name indicates that the item is a folder. So, for example, if you include a forward slash at the end of the object name in a request to store an object, HCP creates an empty folder with that name and does not store the object.

Period or forward slash as an object or folder name

You cannot use a period (.) or forward slash (/) by itself as the name of an object or folder.

Object names with non-ASCII, nonprintable characters

When you store an object with non-ASCII, nonprintable characters in its name, those characters are percent-encoded in the name displayed back to you.

Regardless of how the name is displayed, the object is stored with its original name, and you can access it either by its original name or by the names with the percent-encoded characters.

Object names and access through the CIFS and NFS protocols

The CIFS and NFS protocols cannot handle object or folder names that are longer than 255 bytes. An object stored through the S3 compatible API is inaccessible through CIFS and NFS if:

  • The object name is longer than 255 bytes and does not include any forward slashes.
  • The object name includes one or more forward slashes and any part of the name is longer than 255 bytes. In this case, a part of an object name is any character string that either precedes the first forward slash, comes between two forward slashes, or follows the last forward slash.
Percent-encoding special characters

With the S3 compatible API, object names are specified in URLs. Some characters have special meaning in URLs and may be interpreted incorrectly when used for other purposes. To avoid ambiguity, percent-encode the special characters listed in the table below.

CharacterPercent-encoded values
Space%20
Tab%09
New line%0A
Carriage return%0D
+%2B
%%25
#%23
?%3F
&%26
\&5C

Percent-encoded values are not case sensitive.

UTF-8 encoding

These considerations apply to using UTF-8 encoding for object names:

  • Some character-set encoding schemes, such as UTF-8, can require more than one byte to encode a single character. As a result, such encoding can invisibly increase the length of an object name, causing it to exceed the HCP limit of 4,095 bytes.
  • When searching buckets, Hitachi Data Discovery Suite and HCP rely on UTF-8 encoding conventions to find objects by name. If the name of an object is not UTF‑8 encoded, searches for the object by name may return unexpected results.
  • When the metadata query engine or HCP search facility indexes an object with a name that includes certain characters that cannot be UTF‑8 encoded, it percent-encodes those characters. Searches for such objects by name must explicitly include the percent-encoded characters in the name.

Retention

The retention setting determines how long the object must remain in the bucket before it can be deleted. This can range from allowing the object to be deleted at any time to preventing the object from ever being deleted. While an object cannot be deleted due to retention, it is said to be under retention. You can use the S3 compatible API to change the retention setting for an object.

Retention terms

  • Default retention setting

    Each bucket in HCP has a default retention setting. This is the setting that is applied to objects when they are first stored in the bucket. When you create a bucket, its default retention setting is to allow deletion at any time.

  • Hold

    Objects can be placed on hold. An object on hold cannot be deleted. You can use the S3 compatible API to place an object on hold or release an object from hold.

  • Labeled hold

    An object can also be placed on one or more labeled holds. A labeled hold is a named retention hold that has a unique identifier. An object can have both a single, unlabeled hold and one or more labeled holds at the same time.

    An object on labeled hold cannot be deleted or have its metadata modified until all holds (labeled and unlabeled) are explicitly released. You can use either the S3 compatible API or REST API to place one or more labeled holds on an object or release an object from labeled hold.

  • Privileged delete

    Privileged delete enables you to delete objects even if they are under retention. However, you cannot use privileged delete operations to delete objects that are on hold or labeled hold, regardless of the retention settings of those objects.

  • Retention periods

    The retention period for an object is the length of time the object must remain in the bucket. A retention period can be a specific length of time, infinite time, or no time, in which case the object can be deleted at any time.

    When the retention period for an object expires, the object becomes deletable.

    Normally, if you try to delete an object that is under retention, HCP prevents you from doing so.

  • Retention classes

    A retention class is a named retention setting. Retention classes let you manage object retention consistently by specifying how long the object must remain in the repository. Retention classes cannot be created using the S3 compatible API.

  • Automatic deletion

    A bucket can be configured to automatically delete objects after their retention periods expire. For an object to be deleted automatically:

    • A specified retention period must expire. Objects with Deletion Allowed (0) or Initial Unspecified (-2) retention settings are not automatically deleted.
    • If the object is in a retention class, the class must have automatic deletion enabled.

    Automatic deletion cannot be configured using the S3 compatible API.

Object retention settings

Every object, including objects assigned to retention classes, has a specific retention setting that specifies when (or whether) the retention period for the object ends. HCP represents the retention setting in numeric and text formats, as shown in the following table.

Numeric valueText valueMeaning
0Deletion AllowedThe object can be deleted at any time.
-1Deletion ProhibitedThe object cannot be deleted, except by a privileged delete, and the retention setting cannot be changed.
-2Initial UnspecifiedThe object does not yet have a retention setting. An object that has this value cannot be deleted, except by a privileged delete. You can change this retention setting to any other setting.
An integer number of seconds since January 1, 1970, at 00:00:00 UTCA datetime valueThe object cannot be deleted, except by a privileged delete, until the specified date and time.

The format used for retention settings depends on whether you are specifying or retrieving a setting and on the protocol you are using.

Deleting an object under retention

If you have privileged permission and the bucket allows it, you can use the S3 compatible API to delete an object that is under retention. This action is called privileged delete.

You cannot use privileged delete operations to delete objects that are on hold, regardless of the retention settings of those objects.

Holding an object

If you have both write and privileged permissions, you can place an object on hold. An object on hold cannot be deleted, even by a privileged delete operation. Also, you cannot store new versions of an object on hold. Holding objects is particularly useful when the objects are needed for legal discovery.

While an object is on hold, you cannot change its retention setting. If the bucket is configured to allow changes to custom metadata for objects under retention, you can also change its custom metadata.

If you have both write and privileged permissions, you can also release an object from hold. When an object is released, its previous retention setting is again in effect.

Holding an object using labeled holds

A labeled retention hold, or labeled hold, is a named retention hold that has a unique identifier. The ability to set multiple, independent labeled holds on an object enables multiple organizations or custodians to place a hold on the same object to prevent its deletion or versioning, or metadata modification, until all holds are explicitly released. This feature is supported by both the REST API and Hitachi API for Amazon S3.

A common use of labeled holds is when an object is needed for legal discovery by multiple organizations or custodians. For example, there might be several lawsuits handled by different organizations within a company. The investigations or inquiries have different but overlapping time lines. A file relevant to all of the lawsuits must not be deleted when the first legal hold ends because the other lawsuits need the file as well.

A labeled hold can be placed on a new object at ingest time, or on an existing object in the namespace, provided the client application has write and privileged data access permissions. Similarly, the application can modify a labeled hold on an existing object.

To place a labeled hold on an object at ingest time, use the Hitachi API for Amazon S3 or REST API with the HTTP PUT object method and the X-HCP-LabelRetentionHold header.

To add or remove a labeled hold on an existing object, use the S3 compatible API with the HTTP PUT object copy method and the X-HCP-LabelRetentionHold header, or the REST API HTTP PUT method with the query parameter method to add or remove system metadata.

Retention holds and labeled retention holds on the same object

An object can have both a single hold and one or more labeled holds at the same time. These hold settings are independent of one another. All holds must be released on the object before it can be deleted or have its metadata modified, regardless of the retention setting.

Prerequisite to set privileged data access and write permissions

Before an application can place a labeled hold on an object, the application must have write and privileged data access permissions for the associated namespace.

Privileged permission can be set either programmatically by using the HCP Management API to update the dataAccessPermissions.xml file, or through the Tenant Management Console on the Users page of the Security menu.

Label limits

The following limits apply to the labeled hold feature:

  • Maximum label size

    64 characters

  • Maximum number of labeled holds per object

    100

Retention classes

A retention class is a named retention value that, when used as the retention setting for an object, specifies how long the object must remain in the repository. This value can be:

  • A duration after object creation. For example, a retention class named HlthReg-107 could have a duration of 21 years. All objects that have that class as their retention setting could not be deleted for 21 years after they’re created.
  • One of these special values:
    • Deletion Allowed (0)
    • Deletion Prohibited (-1)
    • Initial Unspecified (-2)

Retention class duration values use this format:

A+yearsy+monthsM+daysd

In this format, A represents the time at which the object was created. For example, this value specifies a retention period of one year, two months, and three days:

A+1y+2M+3d

The duration specification can omit portions with zero values. For example, this value specifies a six-month retention period:

A+6M

You can use retention classes to consistently manage data that must conform to a specific retention rule. For example, if local law requires that medical records be kept for a specific number of years, you can use a retention class to enforce that requirement.

Tenant administrators create retention classes for namespaces. When creating a class, the administrator specifies the class name, the retention setting, and whether HCP can automatically delete objects in the class when the retention period expires.

NoteAutomatic deletion must be enabled for the namespace for objects under retention to be automatically deleted.

Each namespace has its own set of retention classes. You cannot apply a retention class defined in one namespace to an object in a different namespace.

A namespace can be configured to allow administrative users to increase or decrease class retention times and delete classes. Otherwise, the retention time for a class can only be increased, and classes cannot be deleted. In all cases, any change to a retention class affects the retention period of all objects in that class.

If a retention class is deleted, the objects assigned to that class have a retention setting of Deletion Prohibited (-1) and cannot be deleted.

If a new retention class is created with the same name as a deleted retention class, existing objects in the deleted class get the retention setting of the new class.

You can assign a retention class to an existing object using any namespace access protocol. With HTTP, you can assign a retention class to an object when you store the object in the namespace, thereby overriding the default retention setting.

Retention-related request headers

The table below describes the retention-related request headers you can use with the S3 compatible API.

HeaderMethodsDescription
x-hcp-retention

PUT object

PUT object copy

POST object complete multipart upload

Specifies the retention value for the object being stored. This value can be a fixed date, an offset, a retention class, or a special value.
x-hcp-retentionhold

PUT object

PUT object copy

POST object complete multipart upload

Specifies whether the object is on hold. This value can be either true or false.
x-hcp-privileged

DELETE object

DELETE multiple objects

If you have privileged permission and the bucket allows it, this enables a privileged delete on an object that is under retention. The valid value for this header is a text string of up to 1,024 characters long, which can contain any valid UTF-8 characters, including white space.

Retention-related response headers

The table below describes the retention-related response headers that are returned with the S3 compatible API.

HeaderMethodsDescription
x-hcp-retention

GET object

HEAD object

Specifies the retention value for the object being stored. This value can be a fixed date, an offset, a retention class, or a special value.
x-hcp-retentionhold

GET object

HEAD object

Specifies whether the object is on hold. This value can be either true or false.

Specifying retention settings

You can change the retention setting for an existing object. With the S3 compatible API, you can also specify a retention setting when you store an object in a bucket. The following information describes the retention settings you can specify.

These general rules apply to values you can set:

  • If you are storing an object in a bucket with the S3 compatible API or setting retention for an existing object that is not under retention, you can specify any retention setting — past or present. If you set a time in the past, the object is immediately deletable.
  • If you are changing the setting for an existing object that is in a retention class, you can replace the class with another class, as long as the duration of the replacement class is not shorter than the duration of the original class. You cannot replace the class with an explicit retention setting, such as Deletion Prohibited or a datetime value.
  • If you are changing the setting for an existing object that has an explicit datetime retention setting, you can change the setting to lengthen the retention period but not to shorten it.

The list below describes the values you can use to specify the retention setting for an object and contains rules for specifying these values. These values are not case sensitive.

  • 0 (zero) or Deletion Allowed

    Allows the object to be deleted at any time. You can assign this value to an object only when you add it to the namespace or when its retention setting is -2.

    The value -0 is equivalent to 0 (zero).

  • -1 or Deletion Prohibited

    Prevents the object from being deleted and its retention setting from being changed. You can assign this value to an object that is not in a retention class at any time.

    If an object is assigned to a retention class and that class is then deleted, the retention setting for that object changes to -1.

  • -2 or Initial Unspecified

    Specifies that the object does not yet have a retention setting. You can assign this value to an object when you add the object to the namespace. You can also directly change the retention setting for an object from 0 to ‑2.

    You can change ‑2 to any other retention setting.

  • datetime

    Prevents the object from being deleted until the specified date and time. You can assign this value to an object that is not in a retention class if the specified date and time is later than the current retention setting for the object. You cannot assign it to an object for which the current retention setting is -1.

  • offset

    Specifies a period for which to retain the object. You can assign this value to an object that is not in a retention class at any time, except when its current retention setting is -1.

    HCP calculates the retention setting date and time from the offset and one of these:

    • The time the object was added to the namespace
    • The current retention setting
    • The current time
  • C+retention_class_name

    Assigns the object to a retention class.

    You can assign this value to an object if any one of these is true:

    • The original retention period for the object has expired.
    • The original retention period for the object has not expired, and the retention class results in a retention period that is longer than the current retention period.
    • The retention setting for the object is 0 or -2.
    • The retention setting for the object is -1, and the class has a retention setting of -1.
    • The object is in a retention class, and the duration of the new class is not shorter than the duration of the original class.
    • The retention class you assign must already be defined for the namespace.

Specifying a date and time

You can set retention by specifying a date and time in either of these formats:

  • Time in seconds since January 1, 1970, at 00:00:00 UTC. For example:
    1450137600

    The calendar date that corresponds to 1450137600 is Tuesday, December 15, 2015, at 00:00:00 EST.

  • Date and time in this ISO 8601 format:
    yyyy-MM-ddThh:mm:ssZ

    In this format, Z represents the offset from UTC and is specified as:

    (+|-)hhmm

    For example, 2015-11-16T14:27:20-0500 represents the start of the 20th second into 2:27 PM, November 16, 2015, EST.

If you specify certain forms of invalid dates, HCP automatically adjusts the retention setting to make a real date. For example, if you specify 2015-11-33, which is three days past the end of November, HCP changes it to 2015-12-03.

Specifying an offset

You can set retention by specifying an offset from:

  • The time at which the object was added to the namespace
  • The current retention setting for the object
  • The current time

Because you can only extend a retention period, the offset must be a positive value.

Offset syntax

To use an offset as a retention setting, specify a standard expression that conforms to this syntax:

^([RAN])?([+-]\d+y)?([+-]\d+M)?([+-]\d+w)?([+-]\d+d)?([+-]\d+h)?([+-]\d+m)?([+-]\d+s)?

The list below explains this syntax.

  • ^

    Start of the expression

  • ( )

    Sequence of terms treated as a single term

  • ?

    Indicator that the preceding term is optional

  • [ ]

    Group of alternatives, exactly one of which must be used

  • +

    Plus

  • -

    Minus

  • R*

    The current retention setting for the object

  • A*

    The time at which the object was added to the namespace

  • N*

    The current time

  • d+

    An integer in the range 0 (zero) through 9,999

  • y

    Years

  • M

    Months

  • w

    Weeks

  • d

    Days

  • h

    Hours

  • m

    Minutes

  • s

    Seconds

Offset examples

Here are some examples of offset values used to extend a retention period:

  • This value sets the retention value to 100 years past the time when the object was stored:
    A+100y
  • This value sets the end of the retention period to 20 days minus five hours past the current date and time:
    N+20d-5h
  • This value extends the current retention period by two years and one day:
    R+2y+1d

Custom metadata

Objects can optionally have custom metadata in the form of one or more annotations. Annotations are a powerful means for understanding and analyzing the contents of buckets. Using the HCP metadata query API or Search Console, you can search for objects based on the content of their annotations.

Storing custom metadata with the S3 compatible API

With the S3 compatible API, you use x-amz-meta- request headers to specify custom metadata. You can use these headers when you store or copy an object.

With the x-amz-meta- header, you specify custom metadata as property/value pairs. You append the property name to the header and specify the value of the property as the value of the header. For example, to give an object a department property with a value of Sales&Mktg and a year property with a value of 2020, you would specify these headers:

x-amz-meta-department: Sales&Mktg
x-amz-meta-year: 2020

HCP stores the custom metadata you specify with the S3 compatible API as an annotation named .metapairs. In this annotation, the property/value pairs are stored as well-formed XML in which each property is represented by an element. For example, the XML stored for the headers shown above is:

<metapairs version="600">
     <meta-department><![CDATA[Sales&Mktg]]></meta-department>
     <meta-year><![CDATA[]]></meta-year>
</metapairs>2020

The root element in the .metapairs annotation is metapairs.

For each property/value pair, the name of the corresponding element is the concatenation of meta- and the property name, modified if necessary to be a valid XML element name. Valid XML element names can contain alphanumeric characters, periods (.), hyphens (-), underscores (_), and colons (:). When creating element names from property names, HCP changes any other character to an underscore. For example, the property name city/town becomes the element name city_town.

For each property/value pair, the property value becomes the value of the corresponding element. This value is enclosed in a CDATA section.

If you specify an x-amz-meta- header with no value, HCP doesn’t store an element for the property named in the header. If all the x-amz-meta- headers you specify have no value, HCP doesn’t create a .metapairs annotation for the object.

In a request to store or copy an object, you can specify the same x-amz-meta- property multiple times with different values. In the .metapairs annotation XML, these values are stored as comma-separated values for a single element.

Here’s an example that shows three occurrences of the same property along with the resulting XML:

x-amz-meta-author: P.D. Grey
x-amz-meta-author: Morgan White
x-amz-meta-author: Paris Black

<metapairs>
     <meta-author>
          <![CDATA[P.D. Grey,Morgan White,Paris Black]]>
     </meta-author>
</metapairs>

Property names are case sensitive, so names that differ only in case correspond to separate XML elements. For example, these x-amz-meta- headers result in three separate XML elements: x-amz-meta-date_written, x-amz-meta-Date_Written, and x-amz-meta-DATE_WRITTEN.

Retrieving custom metadata with the S3 compatible API

When you check the existence of or retrieve an object that has a .metapairs annotation containing well-formed XML, the response headers include x-amz-meta- headers with property/value pairs that correspond to the XML elements in the annotation. The property names in these headers are the element names with the initial meta- removed.

An element with comma-separated values is returned as a single x-amz-meta- header with comma-separated values. Here’s an example that shows an XML element with comma-separated values and the x-amz-meta- header that results from that element:

<meta-author>
     <![CDATA[P.D. Grey,Morgan White,Paris Black]]>
</meta-author>

x-amz-meta-author:P.D. Grey,Morgan White,Paris Black

Custom metadata usage considerations

The following considerations apply to using custom metadata with the S3 compatible API.

Custom metadata and object versions

Custom metadata is specific to the object version for which it's stored:

  • If you store an object without custom metadata and then store a new version of the object with custom metadata, the custom metadata is not added to the old version of the object.
  • If you store an object with custom metadata and then store a new version of the object without custom metadata, the old version of the object has custom metadata, but the new version does not.

You cannot add or replace custom metadata for an old version of an object.

Property names

When naming custom metadata properties, you should use names that, when concatenated with meta-, result in valid XML element names. That way, the x-amz-meta- headers returned when you retrieve or check the existence of an object match the x-amz-meta- headers you specified when you stored or copied the object.

If HCP has to modify a property name to create a valid element name, the returned x-amz-meta- header won’t match the x-amz-meta- header specified when the object was stored or copied. For example, if the specified header is x-amz-meta-city/town, the returned header is x-amz-meta-city_town.

.
Custom metadata size

When you use the S3 compatible API to store or copy an object, you can specify at most two kilobytes of custom metadata. The size of the custom metadata you specify is the sum of the number of bytes in the UTF-8 encoding of each property name and value.

Allowed operations

Whether you can add, replace, or delete custom metadata for objects under retention depends on a bucket setting. When you create a bucket, it’s set to allow only the addition of custom metadata for objects under retention. You cannot use the S3 compatible API to change this setting. However, tenant administrators can change this setting for the buckets you create.

.metapairs annotations with unexpected content

You should not use HCP interfaces other than the S3 compatible API to store annotations named .metapairs. However, HCP does not prevent you from doing this. As a result, annotations named .metapairs are not guaranteed to be compatible with the S3 compatible API.

Here are some ways in which HCP responds to S3 compatible requests for objects that have .metapairs annotations with unexpected content:

  • If the .metapairs annotation doesn’t contain valid XML or if the first line in the annotation doesn’t begin with the metapairs element, HCP returns an x-amz-missing-meta header with a value of 1 (one) and does not return any x-amz-meta- headers.
  • If an element name doesn’t start with meta-, HCP doesn’t return an x-amz-meta- header for the element.
  • If a meta- element has no value, HCP doesn’t return an x-amz-meta- header for the element.
  • If a meta- element has an attribute, HCP ignores the attribute and returns the applicable x-amz-meta- header.
  • If the XML contains nested elements and the lowest-level element is a meta- element, HCP returns an x-amz- header for that element. It does not return x-amz- headers for any other elements in that nested structure.

Bucket owners

Buckets can have owners that correspond to HCP or AD user accounts. By default, when using an HCP user account, you own the buckets you create through the S3 compatible API. With an AD user account, you cannot use the S3 compatible API to create buckets.

Normally, as the owner of a bucket, you can use the S3 compatible API to view and change the versioning status of the bucket and to delete the bucket if it’s empty. If you have write ACL and change owner permissions for a bucket you own, you can use the S3 compatible API to change the bucket owner to a different user. However, tenant administrators can reconfigure user accounts such that users using those accounts cannot manage the buckets they own.

With the S3 compatible API, you use an ACL to change the owner of a bucket.

Tenant administrators can take the same actions on a bucket as the bucket owner can. Additionally, tenant administrators can change a bucket to have no owner.

The maximum number of buckets you can own is limited by a tenant-level configuration setting.

Object owners

Like buckets, objects can have owners that correspond to HCP or AD user accounts. By default, if you’re an authenticated user, you own the objects you create in a bucket, regardless of whether you own the bucket. As the owner of the object, you have read, read ACL, write, write ACL, and delete permissions for that object.

If you’re not an authenticated user, the objects you create have no owner.

If you have write ACL and change owner permissions for a bucket, you can use the S3 compatible API to change the owner of any object in that bucket to a different user. To change the owner of an object, you use an ACL.

Users with the change owner data access permission for a bucket can use the HTTP protocol to change the owner of any object in the bucket to a different user. They can also change any object in the bucket to have no owner.

Access control lists

HCP supports access control lists (ACLs) for both buckets and objects. An ACL grants specified users or groups of users permissions to perform specific operations. An ACL can also be used to change the owner of a bucket or object.

A bucket ACL grants permissions to perform operations on a bucket and on all objects in the bucket. For example, an ACL for a bucket could give all users read permission for that bucket. All users in that group would then be able to retrieve all the objects in that bucket.

An object ACL grants permissions to perform operations on an individual object. For example, an ACL for an object could give a specified user write ACL permission for that object. That user would then be able to change the ACL for that object regardless of whether the user had write ACL permission for the bucket that contained the object.

You can add an ACL to a bucket when you create the bucket or in a separate operation. You can add an ACL to an object when you create or copy the object or in a separate operation. When you add an ACL to an existing bucket or object that already has an ACL, the new ACL replaces the old one in its in entirety.

ACLs can be added to buckets and objects through other HCP interfaces. However, regardless of how they are added, they apply to all HCP interfaces that provide access to objects.

An ACL added through the S3 compatible API can include at most one hundred permission grants. ACLs added through other HCP interfaces can include more than that. If you retrieve an ACL with more than one hundred grants, HCP returns only the first hundred.

Whether objects in a bucket can have ACLs and whether those ACLs are enforced depend on bucket settings. When you use the S3 compatible API to create a bucket, the use of ACLs is automatically enabled. This setting cannot be disabled through any HCP interface.

Also when you use the S3 compatible API to create a bucket, ACLs are automatically set to be enforced. When enforcing ACLs, HCP honors the permission grants in object ACLs. When ACLs are not enforced, HCP ignores those grants. HCP always honors permission grants in bucket ACLs.

You cannot use the S3 compatible API to change the ACL enforcement setting. However, tenant administrators can use other HCP interfaces to change this setting.

ACL permissions

Granting a permission in an ACL for a bucket gives the grantee certain data access permissions for that bucket. Granting a permission in an ACL for an individual object gives the user certain data access permissions just for that object.

The list below lists the permissions you can grant in an ACL and shows the data access permissions that correspond to each ACL permission.

  • Read

    Browse and read

  • Read ACP

    Read ACL

  • Write

    Write and delete

  • Write ACP

    Write ACL

  • Full control

    Browse, read, read ACL, write, write ACL, and delete

By default, a bucket or object owner that corresponds to an HCP user account or an object owner that corresponds to an AD user account has full control over the applicable bucket or object. For a bucket owner that corresponds to an AD user account, the permissions depend on the tenant configuration.

When adding an ACL to a bucket or object, you can grant only the permissions you already have for that bucket or object. For example, suppose you have read, read ACP, and write ACP permissions for an object. In this case, you can grant read, read ACP, and write ACP permissions for the object to other users, but you cannot grant write permission or full control.

Tenant administrators can change the permissions that users, including the bucket owner, have for a bucket. They cannot change the permissions users have for objects.

ACL grantees

An ACL can grant permissions to individual users or to groups of users. An individual user is represented by either an HCP user account or, for object ACLs only, an AD user account. A group can be either all authenticated users or all users (both authenticated and anonymous).

To specify an HCP user account, you can use either the account username or the account user ID. To specify an AD user account, you can use either the account user name followed by an at sign (@) and the AD domain name (for example, sgold@ad-1.example.com) or the security ID (SID) for the account.

To specify the group of all authenticated users, you can use either the name authenticated or this URI:

http://acs.amazonaws.com/groups/global/AuthenticatedUsers

To specify the group of all users, you use either the name all_users or this URI:

http://acs.amazonaws.com/groups/global/AllUsers

The names authenticated and all_users are case sensitive. In the URIs, AuthenticatedUsers and AllUsers are case sensitive.

Canned ACLs

When specifying an ACL for a bucket or object, you can use a canned ACL instead of specifying permission grants individually. A canned ACL is a predefined set of grants of permissions.

HCP has these canned ACLs:

  • private

    Grants full control to the bucket or object owner

  • public-read

    Grants full control to the bucket or object owner and read permission to all users

  • public-read-write

    Grants full control to the bucket or object owner and read and write permissions to all users

  • authenticated-read

    Grants full control to the bucket or object owner and read permission to all authenticated users

  • bucket-owner-read

    Grants full control over the object to the object owner and read permission to the bucket owner

  • bucket-owner-full-control

    Grants full control over the object to the bucket owner and object owner

Specifying ACLs

To specify an ACL in an S3 compatible request to create a bucket or store or copy an object, you use request headers. To specify an ACL in a separate operation, you can either use request headers or specify the grants in the request body.

You can also use an ACL request body to change the owner of a bucket or object. You cannot do that with request headers.

Specifying an ACL with headers

Using request headers, you can specify either a canned ACL or individual ACL grants of permissions. You cannot specify both a canned ACL and individual grants in the same request.

Using a canned ACL

To specify a canned ACL, you use the x-amz-acl request header. The value of this header can be the name of any one of the canned ACLs. These names are case sensitive.

Here’s a sample x-amz-acl header that specifies the canned ACL named authenticated-read:

x-amz-acl: authenticated-read
Using individual grant headers

To grant specific permissions to specific users or groups, you use these headers:

x-amz-grant-read
x-amz-grant-read-acp
x-amz-grant-write
x-amz-grant-write-acp
x-amz-grant-full-control

Each header grants the permission indicated by the header itself.

The value for any of these headers is a comma-separated list of one or more grantees, in this format:

identifier-type=grantee-identifier

The list below lists the identifier types and indicates how you identify the grantee with each type.

  • id

    User ID of an HCP user account or, for object ACLs only, SID of an AD user account.

    To learn the ID or SID for a user account, see your tenant administrator.

  • emailAddress

    One of these:

    • Username of an HCP user account
    • For object ACLs only, username of an AD user account followed by an at sign (@) and the AD domain name
    • authenticated
    • all_users

    When specifying a username, percent-encode non-ASCII characters and reserved special characters such as ampersands (&), commas (,) and equal signs (=). If a username contains spaces, enclose it in quotation marks.

    Third-party tools that are compatible with the Hitachi API for Amazon S3 may not be able to handle usernames with non-ASCII characters, special characters, or spaces. When using such tools, identify the user by user ID rather than by username.

  • uri

    URI for the group of all authenticated users or the group of all users

Identifier types are case sensitive.

Here’s a sample x-amz-grant-write header that grants write permission to two users who are identified by their HCP user account IDs:

x-amz-grant-write: id=53344e3b-00de-4941-962e-827ac143fa84,
     id=53344e3b-00de-494e-962e-827ac143fa84

Here's a sample x-amz-grant-read header that grants read permission to all users:

x-amz-grant-read: uri=http://acs.amazonaws.com/groups/global/AllUsers

If you include the same header multiple times in a single request, HCP uses only the first one.

Specifying an ACL in the request body

An ACL request body can specify one or more permission grants and/or an owner for the bucket or object. If the specified owner is not the current owner, the owner changes to the specified owner (provided that you change owner permission for the bucket).

For the content of an ACL request body, you use XML in this format:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
To specify the current owner or change the owner, include the Owner element.
  <Owner>
    <ID>user-id</ID>
    <DisplayName>username</DisplayName>
  </Owner>
  <AccessControlList>
Include one Grant element for each combination of grantee and permission.
    <Grant>
      <Grantee identifier-type
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
To identify the grantee, use either the ID and, optionally, DisplayName elements, the URI element, or the EmailAddress element.
        <ID>user-id</ID>
        <DisplayName>username</DisplayName>
        <URI>group-uri</URI>
        <EmailAddress>username</EmailAddress>
      </Grantee>
      <Permission>permission</Permission>
    </Grant>
  </AccessControlList>
</AccessControlPolicy>

The list below describes XML elements in an ACL request body. The elements are listed in alphabetical order.

  • AccessControlList

    Child of the AccessContolPolicy element and container for zero or more grants of permissions to individual users or groups.

    Each grant is represented by a Grant element.

    The AccessControlList element is required in an ACL request body.

  • AccessControlPolicy

    Root element. This must be the first element in the ACL request body.

    The AccessControlPolicy element must include this XML namespace specification:

    xmlns="http://s3.amazonaws.com/doc/2006-03-01/"
  • DisplayName

    Child of the Owner element or of the Grantee element when the identifier type is CanonicalUser.

    The value of the DisplayName element can be:

    • User name of an HCP user account
    • User name of an AD user account followed by an at sign (@) and the AD domain name
    • authenticated
    • all_users

    The DisplayName element is optional and ignored.

  • EmailAddress

    Child of the Owner element or of the Grantee element when the identifier type is AmazonCustomerByEmail.

    The value of the DisplayName element can be:

    • User name of an HCP user account
    • For object ACLs only, username of an AD user account followed by an at sign (@) and the AD domain name
  • Grant

    Child of the AccessControlList element and container for the Grantee and Permission elements, which can occur in either order.

    Each occurrence of the Grant element grants one permission to one grantee.

  • Grantee

    Child of the Grant element and container for the grantee identifier.

    The Grantee element must include this XML namespace specification:

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

    The Grantee element must also include one of these specifications of identifier type to indicate how the grantee is identified:

    • xsi:type="CanonicalUser"

      The grantee is identified by the ID and, optionally, the DisplayName element. If present, the DisplayName element is ignored.

    • xsi:type="Group"

      The grantee is identified by the URI element.

    • xsi:type="AmazonCustomerByEmail"

      The grantee is identified by the EmailAddress element.

  • ID

    Child of the Owner element or of the Grantee element when the identifier type is CanonicalUser.

    The value of the ID element can be the user ID of an HCP user account or, for object ACLs only, the SID of an AD user account.

    The ID element is required in the context of the Owner element and in the context of the Grantee element when the identifier type in the Grantee element is CanonicalUser.

    To learn the ID or SID for a user account, see your tenant administrator.

  • Owner

    Child of the AccessControlPolicy element and container for the owner identifier.

    The owner is identified by the ID and, optionally, DisplayName elements.

    The Owner element is optional in an ACL request body. If you omit it, the bucket or object owner does not change.

  • Permission

    Child of the Grant element. Valid values for the Permission element are:

    • READ
    • READ_ACP
    • WRITE
    • WRITE_ACP
    • FULL_CONTROL

    These values are case sensitive.

  • URI

    Child of the Owner element or of the Grantee element when the identifier type is Group.

    Valid values for the URI element are the URI for the group of all authenticated users and the URI for the group of all users.

Here’s a sample ACL that sets the owner to the user named lgreen and grants read permission to all users and write permission to the user named pdgrey:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
     <Owner>
          <ID>53344e3b-00de-494b-962e-827ac143fa84</ID>
          <DisplayName>lgreen</DisplayName>
     </Owner>
     <AccessControlList>
          <Grant>
               <Grantee xsi:type="Group"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
                    <URI>http://acs.amazonaws.com/groups/global/AllUsers</URI>
               </Grantee>
               <Permission>READ</Permission>
          </Grant>
          <Grant>
               <Grantee xsi:type="AmazonCustomerByEmail"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
                    <EmailAddress>pdgrey</EmailAddress>
               </Grantee>
               <Permission>WRITE</Permission>
           </Grant>
     </AccessControlList>
</AccessControlPolicy>

Removing an ACL

You can remove an ACL from a bucket or object by removing all grants from it. To do this by using headers, use the x-amz-acl header with the canned ACL named private, like this:

x-amz-acl: private

To remove an ACL by using an ACL request body, omit all grants from the request the body, like this:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
     <AccessControlList>
     </AccessControlList>
</AccessControlPolicy>

Removing an ACL does not remove full control from the bucket or object owner.

Versioning

Versioning is an optional bucket feature that enables the creation and management of multiple versions of an object. With versioning enabled, when you store an object with the same name as an existing object, a new version of the object is created. If versioning is not enabled, you cannot store an object that has the same name as an existing object.

HCP does not create new versions of objects that are under retention or on hold. An attempt to store a new version of such an object results in an error.

Enabling and disabling versioning

You can use the S3 compatible API to enable, disable, and view the status of versioning for buckets you own. Tenant administrators can also take these actions on buckets you own. Whether versioning is initially enabled when you create a bucket is determined by a tenant-level configuration setting.

NoteTenants can be configured not to allow versioning at all. If the tenant is configured this way, you cannot enable versioning for your buckets.

You cannot enable versioning for a bucket while the WebDAV, CIFS, NFS, or SMTP protocol is enabled for that bucket. You can disable versioning for a bucket at any time.

Disabling versioning does not cause old versions of objects to be deleted. However, you cannot store new versions while versioning is disabled. A request to retrieve an old version of an object while versioning is disabled returns the current version.

Object versions

Each version of an object is an object in its own right. It has system metadata and can have custom metadata and an ACL. However, you can change metadata only on the current version of an object. Changing metadata, other than the owner or ACL, has no effect on old versions of the object. Changes to the object owner apply to all versions of the object. Changes to object ACLs can apply to specific versions or all versions of the object.

Version IDs

All objects, including those created while versioning is not enabled, have version IDs. Version IDs are integers. Each time a new version of an existing object is created, that new version is assigned an ID that is greater than the ID of the previous version of the object. The IDs for multiple versions of an object are not necessarily consecutive numbers.

Version IDs are unique for the versions of a given object but are not necessarily unique across the versions of all objects.

HCP also assigns version IDs to folders. However, folders cannot have multiple versions.

When you store an object while versioning is enabled, HCP returns the version ID of the object in the x-amz-version-id response header. When you store an object while versioning is disabled, the response headers do not include x-amz-version-id.

Version pruning

If versioning is enabled for a namespace, pruning may also be enabled. Pruning is the automatic deletion of old versions of objects after a specific length of time since their creation. HCP does not prune old versions of objects that are on hold.

Whether pruning is initially enabled when you create a bucket is determined by a tenant-level configuration setting. You cannot use the S3 compatible API to change the pruning setting for a bucket. However, tenant administrators can use other HCP interfaces to do this.

Allocated space

When you create a bucket, the bucket is allocated a certain amount of space. This is the amount of space available for storing objects in the bucket.

The initial amount of space allocated to a bucket is determined by a tenant-level configuration setting. You cannot use the S3 compatible API to change the amount of space allocated to a bucket. However, tenant administrators can use other HCP interfaces to do this.

Search

Buckets can be enabled for search. If a bucket is search enabled, users with the search data access permission can use the HCP metadata query API or Search Console to search for objects in the bucket. If a bucket is not search enabled, users cannot search the bucket.

Whether search is initially enabled when you create a bucket is determined by a tenant-level configuration setting. You cannot use the S3 compatible API to enable or disable search for a bucket. However, tenant administrators can use other HCP interfaces to take these actions.

Data access permission masks

A data access permission mask determines which operations on objects are allowed in a bucket. If the permission mask does not include the permission to perform a particular operation, you cannot perform that operation, regardless of your data access permissions for the bucket or target object.

Data access permission masks are set at the system, tenant, and bucket level. The effective permission mask for a bucket allows only the operations that are allowed at all three levels.

For example, for you to be able to delete an object in a bucket:

  • The system-level permission mask must include the delete permission
  • The tenant-level permission mask must include the delete permissions
  • The permission mask for the bucket must include the delete permission
  • Either of these must be true:
    • Your data access permissions for the bucket include delete.
    • You have delete permission for the target object either because you are the object owner or because the object has an ACL that grants you delete permission.

When you create a bucket, its data access permission mask allows all operations. Tenant administrators can change the data permission mask for the buckets you create. You cannot use the S3 compatible API to change the permission mask for a bucket.

Tenant administrators can also change the tenant-level permission mask, and HCP system administrators can change the system-level permission mask. Changes to the permission mask at any level may affect which operations you can perform with the S3 compatible API.

Replication collisions

If users can write to multiple systems in a replication topology, collisions can occur when different changes are made to the same objects on different systems. The way HCP handles collisions that occur due to replication depends on the type of collision. However, the general rule is that more recent changes have priority over conflicting less recent changes.

Object content collisions

An object content collision occurs when, for a namespace without versioning enabled, these events occur in the order shown:

  1. An object is created with the same name in that namespace on two systems in a replication topology, but the object has different content on the two systems.
  2. The object on one of the systems is replicated to the other system.

If versioning is enabled, no collision occurs. Instead, the less recently created of the two objects becomes an old version of the more recently created object.

When an object content collision occurs, the more recently created object keeps its name and location. The other object is either moved to the .lost+found directory in the same namespace or renamed, depending on the namespace configuration.

When HCP moves an object to the .lost+found directory, the full object path becomes .lost+found/replication/system-generated-directory/ old-object-path.

When renaming an object due to a content collision, HCP changes the object name to object-name.collision or object-name.version-id.collision, where version-id is the version ID of the object. HCP uses the second format only if versioning has ever been enabled for the namespace that contains the object but is not currently enabled.

If the new name is already in use, HCP changes the object name to object-name.1.collision or object-name.version-id.1.collision, as applicable. If that name is already in use, HCP successively increments the middle integer by one until a unique name is formed.

Objects that have been relocated or renamed due to content collisions are flagged as replication collisions in their system metadata. Clients can use the metadata query API to search for objects that are flagged as replication collisions.

If an object that’s flagged as a replication collision changes (for example, if its retention period is extended), its collision flag is removed. If a client creates a copy of a flagged object with a new name, the collision flag is not set on the copy.

Depending on the bucket configuration, objects flagged as replication collisions may be automatically deleted after a set number of days. The days are counted from the time the collision flag is set. If the collision flag is removed from an object, the object is no longer eligible for automatic deletion.

You cannot use the S3 compatible API to change the way HCP handles objects that are flagged as replication collisions. However, tenant administrators can change this configuration for the buckets you create.

Custom metadata collisions

A custom metadata collision occurs when these events occur in the order shown:

  1. One of these changes occurs:
    • Custom metadata is added to a given object on each of two systems in a replication topology, but the added custom metadata is different on the two systems.

      The addition of custom metadata to an object on only one of the systems does not result in a custom metadata collision. Instead, the new custom metadata is replicated from that system to the other system without conflict.

    • The custom metadata for a given object is replaced on each of two systems in a replication topology, but the replacement custom metadata is different on the two systems.
    • The custom metadata for a given object is replaced on one system in a replication topology, and the same custom metadata is removed on another system in the topology.
  2. The change made on one of the systems is replicated to the other system.

Custom metadata is treated as a single unit. If a collision occurs when a custom metadata change for a given object is replicated from one system (system A) in a replication topology to another system (system B) in the topology:

  • If the last change on system A is more recent than the last change on system B, HCP applies the change from system A to the custom metadata on system B
  • If the last change on system B is more recent than the last change on system A, HCP does not change the custom metadata on system B

For example, suppose a given object starts out with the same custom metadata on system A and system B. The table below shows a sequence of events in which the custom metadata for the object is changed and the change is then replicated.

  1. On system B, a client replaces the custom metadata for the object with new custom metadata.
  2. On system A, a client replaces the custom metadata for the object with different custom metadata from the custom metadata used on system B.
  3. The change on system A is replicated to system B. The resulting custom metadata for the object on system B is the new custom metadata from system A.

Access control list collisions

An ACL collision occurs when these events occur in the order shown:

  1. Different changes are made to the ACL for a given object on each of two systems in a replication topology.
  2. The changed ACL on one of the systems is replicated to the other system.

An ACL is treated as a single unit. If a collision occurs when a changed ACL for a given object is replicated from one system (system A) in a replication topology to another system (system B) in the topology:

  • If the last change to the ACL on system A is more recent than the last change to the ACL on system B, HCP changes the ACL on system B to match the changed ACL on system A
  • If the last change to the ACL on system B is more recent than the last change to the ACL on system A, HCP does not change the ACL on system B
For example, suppose the ACL for a given object starts out with these grants on both system A and system B:
  • All users: read
  • User lgreen: write
  • User mwhite: write, delete

The list below shows a sequence of events in which the ACL for the object is changed and the change is then replicated.

  1. On system B, a client changes the grants in the ACL to:
    • All users: read
    • User lgreen: write, delete
    • User pdgrey: write
  2. On system A, a client changes the grants in the ACL to:
    • All users: read
    • User mwhite: write
    • User pdgrey: write
  3. The changed ACL on system A is replicated to system B. The resulting ACL for the object on system B contains these grants:
    • All users: read
    • User mwhite: write
    • User pdgrey: write

 

  • Was this article helpful?