About the Content Software for File system

The Content Software for File provides a number of unique features and functions.

Parent Topic

• Basic Content Software for File system deployment

Child Topics

• Protection
• Distributed network scheme
• Failed component replacement as a background process
• Failure domains
• Prioritized data rebuild process
• Seamless distribution, scale, and performance
• Data reduction

The net capacity of the Content Software for File system is obtained after the following three deductions performed during configuration:

1. Level of protection required, which is the amount of storage capacity to be dedicated for system protection.
2. Hot spare(s), that is the amount of storage capacity to be set aside for redundancy and to allow for rebuilding following a component failure.
3. Content Software for File filesystem overhead, in order to improve overall performance.

Parent Topic

• SSD capacity management

SSDNetCapacity = 10TB * (10-1) / 10 * 6/(6+2) * 0.9 = 6.075TB

SSDNetCapacity = 20TB * (20-2) / 20 * 16/(16+2) * 0.9 = 14.4TB

Parent Topic

• SSD capacity management

A Content Software for File filesystem is similar to a regular on-disk filesystem while distributed across all the hosts in the cluster. Consequently, filesystems are not associated with any physical object in the Content Software for File system and act as root directories with space limitations.

The system supports a total of up to 1024 filesystems. All of which are equally balanced on all SSDs and CPU cores assigned to the system. This means that the allocation of a new filesystem or resizing a filesystem are instant management operations performed without any constraints.

A filesystem has a defined capacity limit and is associated with a predefined filesystem group. A filesystem that belongs to a tiered filesystem group must have a total capacity limit and an SSD capacity cap. All filesystems' available SSD capacity cannot exceed the total SSD net capacity.

Parent Topic

• Filesystems, object stores, and filesystem groups

Parent Topic

• Filesystems, object stores, and filesystem groups

Parent Topic

• Filesystems, object stores, and filesystem groups

Both data at rest (residing on SSD and object store) and data in transit can be encrypted. This is achieved by enabling the filesystem encryption feature. A decision on whether a filesystem is to be encrypted is made when creating the filesystem.

To create encrypted filesystems, deploy a Key Management System (KMS).

Parent Topic

• Filesystems, object stores, and filesystem groups

In addition to the capacity limitation, each filesystem has a limitation on the amount of metadata. The system-wide metadata limit is determined by the SSD capacity allocated to the Content Software for File system and the RAM resources allocated to the Content Software for File system processes.

The Content Software for File system keeps tracking metadata units in the RAM. If it reaches the RAM limit, it pages these metadata tracking units to the SSD and alerts. This leaves enough time for the administrator to increase system resources, as the system keeps serving IOs with a minimal performance impact.

By default, the metadata limit associated with a filesystem is proportional to the filesystem SSD size. It is possible to override this default by defining a filesystem-specific max-files parameter. The filesystem limit is a logical limit to control the specific filesystem usage and can be updated by the administrator when necessary.

The total metadata limits for all the filesystems can exceed the entire system metadata information that can fit in the RAM. For minimal impact, in such a case, the least-recently-used units are paged to disk, as necessary.

Parent Topic

• Filesystems, object stores, and filesystem groups

Child Topics

• Metadata units calculation

In the Content Software for File system, object stores represent an optional external storage media, ideal for storing warm data. Object stores used in tiered Content Software for File system configurations can be cloud-based, located in the same location (local), or at a remote location.

Content Software for File supports object stores for tiering (tiering and local snapshots) and backup (snapshots only). Both tiering and backup can be used for the same filesystem.

Using object store buckets optimally is achieved when a cost-effective data storage tier is required at a price point that cannot be satisfied by server-based SSDs.

An object store bucket definition contains the object store DNS name, bucket identifier, and access credentials. The bucket must be dedicated to the Content Software for File system and not be accessible by other applications.

Filesystem connectivity to object store buckets can be used in the data lifecycle management and Snap-to-Object features.

Parent Topic

• Filesystems, object stores, and filesystem groups

In the Content Software for FileContent Software for File system, filesystems are grouped into a maximum of eight filesystem groups.

Each filesystem group has tiering control parameters. While tiered filesystems have their object store, the tiering policy is the same for each tiered filesystem under the same filesystem group.

Parent Topic

• Filesystems, object stores, and filesystem groups

The Content Software for File system supports the following types of networking technologies:

• InfiniBand (IB)
• Ethernet‌

The currently-available networking infrastructure dictates the choice between the two. If a Content Software for File cluster is connected to both infrastructures, it is possible to connect Content Software for File clients from both networks to the same cluster.

The Content Software for File system networking can be configured either as performance-optimized, where the CPU cores are dedicated to Content Software for File and the use of DPDK networking takes place and cores, or, as CPU-optimized where cores are not dedicated and we use either DPDK (when supported by the NIC drivers) or in-kernel networking (UDP mode).

Parent Topic

• networking

For performance-optimized networking, the Content Software for File system does not use standard kernel-based TCP/IP services, but a proprietary infrastructure based on the following:

• Use of DPDK to map the network device in the user space and make use of the network device without any context switches and with zero-copy access. This bypassing of the kernel stack eliminates the consumption of kernel resources for networking operations and can be scaled to run on multiple hosts. It applies to both backend and client hosts and enables the Weka system to fully saturate 200 GB links.
• Implementation of a proprietary Content Software for File protocol over UDP, meaning that the underlying network may involve routing between subnets or any other networking infrastructure that supports UDP.

The use of DPDK delivers operations with extremely low-latency and high throughput. Low latency is achieved by bypassing the kernel and sending and receiving packages directly from the NIC. High throughput is achieved because multiple cores in the same host can work in parallel, without a common bottleneck.

Before proceeding, it is important to understand several key terms used in this section, namely DPDK, SR-IOV.

Parent Topic

• networking

Child Topics

• DPDK
• SR-IOV

For CPU-optimized networking Content Software for File can yield CPU resources to other applications. That is useful when the extra CPU cores are needed for other purposes. However, the lack of CPU resources dedicated to the Weka system comes with the expense of reduced overall performance.

Parent Topic

• networking

Child Topics

• DPDK without core dedication
• UDP mode

In the Content Software for File system, data can be stored on two forms of media:

1. On locally-attached SSDs, which are an integral part of the Content Software for File system configuration.
2. On object-store systems external to the Content Software for File system, which are either third-party solutions, cloud services, or part of the Content Software for File system.

The Content Software for File system can be configured either as an SSD-only system or as a data management system consisting of both SSDs and object stores. By nature, SSDs provide high performance and low latency storage, while object stores compromise performance and latency but are the most cost-effective solution available for storage. Consequently, users focused on high performance only should consider using an SSD-only Content Software for File system configuration, while users are seeking to balance performance and cost should consider a tiered data management system, with the assurance that the Content Software for File system features will control the allocation of hot data on SSDs and warm data on object stores, thereby optimizing the overall user experience and budget.

Parent Topic

• Data lifecycle management

In tiered Content Software for File system configurations, there are various locations for data storage as follows:

1. Metadata is stored only on SSDs.
2. Writing of new files, adding data to existing files, or modifying the content of files is always terminated on the SSD, irrespective of whether the file is currently stored on the SSD or tiered to an object-store.
3. When reading the content of a file, data can be accessed from either the SSD (if it is available on the SSD) or rehydrated from the object store (if it is not available on the SSD).

This data management approach to data storage on one of two possible media requires system planning to ensure that most commonly-used data (hot data) resides on the SSD to ensure high performance, while less-used data (warm data) is stored on the object store. In the Content Software for File system, this determination of the data storage media is a completely seamless, automatic, and transparent process, with users and applications unaware of the transfer of data from SSDs to object stores, or from object stores to SSDs. The data is accessible at all times through the same strongly-consistent POSIX filesystem API, irrespective of where it is stored. Only latency, throughput, and IOPS are affected by the actual storage media.The network resources allocated to the object store connections can be controlled. This enables cost control when using cloud-based object storage services since the cost of data stored in the cloud depends on the quantity stored and the number of requests for access made.

Furthermore, the Weka system tiers data in chunks, rather than complete files. This enables the smart tiering of subsets of a file (and not only complete files) between SSDs and object stores.

The network resources allocated to the object store connections can be controlled. This enables cost control when using cloud-based object storage services since the cost of data stored in the cloud depends on the quantity stored and the number of requests for access made.

Parent Topic

• Data lifecycle management

Data management represents the media being used for the storage of data. In tiered Content Software for File system configurations, data can exist in one of three possible states:

1. SSD-only: When data is created, it exists only on the SSDs.
2. SSD-cached: A tiered copy of the data exists on both the SSD and the object store.
3. Object Store only: Data resides only on the object store.

The Data Lifecycle Diagram represents the transitions of data between the above states. #1 represents the Tiering operation, #2 represents the Releasing operation, and #3 represents the Rehydrating operation:

1. Tiering of data from the SSD to create a replicate in the object store. A guideline for the tiering of data is based on a user-defined, time-based policy Tiering Cue.
2. Releasing data from the SSD, leaving only the object store copy (based on the demand for more space for data on the SSD). A guideline for the release of data is based on a user-defined, time-based policy Retention Period.
3. Rehydrating data from the object store to the SSD for the purpose of data access.

In order to read data residing only on an object store, the data must first be rehydrated back to the SSD.

In the Content Software for File system, file modification is never implemented as in-place write, but rather as a write to a new area located on the SSD, and the relevant modification of the meta-data. Consequently, write operations are never associated with object store operations.

Parent Topic

• Data lifecycle management

All writing in the Content Software for File system is performed to SSDs. The data residing on SSDs is hot data, that is data that is currently in use. In tiered Content Software for File configurations, SSDs have three primary roles in accelerating performance: metadata processing, a staging area for writing, and as a cache for read performance.

Parent Topic

• Data lifecycle management

Child Topics

• Metadata processing
• SSD as a staging area
• SSD as a cache

The Content Software for File system includes user-defined policies which serve as guidelines to control the data storage management. They are derived from a number of factors:

1. The rate at which data is written to the system and the quantity of data.
2. The capacity of the SSDs configured to the Content Software for File system.
3. The speed of the network between the Content Software for File system and the object store, and the performance capabilities of the object store itself, for example, how much the object store can actually contain.

Filesystem groups are used to define these policies, while a filesystem is placed in a filesystem group according to the desired policy if the filesystem is tiered.

For tiered filesystems, the following parameters should be defined per filesystem:

1. The size of the filesystem.
2. The amount of filesystem data to be stored on the SSD.

The following parameters should be defined per filesystem group:

1. The Data Retention Period Policy, a time-based policy which is the target time for data to be stored on an SSD after creation, modification or access, and before release from the SSD, even if it is already tiered to the object store, for metadata processing and SSD caching purposes (this is only a target; the actual release schedule depends on the amount of available space).
2. The Tiering Cue Policy, a time-based policy which determines the minimum amount of time that data will remain on an SSD before it is considered for release to the object store. As a rule of thumb, this should be configured to a third of the Retention Period, and in most cases, this will work well. The Tiering Cue is important because it is pointless to tier a file which is about to be modified or deleted from the object store.

For example, when writing log files which are processed every month but retained forever, it is recommended to define a Retention Period of 1 month, a Tiering Cue of 1 day, and ensure that there is sufficient SSD capacity to hold 1 month of log files.

When storing genomic data which is frequently accessed during the first 3 months after creation, requires a scratch space for 6 hours of processing, and requires output to be retained forever: It is recommended to define a Retention Period of 3 months and to allocate an SSD capacity that will be sufficient for 3 months of output data and the scratch space. The Tiering Cue should be defined as 1 day, in order to avoid a situation where the scratch space data is tiered to an object store and released from the SSD immediately afterwards.

Parent Topic

• Data lifecycle management

Child Topics

• Bypassing the time-based policies

The Content Software for File system client is a standard, POSIX-compliant filesystem driver installed on application servers that enable file access to the filesystems. Similar to any other filesystem driver, the system client intercepts and executes all filesystem operations. This enables the Content Software for File system to provide applications with local filesystem semantics and performance (as opposed to NFS mounts) while providing a centrally managed, shareable resilient storage.

The Content Software for File system client is tightly integrated with the Linux operating system page cache, which is a transparent caching mechanism that stores parts of the filesystem content in the client host RAM. The operating system maintains a page cache in unused RAM capacity of the application server, delivering quick access to the contents of the cached pages and overall performance improvements.

The page cache is implemented in the Linux kernel and is fully transparent to applications. All physical memory not directly allocated to applications is used by the operating system for the page cache. Since the memory would otherwise be idle and is easily reclaimed when requested by applications, there is usually no associated performance penalty and the operating system might even report such memory as "free" or "available". For a more detailed description of the page cache, see Page Cache, the Affair Between Memory and Files.

The Content Software for File client can control the information stored in the page cache and also invalidate it, if necessary. Consequently, the system can utilize the page cache for cached high-performance data access while maintaining data consistency across multiple hosts.

Each filesystem can be mounted in one of two modes of operation in relation to the page cache:

• Read cache

  Used where only read operations are using the page cache, file data is coherent across hosts, and resilient to client failures.

• Write cache (default)

  Used where both read and write operations are using the page cache while keeping data coherency across hosts, which provides the highest data performance.

Parent Topic

• Content Software for File client and mount modes

When mounting in this mode, the page cache uses write cache in the write-through mode; so, any write is acknowledged to the customer application only after being safely stored on resilient storage. This applies to both data and metadata writes. Consequently, only read operations are accelerated by the page cache.

In the Content Software for File system, by default, any data read or written by customer applications is stored on a local host read page cache. As a shareable filesystem, the Content Software for File system monitors whether another host tries to read or write the same data and if necessary, invalidates the cache entry. Such invalidation may occur in two cases:

• If a file that is being written by one client host is currently being read or written by another client host.
• If a file that is being read by one host is currently being written from another host.

This mechanism ensures coherence, providing the Content Software for File system with full page cache utilization whenever only a single host or multiple hosts access a file for read-only purposes. If multiple hosts access a file and at least one of them is writing to the file, the page cache is not used and any IO operation is handled by the backends. Conversely, when either a single host or multiple hosts open a file for read-only purposes, the page cache is fully utilized by the Content Software for File client, enabling read operations from memory without accessing the backend hosts.

Parent Topic

• Content Software for File client and mount modes

In this mount mode, the Linux operating system is used as write-back, rather than write-through; specifically, the write operation is acknowledged immediately by the Content Software for File client and is stored in resilient storage as a background operation.

This mode can provide significantly more performance, particularly in relation to write latency, while keeping data coherency; meaning, if a file is accessed through another host, it invalidates the local cache, and syncs the data to get a coherent view of the file.

To sync the filesystem and commit all changes in the write cache, use the following system calls: sync, syncfs, and fsync.

Parent Topic

• Content Software for File client and mount modes

The Content Software for File client supports multiple mount points of the same file system on the same host, even with different mount modes. This can be effective in environments such as containers where different processes in the host need to have different definitions of read/write access or caching schemes.

Parent Topic

• Content Software for File client and mount modes