Item

Description

WEKAPP-224233

Resources restore CLI command

The weka local resources restore command does not correctly restore the resources.

WEKAPP-225760

Adjusted mount behavior for non-supported mount option by networking device

When mounting, prevent and block setting dedicated_mode=NONE when RX interrupt is not supported by the network device Device with DPDK networking.

WEKAPP-227771

Un-mounting during upgrade can cause the upgrade to fail

Un-mounting a mount-point from a host that is part of the cluster during an upgrade can prevent the host from joining back to the cluster and might cause the upgrade to fail.

Impact: Upgrade failure

Configuration: Converged environments; Legacy clients

WEKAPP-228463

Cannot start-io after changing IPs on all hosts

The system might fail to start-io after changing IPs for all of the hosts.

WEKAPP-229859

Incorrectly adding new network devices in cpu-optimized mode (UDP)

When manually adding network devices after the cluster is up and running, they were added in UDP mode instead of DPDK.

Impact: Reduced performance

WEKAPP-230789

Thin-provisioned filesystems in multiple organizations

Miscalculation of free space in non-root organizations without an organization quota and thin-provisioned filesystems might lead to failures in creating new (thick provisioned) filesystems in that organization.

Configuration: Multiple organizations and thin-provisioned filesystems

WEKAPP-234726

Clients connectivity not recovering after heavy load

In some circumstances, under heavy load, some clients might experience permanent degraded connectivity instead of a transient degradation.

WEKAPP-244236

Add a backend host failure

When adding a backend host using the weka local setup command, the backend host may fail to join the cluster during a race condition.

Item

Description

WEKAPP-205068

Reduced excessive logging to Syslog

Reduced the amount of log messages that log the output of the NFS server get_command_output command to Syslog.

Configuration: Hosts configured with the new NFS stack

WEKAPP-215939

Reduce messages generated to Syslog

Removed redundant DEBUG: Answering if container samba is running messages to Syslog.

WEKAPP-215956

Mount failure due to agent failure during startup

The agent process might crash on startup in case of an ungraceful shutdown.

Impact: Mount failure

WEKAPP-217244

Mount option dedicated_mode using lowercase value resulting in failure

Mount option value for dedicated_mode only accepts uppercase - either "NONE" or "FULL" instead of other case insensitive variations, as it should.

Impact: Mount failure

WEKAPP-235449

Updated Syslog level for mount and umount operations

The Syslog log level of the mount and umount commands is now DEBUG, improving diagnostic capabilities.

WEKAPP-237963

Resolved issue where system agent may restart

System agent might restart under concurrent API calls load.

WEKAPP-242300

API Output log rotation

API output logs (api-stdout.log) were not rotated, leading to logs directory becoming completely full.

WEKAPP-242464

Monitor the capacity of the logs

Added alerts and events indicating when the logs have low disk space.

Item

Description

WEKAPP-261888

Cannot add a drive to a host that's on a different failure domain

The system does not prompt a message and does not prevent adding a drive to a host if the drive was previously configured with a different failure domain.

The solution includes adding a relevant message and preventing this action.

Item

Description

WEKAPP-205945

REST API /security/tokensExpiry route rename

Change route name of /security/tokensExpiry (deprecated) to /security/defaultTokensExpiry, to avoid confusion.

Item

Description

WEKAPP-216114

Organization creation password complexity enforcement

When creating a new organization, along with its org-admin, the system's password complexity policy is not enforced.

WEKAPP-238869

Enabled usage of a user-obtained token during sudo operations.

HCSF does not allow an interactive login during mount, and the user must log in before the mount command when working with a filesystem with authentication-required.

The weka user login CLI command saves the auth-token to the user's home directory, whereas sudo mount will look for the auth- token in the root user home directory.

The result will be that users will perform a login, try to mount, and will be told to log in even though they're logged in.

Impact: Mount failure

WEKAPP-250579

LDAP/AD usernames with special characters

Setting up an AD/LDAP server for system management fails if it includes usernames with special characters.

Item

Description

Failures in the cluster leader when in the stopped IO state (as during an

upgrade) might result in drives not identified for phasing out, IO not being able to start and causes the upgrade to fail.

WEKAPP-224981

Unavailability issue due to large config change sets

Under a rare statistical race condition in the system management process, due to a very large configuration delta being created, exceeding an internal buffer that was incorrectly defined being too small.

This causes processes to continuously crash and for system unavailability.

Impact: API and Data unavailability

Configuration: More than 800 hosts (clients and backends)

WEKAPP-225593

Cluster initialization failure

During cluster initialization, if a host that has drives fails to apply its resources correctly and a consecutive initial start-io is invoked, the cluster will never be able to start serving IOs.

WEKAPP-236040

Stateless client deactivation

When the weka cluster host deactivate CLI command is invoked on a stateless client, the client rejoins the cluster after a few seconds as active.

WEKAPP-246401

Process restarts on role changes

If the internal system process changes its role from Compute to FrontEnd, it can cause other processes to crash when they join the system bucket council that the process was once a member of.

WEKAPP-253026

CLI command weka status is unavailable

If the cluster contains unresponsive buckets, the CLI command weka status can be unavailable.

Item

Description

WEKAPP-215875

Diagnostic uploading failure

Diagnostic uploading fails, if the current host was not chosen to collect diagnostic information from.

When running the command 'weka diags upload --clients' without

specifying the host running the upload using the '–host-id' option, the upload will fail on an assertion error

Item

Description

WEKAPP-212031

"ObjectStoreStatusChanged" event was split into "ObjectStoreStatusUp" and "ObjectStoreStatusDown" events

"ObjectStoreStatusChanged" of severity INFO used to report both UP and DOWN status changes.

Instead, the new event "ObjectStoreStatusUp" with severity INFO and the new event "ObjectStoreStatusDown" with severity MAJOR will be reported.

Item

Description

WEKAPP-221941

License display error

The weka cluster license CLI command sometimes shows parsing errors instead of displaying a properly formatted error

Item

Description

WEKAPP-228495

Authenticated Mount

When using a filesystem with auth-required (this is required when using multiple organizations for the non-root organization), if the mount command is run by the user without supplying the authentication token, the interactive mount command asking for user/password will create an authentication token and store that token in the root user directory without removing that token later.

We changed the behavior to not allow interactive login via the mount command (which is also problematic when mounting via

fstab/systemd). Users should obtain an authentication token beforehand (via weka user login) and provide it to the mount command (with the auth_token_path option).

Item

Description

WEKAPP-168666

Statistics output sorting

The weka stats … --sort=stat,value CLI command sorts the output lexicographically instead of numerically.

WEKAPP-192827

Alert on low local disk space on the host

Raise an alert and issue an event when hosts reach a low local disk space threshold. These warn what services might be impacted. When enough disk space is freed, the alert is cleared (and a corresponding clear event is issued).

WEKAPP-230113

Incorrect units in realtime CLI stats

The weka stats realtime CLI command incorrectly displays MS (milliseconds) instead of "μS" (microseconds).

Item

Description

WEKAPP-217203

Upgrade failure due to local disk out-of-space

Existing core files are not rotated correctly and might cause upgrade issues if exist and those files consume too much space.

Item

Description

WEKAPP-252758

Periodic restarts when removing an SMB host

Removing a host part of an SMB cluster causes periodic restarts of management processes.

Item

Description

WEKAPP-217545

Misleading error message when not specifying the network interfaces when mounting on a physical server

Mounting on a physical server without the net option should not be possible and should yield an appropriate error message. Instead, the command issues a wrong error message.

WEKAPP-242126

Stateless client mount

The client receives a list of IP addresses from the backend and always tries even recent invalid IP addresses, which can lead to long mount times and timeouts.

As part of the fix, two new mount options are introduced to control different timeouts during the mount operation: -- connect_timeout_secs and --response_timeout_secs

Impact: Stateless client mount can take a long time if there are incompatible IP addresses in the backend

Item

Description

WEKAPP-55041

Manual Prefetch Can Skip Content When Snapshots Are Used Manual prefetch may skip content due to snapshot and prefetch only

some of the content, as some of the read I/Os after the prefetch will still be directed to the object tier.

WEKAPP-220517

umask applied on default-acl

umask is not ignored when applying default ACL (not adhering to the POSIX standard).

WEKAPP-222922

Snapshot upload during object-store detach from filesystem might lead to cluster unavailability

Uploading a snapshot to a remote object-store from a downloaded filesystem that is undergoing detach from object-store might lead to system processes restarts and cluster unavailability.

WEKAPP-224783

Unavailability after snapshot deletion

A race condition between RAID shrinkage and background snapshot deletion processes can cause an internal inconsistency that leads to system compute-processes restarting. This can lead to full cluster unavailability.

Impact: Unavailability

WEKAPP-228311

Remote snapshot upload during migration

Remote snapshot upload during migration of a downloaded filesystem can fail to upload file content from the downloaded filesystem. This may cause a loss of data in the remote copy, which means you cannot recover from the remote backups.

The detach operation should be completed before uploading snapshots to the remote object store.

WEKAPP-251689

Inaccurate capacity indications

The system can indicate inaccurate capacities due to rounding when converting values from bytes to MB, GB, TB, and so on.

Item

Description

WEKAPP-214710

Quota Accounting

A rare race condition might lead to users being able to consume more space than the quota allocation allows by conducting a false subtraction.

WEKAPP-216771

Exceeding directory quota limit

Multiple writers to the same directory quota can sometimes fill the quota beyond the hard limit.

WEKAPP-219781

Directory quota set/unset failure

Setting or unsetting a directory quota fails if the directory contains symbolic links.

WEKAPP-220765

Writing above the filesystem SSD limit

Clients might be able to write some data above the filesystem SSD capacity limit. IF the entire SSD is occupied, this might come at the cost of the hot-spare capacity, reducing the ability to complete a rebuild and restore resiliency after a failure.

WEKAPP-225118

Filesystem total capacity corrected reporting

When SSD capacity is full, the system fails to change the filesystem's total capacity.

WEKAPP-231509

Removing a directory quota

Failure to remove a nested directory quota after it has canceled/failed during the setting process

WEKAPP-240948

Quota REST API returns a wrong inode_id

Directory quotas REST API returns the internal inode ID instead of the

inode context (user-facing ID).

WEKAPP-252153

Incorrect high drive capacity alert and event

HighDrivesCapacity alert and UsedSSDCapacityCriticalOverflow event can be incorrectly issued temporarily when too much capacity is removed (failed drives/hosts). Replace the capacity to cease oversubscribed errors.

WEKAPP-253057

Hanging IOs after setting nested directory quotas

Setting a nested quota while it is used may cause clients' IOs to hang when the quota becomes active.

WEKAPP-258165

Setting directory quotas

Directory quota set/unset fails if the directory contains UNIX sockets.

WEKAPP-268022

IO operations hang when a directory quota reaches its exact value

When the directory quota is filled to its exact value, the QuotaHardLimitReached event is not issued, and IO operations to this directory may hang.

WEKAPP-269936

Default quota REST API returns a wrong inode_id

Default directory quota listing REST API returns the internal inode ID instead of the external one.

Item

Description

WEKAPP-232512

Resuming an object-store detach task while it is still pausing

Resuming a cluster task (object-store detach) while it is still in PAUSING state (visible in weka cluster tasks CLI) might cause the task to resume in an incorrect place. This might lead part of the detach task to be skipped, leading to data unavailability.

Item

Description

WEKAPP-238775

IOs might hang longer than required

In some cases, locks recovery might wait for longer than the locks grace time, which is noticeable by the clients' applications.

Item

Description

WEKAPP-214538

False drive failure events

A single IO core serving both COMPUTE and Front End (FE) processes might cause false identification of drive failures, which can lead to false events about drive failures or TooManyFailures.

WEKAPP-221291

Stale file attributes cache

In rare circumstances, stale file attributes might be returned from cache.

Item

Description

WEKAPP-238022

Memory leak on full capacity

System process may restart when a directory quota or a filesystem is at full capacity due to a memory leak.

Item

Description

WEKAPP-60978

OBS Prefetch fetches files more accurately

When configuring prefetch-mib in weka fs tier s3 add, it now prefetches files better and increases the performance of files residing in the object- store.

WEKAPP-135311

Prefetching a Large File from OBS

In rare instances, using weka fs tier fetch CLI command to prefetch a large file from object store to SSD may result in a node failure event (ASSERT failed: mbuf pool is empty (pool dumped to file).

WEKAPP-216932

Updating object-store bucket site parameter

Remove support for directly updating the object-store bucket site, as it should always be the same as the object-store site it's attached to. To change the site remove and add a new object-store bucket instead, with the desired object-store.

WEKAPP-217406

Filesystem download from multiple object-store buckets

When downloading a filesystem from an object-store, if the object-store is specified as the additional-obs-bucket, while the snapshot locator is only present in one object-store, the download task may be stuck.

WEKAPP-221605

Changes in local IP binding behavior for OBS connections

For REMOTE OBS buckets, IP binding will no longer be used for connections.

This is important for connections to REMOTE object stores that reside in an external network that is not accessible by connections bound to a local NIC's IP address and/or subnet.

WEKAPP-222344

Remote snapshot upload from a downloaded filesystem

If a downloaded filesystem is detached from the local object-store (containing the snapshot) and attached to a remote object-store instead, uploading snapshots to the remote object-store might not contain all the snapshot data.

WEKAPP-222658

Changed the default object-stores names

The default object-stores names have been renamed from "default" & "remote_default" to "default-local" & "default-remote" respectively.

WEKAPP-228356

Uploading small files to object-storage might under-utilize available bandwidth

Uploading small files to object-storage might not utilize the entire available bandwidth to the object-store, due to overestimation of the RAM required for small file uploads.

Item

Description

WEKAPP-209705

False Hot Tier Near Capacity alerts

Hot Tier Near Capacity alert might be issued incorrectly when the SSD capacity is not full (as can be seen in the weka fs CLI command output), and contain an incorrect percentage value for the drive fill leve

WEKAPP-217232

Cluster unavailability after upgrade

When upgrading to 3.9.x, 3.10.x, or 3.11.x versions that are not including this fix, under very rare circumstances, the cluster might reach an internally inconsistent state resulting in cluster unavailability.

Impact: Data Unavailability

WEKAPP-220085

Cluster unavailability

Under very rare circumstances involving both drive and compute processes failures, the cluster might become unavailable.

WEKAPP-220807

Under rare circumstances, overwrites might get corrupted

Under rare race conditions when overwriting a file during space reclamation or releasing it to the object-store, the overwrite might get corrupted.

Configuration: Tiered filesystems

Item

Description

WEKAPP-229880

Snapshot upload from downloaded filesystem to remote object-store might hang

Uploading a snapshot to a remote object-store bucket from a downloaded filesystem might hang if the filesystem has not begun detaching from the local object-store bucket.

Item

Description

WEKAPP-88400

Snapshot rename operation is not idempotent

Rename a snapshot with existing name is not possible.

WEKAPP-228616

Uploading a snapshot to a remote object-store during an object-store detach operation

Uploading a snapshot to a remote object-store from a downloaded filesystem while it is undergoing an object-store detach operation can cause cluster unavailability.

WEKAPP-241668

Snapshot creation timestamp

A snapshot creation timestamp could change to another snapshot's timestamp after a specific sequence of snapshot actions.

This issue doesn't affect the content of the snapshots, only the reported timestamp in the snapshot list.

Item

Description

WEKAPP-213016

Rare race condition for metadata update during upgrade may lead to data unavailability

HCSF has special internal blocks in which it stores a lot of small differences regarding various metadata blocks, in order to reduce both metadata updates and write amplification. These diff-blocks are associated with the addresses of the metadata block database we call a "registry". Portions of the registry, while rare, may split to handle growth of the number of metadata blocks.

HCSF version v3.10 introduced new features that required to upgrade these diff blocks. Due to the introduction of this new feature a race condition may occur during the upgrade process of these diff-blocks and during registry splitting due to growth. When this occurs, it can cause

diff-blocks to be associated with the incorrect addresses (addresses they had prior to the split). When the diff-blocks are later read (for example, at start-io that happened after the rollback), their addresses undergo a validation process. Due to the race condition, the validation process fails preventing access to the data portion of the blocks.

Impact: Data Unavailability

WEKAPP-227934

Unavailability after a multi-hop upgrade

The system might become unavailable after the upgrade, if there were no modifying IO operations (for example, writes, file creations) on it since the last upgrade.

Configuration: Multi-hop upgrade

Item

Description

WEKAPP-227768

Unable to restart a front-end process

If a front-end process is terminated while performing a GPUDirect GDS operation, it cannot be restarted.

WEKAPP-230629

Client gets hung on remount

When incorrect options are provided to the mount -o remount command during the remount of a filesystem, the system client might get a kernel panic instead of returning a graceful error.

WEKAPP-232988

Calling stat immediately after close may report wrong size

After a file is written and closed, there is a short window that calling

stat() on the file may return the wrong size.

WEKAPP-234255

SELinux support with kernels later than 4.5

On kernels later than v4.5, XATTR-NOT-SUPPORTED is wrongly reported as unsupported. Note, to set SELinux, you must mount with -o acl option.

WEKAPP-234257

Negative directory entry cache is not invalidated on hardlinks

When using a non-zero dentry_max_age_negative mount option, directory entries that become positive due to an hardlink operation do not invalidate negative entries.

WEKAPP-238861

Resolved issue with non-4k aligned AIO.

Non-4k aligned AIO cause a weka client crash and host reboot.

WEKAPP-239429

Resolved issue when using special files on client.

The system client generates a Kernel panic under some conditions when using special files (named pipes, block devices) on the system mount point.

Item

Description

WEKAPP-209853

Unable to mount NFS exports

When an NFS export has been created to a non-existing path, once that path is added, users might fail to mount it. The export should be added to an existing path within the filesystem.

WEKAPP-210775

NFS metadata performance improvement

Significantly improved performance of NFS metadata operations when iterating large directory trees

WEKAPP-217486

NFSv3 statistics

NFSv3 statistics are not available for NFS-W.

WEKAPP-218973

sssd integration with NFS manage-gids option

NFS export defined with manage-gids does not retrieve the user's groups information from the configured service in sssd.

WEKAPP-245771

Added NFSv3 statistics for NFS-W

Added NFSv3 statistics for NFS-W to the GUI and CLI (weka stats

commands).

Item

Description

WEKAPP-218971

Excessive logging to Syslog

The SMB and NFS services print excessive logs to Syslog.

Item

Description

WEKAPP-183914

Potential HA performance impact

HA performance might be impacted due to an unbalanced use of both networks

WEKAPP-206378

Auto increase MTU

Improve MTU increase detection to automatically return to high MTU.

WEKAPP-217449

Potential data corruption on RDMA read

Under some rare race conditions, when there is a packet loss in the network when working with RDMA, data might get corrupted.

Impact: Data corruption upon RDMA read

Configuration: RDMA enabled environments

WEKAPP-220842

NodeRDMANotActive alert for IO processes that are down

NodeRDMANotActive will not be triggered for IO processes that are down and have other indicators.

WEKAPP-223062

Clients mount failure

The incoming traffic on specified MAC may not route the returned packets correctly. To address this and handle the router HA config properly the configured GW is used instead.

WEKAPP-225056

Backend hosts restarts when removing client mounts

When the IB network is under congestion and a client is removed there can be a rare race condition that causes some backends to fail with a NetworkPortDead event.

Configuration: Infiniband

WEKAPP-231080

Unable to remove mixed Infiband and Ethernet networking configuration

Unable to remove a mixed Infiband and Ethernet networking on the server.

WEKAPP-238375

New QoS statistics

Added new statistics to allow a better understanding if a delay is a system side or due to QoS client configuration.

WRITE_QOS_DELAY, READ_QOS_DELAY to show the QoS delay and WRITE_LATENCY_NO_QOS, READ_LATEMCY_NO_QOS to show

delays without QoS.

WEKAPP-238399

Performance hit due to many client restarts

Client log files filling the root directory create multiple client process restarts that lead to a performance hit on the cluster (due to too many configuration changes sent by the servers).

WEKAPP-255458

Memory leak

When the transmitting queues become full, the memory leaks under load.

WEKAPP-259350

A process might restart if the network is congested

When the network is congested, a process might restart due to a lack of memory buffers (mbufs).

Item

Description

WEKAPP-257326

Low efficiency reading

A client may reach a state that causes some reads to be performed inefficiently.

Item

Description

WEKAPP-216923

Authentication preservation during editing of an object-store bucket via the GUI

When clicking on the object-store bucket for editing, the authentication method changes to none, instead of preserving the original authentication method.

WEKAPP-225622

GUI installation stripe size

The stripe size via GUI installation isn't updated to the new limits.

For example, on an 8-host cluster, you cannot create a 6+2 (only 4+2) striping via the GUI.

The CLI works as expected (for example, allows 6+2 striping with 8- hosts cluster)

WEKAPP-229456

Incorrect GUI validation for filesystem minimum capacity

The minimum capacity for creating and updating filesystems via the GUI is 10MB while the actual system enforced minimum is 1GB.

WEKAPP-244172

Machines Page Synchronization Error message in the GUI

After deactivating and deleting a drive, the user cannot add the drive again through the GUI due to a sync issue. A message "Failed Quering Host's Hardware" appears.

Item

Description

WEKAPP-145253

Resolved Authentication Error Handling Issue

Failure to login (weka user login) due to inability to save the auth_token.json file returns success instead of returning an informative error.

WEKAPP-221844

CLI output text for REDISTRIBUTING status

Changed the text that appears in weka status when the system is redistributing data when it is fully protected (system status is REDISTRIBUTING).

WEKAPP-232196

Badly formatted output from the weka alerts command

Some alerts were output with bad formatting making the output unreadable. They are now correctly formatted.

WEKAPP-245797

Use of weka fs tier [fetch | release | location] in parallel might fail

Running weka fs tier fetch, weka fs tier release, or weka fs tier location with more than 16 parallel invocations migth cause internal restarts of system management processes. This will hang and then fail both the invocations and other CLI commands invoked at the time.

WEKAPP-246209

CLI login failure exit code

A CLI login failure returns 0 (success exit code) instead of 5 (failure exit code).

WEKAPP-250411

weka cluster host -v CLI command exits with an error

The weka cluster host -v command exits with an error when the cluster includes clients in the DOWN state.

Impact: weka cluster host -v with a DOWN client is broken

WEKAPP-250489

The login banner message is not displayed

The login-banner show CLI command does not display the login banner message.

Item

Description

WEKAPP-265821

GUI crashes on the statistics page

A missing measurement unit in statistics page charts caused the page to crash

Item

Description

WEKAPP-220438

Filesystem group REST API

Added filesystem groups REST API routes.

Item

Description

WEKAPP-209545

Prepare clients for upgrade fails after timeout

If the connection has timed out while preparing the clients for the upgrade, the preparation of the clients will fail on rare occasions as the cluster does not keep those connections alive.

WEKAPP-220843

Upgrade failure due to incorrect free space validation

The free space validation for the upgrade is not accurate and this will result in the upgrade will start even if one of the upgrading hosts doesn't have enough free space, which can result in an upgrade failure.

WEKAPP-225672

Upgrade is blocked for a period of time after a cluster leadership restarts

Upon a weka management process restart, it appears as if the system comes up from an upgrade. The upgrade task seems to restart its progress from 0% to 100%.

During this time, upgrades are blocked.

Impact: Cannot upgrade during this time

WEKAPP-246233

Upgrade failure

An upgrade might fail if an internal configuration change and the cache becomes full.

WEKAPP-246825

Reduced the default timeout in upgrade

The default timeout during the upgrade process for triggering a rollback is reduced from 5 to 3 minutes.

WEKAPP-249669

Upgrade failure

The upgrade rarely fails when the system agent fails to parse state files. The failure initiates a rollback to the source version.

WEKAPP-254470

Explicit client upgrade failure

Running the weka local upgrade command on a client might fail while there are IOs running on the host during the upgrade.

Item

Description

WEKAPP-269693

Filesystem mount hangs after upgrade

A filesystem mount performed right after a weka local upgrade might hang on the client machine due to the system agent failing to wait for front-end processes to finish starting up.