WEKAPP-190497

Added dark mode to the GUI
Added dark mode to the GUI so the user can switch between the light and dark modes.

WEKAPP-230910

Setting encrypted filesystem only if a KMS is configured
Disable the option to set an encrypted filesystem in the GUI if a KMS is not configured.

WEKAPP-240718

Support installing kernel-ml on RHEL/CentOS
Added support for installing mainline kernel packages (kernel-ml) on RHEL/CentOS.

WEKAPP-244236

Add a backend host failure
When adding a backend host using the weka local setup command, the backend host may fail to join the cluster during a race condition.

WEKAPP-238869

Enabled usage of a user-obtained token during sudo operations.
Weka does not allow an interactive login during mount and the user must log in before the mount command when working with a filesystem with authentication-required.
The weka user login CLI command saves the auth-token to the user�s home directory, whereas sudo mount will look for the auth-token in the root user home directory.
The result will be that the user will perform a login, try to mount, and will be told to log in even though they�re logged in.
Impact: Mount failure

WEKAPP-250579

LDAP/AD usernames with special characters
Setting up an AD/LDAP server for Weka management fails if it includes usernames with special characters.

WEKAPP-242126

Stateless client mount
The client receives a list of IP addresses from the backend and always tries even recent invalid IP addresses, which can lead to long mount times and timeouts.

As part of the fix, two new mount options are introduced to control different timeouts during the mount operation: --connect_timeout_secs and --response_timeout_secs
Impact: Stateless client mount can take a long time if there are incompatible IP addresses in the backend

WEKAPP-220517

umask applied on default-acl
umask is not ignored when applying default ACL, (not adhering to the POSIX standard).

WEKAPP-258165

Setting directory quotas
Directory quota set/unset fails if the directory contains UNIX sockets.

WEKAPP-251283

Error when a read-only user tries to get S3 buckets
When a user with a read-only role tries to get S3 buckets, the system returns an access key error.

WEKAPP-255458

Memory leak
When the transmitting queues become full, the memory leaks under load.

WEKAPP-259350

A process might restart if the network is congested
When the network is congested, a process might restart due to a lack of memory buffers (mbufs).

WEKAPP-244172

Machines Page Synchronization Error message in the GUI
After deactivating and deleting a drive, the user cannot add the drive again through the GUI due to a sync issue. A message �Failed Quering Host�s Hardware� appears.

WEKAPP-265821

GUI crashes on the statistics page
A missing measurement unit in statistics page charts caused the page to crash

WEKAPP-246825

Reduced the default timeout in upgrade
The default timeout during the upgrade process for triggering a rollback is reduced from 5 to 3 minutes.

WEKAPP-63980

Filesystem removal completion event
Added an event to indicate when filesystem removal completes.

WEKAPP-223985

Decimal capacity and bandwidth units
When the cluster�s API emits error messages, capacity and bandwidth units are changed from binary to decimal (for example, GiB to GB). The units appear in the error message.

WEKAPP-244356

Increased number of the local users
The total number of the local users is increased to 1152 (previously, it was 512).

WEKAPP-245464

Added support for LDAP over SSL
Added support for LDAP over SSL (LDAPS).

WEKAPP-249124

Object-store operations list
Enhancement: The command weka fs tier ops runs across all object store buckets when the object store bucket name is not specified.
The command output includes the last HTTP error code an operation encountered.

WEKAPP-254139

weka cluster CLI commands use the singular form
Weka cluster CLI commands use both singular and plural forms:
- Instead of weka cluster nodes, use weka cluster node
- Instead of weka cluster tasks, use weka cluster task
- Instead of weka cluster buckets, use weka cluster bucket
Apart from weka cluster host, the weka cluster hosts CLI command is also supported.

The plural form of these commands still works as an alias for backward compatibility.

WEKAPP-244236

Add a backend host failure
When adding a backend host using the weka local setup command, the backend host may fail to join the cluster during a race condition.

WEKAPP-242300

API Output log rotation.
API output logs (api-stdout.log) were not rotated, leading to logs directory becoming completely full.

WEKAPP-242464

Monitor the capacity of the Weka logs
Added alerts and events indicating when the Weka logs have low disk space.

WEKAPP-238869

Enabled usage of a user-obtained token during sudo operations.
Weka does not allow an interactive login during mount and the user must log in before the mount command when working with a filesystem with authentication-required.
The weka user login CLI command saves the auth-token to the user�s home directory, whereas sudo mount will look for the auth-token in the root user home directory.
The result will be that the user will perform a login, try to mount, and will be told to log in even though they�re logged in.
Impact: Mount failure

WEKAPP-246401

Process restarts on role changes
If the internal Weka process changes its role from Compute to FrontEnd, it can cause other processes to crash when they join the Weka bucket council that the process was once a member of.

WEKAPP-253026

CLI command weka status is unavailable
If the cluster contains unresponsive buckets, the CLI command weka status can be unavailable.

WEKAPP-192827

Alert on low local disk space on the host
Raise an alert and issue an event when hosts reach a low local disk space threshold. These warn what services might be impacted. When enough disk space is freed, the alert is cleared (and a corresponding clear event is issued).

WEKAPP-252758

Periodic restarts when removing an SMB host
Removing a host part of an SMB cluster causes periodic restarts of management processes.

WEKAPP-55041

Manual Prefetch Can Skip Content When Snapshots Are Used
Manual prefetch may skip content due to snapshot and prefetch only some of the content, as some of the read I/Os after the prefetch will still be directed to the object tier.

WEKAPP-251689

Inaccurate capacity indications
The system can indicate inaccurate capacities due to rounding when converting values from bytes to MB/GB/TB etc.

WEKAPP-240948

Quota REST API returns a wrong inode_id
Directory quotas REST API returns the internal inode ID instead of the inode context (user-facing ID).

WEKAPP-252153

Incorrect high drive capacity alert and event
HighDrivesCapacity alert and UsedSSDCapacityCriticalOverflow event can be incorrectly issued temporarily when too much capacity is removed (failed drives/hosts). Replace the capacity to cease oversubscribed errors.

WEKAPP-253057

Hanging IOs after setting nested directory quotas
Setting a nested quota while it is used may cause clients� IOs to hang when the quota becomes active.

WEKAPP-238022

Memory leak on full capacity
Weka process may restart when a directory quota or a filesystem is at full capacity due to a memory leak.

WEKAPP-135311

Prefetching a Large File from OBS
In rare instances, using �weka fs tier fetch� CLI command to prefetch a large file from object store to SSD may result in a node failure event (ASSERT failed: mbuf pool is empty (pool dumped to file).

WEKAPP-217406

Filesystem download from multiple object-store buckets
When downloading a filesystem from an object-store, if the object-store is specified as the additional-obs-bucket, while the snapshot locator is only present in one object-store, the download task may be stuck.

WEKAPP-239897

Failure to get AWS IAM credentials causes a restart of the Weka compute processes
Failure to get AWS IAM credentials, for example, due to a communication timeout, causes a restart of the Weka compute processes.
Configuration: AWS

WEKAPP-241668

Snapshot creation timestamp
A snapshot creation timestamp could change to another snapshot�s timestamp after a specific sequence of snapshot actions.
This issue doesn�t affect the content of the snapshots, only the reported timestamp in the snapshot list.

WEKAPP-217486

NFSv3 statistics
NFSv3 statistics are not available for NFS-W.

WEKAPP-245771

Added NFSv3 statistics for NFS-W
Added NFSv3 statistics for NFS-W to the Weka GUI and Weka CLI (weka statscommands).

WEKAPP-245740

Webhook does not send audit events
Webhook does not send audit events due to a connection failure, and the system does not retry the connection.

WEKAPP-255742

Host removal from an S3 cluster
A host cannot be removed from the S3 cluster if it leaves it with fewer active hosts than the minimum required to function correctly.

WEKAPP-238375

New QoS statistics
Added new statistics to allow a better understanding if a delay is a system side or due to QoS client configuration.
WRITE_QOS_DELAY, READ_QOS_DELAY to show the QoS delay and WRITE_LATENCY_NO_QOS, READ_LATEMCY_NO_QOS to show delays without QoS.

WEKAPP-257326

Low efficiency reading
A client may reach a state that causes some reads to be performed inefficiently.

WEKAPP-242597

ClusterAdmin user cannot add an S3 user
A user with a ClusterAdmin role cannot add an S3 user through the Weka GUI.

WEKAPP-244172

Machines Page Synchronization Error message in the GUI
After deactivating and deleting a drive, the user cannot add the drive again through the GUI due to a sync issue. A message �Failed Quering Host�s Hardware� appears.

WEKAPP-245797

Use of weka fs tier [fetch | release | location] in parallel might fail
Runningweka fs tier fetch, weka fs tier release, or weka fs tier location with more than 16 parallel invocations migth cause internal restarts of Weka management processes. This will hang and then fail both the invocations and other CLI commands invoked at the time.

WEKAPP-246209

CLI login failure exit code
A CLI login failure returns 0 (success exit code) instead of 5 (failure exit code).

WEKAPP-250411

weka cluster host -v CLI command exits with an error
The weka cluster host -v command exits with an error when the cluster includes clients in the DOWN state.
Impact: weka cluster host -v with a DOWN client is broken

WEKAPP-250489

The login banner message is not displayed
The login-banner show CLI command does not display the login banner message.

WEKAPP-246233

Upgrade failure
An upgrade might fail if an internal configuration change and the cache becomes full.

WEKAPP-246825

Reduced the default timeout in upgrade
The default timeout during the upgrade process for triggering a rollback is reduced from 5 to 3 minutes.

WEKAPP-249669

Upgrade failure
The upgrade rarely fails when the Weka agent fails to parse state files. The failure initiates a rollback to the source version.

WEKAPP-254470

Explicit client upgrade failure
Running the weka local upgrade command on a client might fail while there are IOs running on the host during the upgrade.

🔴 WEKAPP-36427

Setuid/Setgid Bits Not Cleared After Write to a File
When writing to a file that has setuid/setgid bit sets, the bits are not cleared.
Workaround: Workaround is to manually remove setuid/setgid bits if files are to be written to.

🔴 WEKAPP-37692

inotify API Not Supported
The inotify interface is not supported across the cluster and returns only local changes.
Workaround: Workaround is to not rely on the inotify API and scan dates manually.

🔴 WEKAPP-76966

Directory Link Count Is Always 1
Creating/deleting sub-directories does not increase/decrease the parent directory link-count. Calling stat on a directory will always return its links count as 1.

🔴 WEKAPP-78789

EBADHANDLE Failure in NFS when Accessing an Overridden Snapshot Directory
In NFS, accessing a snapshot directory that was overridden by another snapshot may result in failure on EBADHANDLE.
Workaround: Workaround is to wait a minute for the dentry cache to expire.

🔴 WEKAPP-89110

Overwrites with Multiple Front-Ends
Overwrites may occur if multiple front-ends write to a file that is opened with the O_APPEND flag.
Workaround: Workaround is to not append to the same file from multiple hosts simultaneously.

🔴 WEKAPP-80978

Cluster Unavailability while Creating Very Large File System
Creation of a file system with between 18,446,744,073,709,547,520 and 18,446,744,073,709,551,616 bytes may cause cluster unavailability.
Workaround: Workaround is to create file system with capacity that does not fall between 18,446,744,073,709,547,520 bytes and 18,446,744,073,709,551,616 bytes.

🔴 WEKAPP-198777

Enforcing Quotas and writecache mount mode
Working with enforcing quotas along with a writecache mount-mode, similarly to other POSIX solutions, getting above the quota might not sync all the cache writes to the backend servers.
When using the default wekafs writecache mount-mode or SMB writecacheshare, close() might also not return an ENOSPC, which is according to the standard POSIX behavior, as detailed in the close() man page.
Workaround: To ensure data has been written to disk use the sync(), fsync(), syncfs() calls, which would return ENOSPC, as stated in the close() man page. 
Note that the cached data might not be written out after getting an ENOSPC even if the quota limit is increased.

🔴 WEKAPP-258165

Setting directory quotas
Directory quota set/unset fails if the directory contains UNIX sockets.

🔴 WEKAPP-76981

Cluster �Choke� State from One Slow, Overloaded Host
A problematic host with slower or faulty hardware or software may become a cluster bottleneck. In such a situation, the entire cluster may be put into a �choking� state which degrades performance.
Workaround: If a host is determined to be a bottleneck because of a particular problem, replace or remove it.

🔴 WEKAPP-135254

Detach of Object Store from File System May Appear to Be Stuck
Progress of detaching object store may appear to be stuck and then will finish without showing progress.

🔴 WEKAPP-154176

Update S3 Bucket After Attached Filesystem Uploaded Data
Updating a S3 bucket after an attached filesystem already uploaded data to it may result in data inaccessibility.

🔴 WEKAPP-227556

High prefetch configuration for object-store buckets
Configuring a large prefetch size (using the weka fs tier s3 add/update --prefetch-mib CLI command) may cause a high concurrent readahead load on the object store, which may slow down downloads and other background operations.
Workaround: Reduce the configured prefetch size for the object-store bucket using the weka fs tier s3 update --prefetch-mib CLI command.

🔴 WEKAPP-134569

Deactivation of drives is allowed even when it may cause unavailability
A cluster�s running compute resources require a minimal number of drives and minimal capacity in order to operate.
If the cluster is left with too few drives available due to failures and/or deactivations, it can cause compute resources to become unavailable, leading to general unavailability.
Workaround: Do not deactivate too many drives, replace failed drives.

🔴 WEKAPP-87146

Slow SSD Phase Out
If a SSD becomes too slow, it is automatically phased out. In rare instances, it may take a long time to phase out the slow SSD.
Workaround: Workaround is to physically disconnect the slow SSD, if there are not other failures in the cluster.

🔴 WEKAPP-143548

Resize While Downloading Filesystem
While downloading a filesystem, it is possible to resize it, to the point where not enough capacity is available for the download.

🔴 WEKAPP-28226

Incorrect Behavior for O_ACCMODE
Opening a file with O_ACCMODE permissions, which is an invalid permission, may return a file descriptor that is valid for read and write, even though it should disallow any data-related operation.
Workaround: Workaround is to avoid opening a file with O_ACCMODE permissions.

🔴 WEKAPP-58651

DENTRY Cache of Open Directories Not Updated on Rename Across Hosts
The aging parameters for directories does not apply to open directories.

🔴 WEKAPP-60921

Driver May Block System Calls on Client OS Reboot
The reboot sequence may deadlock when the WekaIO driver mounts block all system calls.
Workaround: Workaround is to use reboot nf flags, or unmount the WekaIO filesystem before reboot.

🔴 WEKAPP-71988

Mixing AIO and Direct Mode on the Same File May Cause Incoherent Writes
When a filesystem is mounted in the write cache mode, and a file is opened on the same client, once with O_DIRECT and accessed through AIO (io_submit functions), the write commands of the two processes are not coherent.

🔴 WEKAPP-72203

Possible Upgrade Failure When Writing to Memory-Mapped File
An upgrade may fail if the user has a memory-mapped file and continues to write to it during the upgrade.
Workaround: Workaround is to ensure that no writes are executed to memory-mapped files during upgrade.

🔴 WEKAPP-85802

Atomic Open Not Supported for WekaIO Client with More than One Core
Atomic Open is not supported if the WekaIO client is configured with more than one core.
Workaround: Workaround is to define only a single core for the WekaIO client.

🔴 WEKAPP-118249

Parallel Write Jobs Throughput
Front ends may become throughput performance bottleneck parallel write jobs.
Workaround: Workaround is to use O_DIRECT (or �readcache� mount) for jobs where throughput of big block size writes is important.

🔴 WEKAPP-150731

NFS IP Failover
Floating IP may not fail over on down link if the host interface group�s port is not the same as the host config net device.

🔴 WEKAPP-260212

NFSv4 IP might take time to failover
In case of a failure on the network or a host serving NFSv4, it might take some time for the IP to failover to another host serving NFS. This identification can take too much time to become disruptive to the NFS clients and in some circumstances may fail to identify an IP failover should occur.

🔴 WEKAPP-31218

UID/GID in NFS Limited to 65536
When using NFS, UID and GIS cannot exceed 65536, even though the standard allows for 32 bits.
Workaround: Workaround is to use the WekaIO driver interface.

🔴 WEKAPP-35257

ENFILE Returned from NFS
If both the driver and NFS interfaces are used on the same host and there are 131,072 unique files open via the driver, any NFS operation may result in the return of an �ENFILE� error, which is not a valid NFS error.
Workaround: Workaround is to ignore the error code and refrain from using both the driver and NFS interfaces on the same host.

🔴 WEKAPP-35951

EROFS Returned Instead of EEXISTS
When trying to create a file which already exists in read-only NFS, EROFS is returned instead of EEXISTS.

🔴 WEKAPP-75079

NFS Locks Not Supported
NFS locking is not supported.
Workaround: Workaround is to use POSIX driver locks.

🔴 WEKAPP-157454

Deleting filesystem/snapshot during NFS client access
Deleting the filesystem or snapshot including a file that is open through an NFS client, might lead to a Weka process restart. Once the process is up again, everything should get back to normal.

🔴 WEKAPP-94171

SMB Floating IP Addresses Not Released
Upon deletion or removal of a cluster, the floating IP addresses are not released and will remain on the interface.
Workaround: Delete the IPs manually with ip address delete.

🔴 WEKAPP-87306

HA and Dual Interconnect Configurations
HA and dual interconnect (Ethernet or InfiniBand) setups need a manual configuration to steer the NFS traffic to the correct interface. Only one of the interfaces on the host can be used. HA support for NFS is limited to move the IP address from that interface when the link of that interface goes down.
Workaround: For each host to be used as a NFS server, a rule needs to employed to set the NFS IP addresses to use the designated interface.

Examples:

ip rule add from 10.222.96.250/4 lookup weka1

or:

ip rule add from 10.222.96.250 lookup weka1
ip rule add from 10.222.96.251 lookup weka1

🔴 WEKAPP-124688

Intermittent Node Failure from Inter-Switch Link MTU Misconfiguration
In an HA configuration, if the links between the switches have misconfigured MTU, there may be intermittent node failures. This issue will be addressed in a future release.
Workaround: Workaround requires WekaIO support team to work customer to fix the misconfigured MTU on customer�s inter-switch link.

🔴 WEKAPP-85745

Error Message When Removing Client in GUI
When removing a physically unavailable host from the UI console, the operation will end with a timeout failure dialog and the host will be removed from the configuration.
Workaround: Workaround is to remove the host from the configuration before physically detaching it from the cluster.

🔴 WEKAPP-128152

Host Configuration Error Messages in GUI
Upon a host configuration failure, the reason for the failure is not part of the error message in the GUI. This issue affects:
- Add/Remove Net Device
- Configure Host
Workaround: Workaround is connect to the host and use the �weka local status� CLI command to see the detailed error message.

🔴 WEKAPP-134013

Empty UI Events Table
In rare instances, the UI events table is empty.
Workaround: Refresh the page or go to another view & return to events view