CData Virtuality 25.4: FIPS support for SSL connections, Git Integration improvements, and more

In this release, we’ve implemented Federal Information Processing Standards (FIPS) support for SSL connections, ensuring that all cryptographic operations used by the CData Virtuality Server and JDBC driver can be performed using a FIPS-approved provider and FIPS-compatible keystores (you’ll need to enable it as described in our documentation).

On the Server side, we’ve upgraded the embedded PostgreSQL used for the configuration database to v17 and significantly improved the structure of the Git repository (which means that it’s incompatible with the old version). Here’s what changed: we’ve improved the storage of recommended indexes, schedules and jobs, foreign functions, users and their roles and changed some other storage locations. Job owner, executor, and "enabled" flag are now saved in the same file as the job itself, materialized tables are stored together with their recommended optimizations (which are now stored in separate files), materialization jobs are located in the "jobs" folder, and last but not least, we’ve added storage of permissions for LDAP roles and improved the related Git folder structure.

We’ve also fixed two bugs affecting our Git Integration implementation: one where strings were not properly escaped on storing to the repository, and another where the "SYSADMIN.purgeObjects" procedure threw an error with 'jobs' item listed multiple times if an incorrect filter value is passed.

Our LDAP Authentication implementation now supports pagination - we’ve described how to to enable it in our documentation.

Two other major improvements concern PostgreSQL and H2. For PostgreSQL, we’ve removed system templates from the "SYSADMIN.getCatalogs" procedure results, and for H2, we’ve updated the JDBC driver to version 2.4.240. Please note that now only databases of version 2.x or later are supported. If your database is currently using version 1.x, please migrate it to a newer version before upgrading.

We’ve also added support for UTF-8-BOM encoding both on the Server and in Studio wizards. For the Studio, we’ve added the "Old password" field to the password change wizard. This entailed one important change to the SYSADMIN.changeUserPwd procedure: it now
requires the oldPwd parameter when users change their own password. Administrative users don’t need to specify oldPwd when changing passwords for other users, but still must provide oldPwd when changing their own password.

As for bug fixes, we’ve fixed a bug affecting Clustering where the Infinispan cache expiration lifespan configuration caused duplicate materialized tables and undesirable job queue cleanup. For our REST API, we’ve resolved a bug causing the error message in XML response to be empty or have incorrect format and two bugs with the "array=true" parameter: one where setting it for the source endpoint had no effect and another where it was ignored for the "query" endpoint when "Accept" header is set to "application/json".

We’ve also resolved the issue with arguments of "AES_ENCRYPT" and "AES_DECRYPT" functions not being masked in logs and another issue preventing changes in LDAP and SSO role mappings from being applied to the existing sessions after running the "SYSADMIN.refreshLdapUserCache" procedure.

For the Exporter, we’ve fixed the bug preventing ODBC settings from being exported. Now all is well here.

As for the Connectors, we’ve created a new TikTok Shop connector and identified duplicate data with automation for several connectors: Bing Ads, Google Search Console, Google Sheets, Google Ads API, Adjust, and Impact. In addition, our Walmart connector now supports authentication with RefreshToken. Some of the connectors also received bug fixes: we’ve fixed the bug with the Google Ads API connector where the Performance_Campaign procedure returned Internal error, another bug with the Adjust connector which caused duplicate rows to be created when calling the Reports procedure (as part of the solution, we’ve removed all rejected_* dimensions from this procedure), and yet another with the Collibra connector where relationships were not imported.

Here are all issues in this release:

Server

• DVCORE-9038 (New Feature): Implement FIPS mode support for SSL connections

• DVCORE-9218 (Improvement): PostgreSQL: remove system templates from the "SYSADMIN.getCatalogs" procedure results

• DVCORE-9095 (Improvement): Git Integration: improve storage of recommended indexes

• DVCORE-9091 (Improvement): Git Integration: improve storage of schedules and jobs

• DVCORE-9085 (Improvement): Git Integration: improve storage of foreign functions

• DVCORE-9083 (Improvement): Git Integration: improve storage of users and their roles

• DVCORE-9082 (Improvement): Git Integration: store job owner, executor, and "enabled" flag in the same file as the job

• DVCORE-9081 (Improvement): Git Integration: store materialized tables together with corresponding recommended optimizations

• DVCORE-9080 (Improvement): Git Integration: store recommended optimizations in separate files

• DVCORE-9079 (Improvement): Git Integration: store materialization jobs in the "jobs" folder

• DVCORE-8834 (Improvement): Git Integration: add storage of permissions for LDAP roles and improve the related Git folder structure

• DVCORE-9062 (Improvement): H2: update the JDBC driver to version 2.4.240

• DVCORE-8725 (Improvement): Add support for UTF-8-BOM encoding both on the server and in Studio wizards

• DVCORE-7216 (Improvement): LDAP Authentication: add support for pagination

• DVCORE-8215 (Improvement): Add UUID field to the procedure arguments of "SYSADMIN.getJobProperty" and "SYSADMIN.setJobProperty"

• DVCORE-9251 (Bug Fix): Snowflake: DROP TABLE is slow on a source schema with an
  underscore in the schema name

• DVCORE-9248 (Bug Fix): Invalid view definition stored in the configuration database blocks the server from starting

• DVCORE-9244 (Bug Fix): The fix implemented in DVCORE-8881 causes significant performance issues and should be rolled back

• DVCORE-9234 (Bug Fix): Permissions are loaded twice during the "SYSADMIN.refreshTables" procedure call

• DVCORE-9191 (Bug Fix): Clustering: Infinispan cache expiration lifespan configuration causes duplicate materialized tables and undesirable job queue cleanup

• DVCORE-9187 (Bug Fix): CData Virtuality REST API: setting the “array=true” parameter for the source endpoint has no effect

• DVCORE-9174 (Bug Fix): CData Virtuality REST API: the "array=true" parameter is ignored for the "query" endpoint when "Accept" header is set to "application/json"

• DVCORE-9172 (Bug Fix): Incorrect results are returned by the LEFT JOIN when multiple data sources are used in the query

• DVCORE-9163 (Bug Fix): Procedure with NULL in the projection of a UNION query causes a type conversion error and prevents server startup

• DVCORE-9154 (Bug Fix): Arguments of "AES_ENCRYPT" and "AES_DECRYPT" functions are not masked in logs

• DVCORE-9145 (Bug Fix): "maxRetries" and "responseTimeout" values are not logged to
  "SYSLOG.JobEmailNotificationHistory" system table

• DVCORE-9111 (Bug Fix): Google BigQuery: base64 marker is absent for password property in SYSADMIN.CliTemplates

• DVCORE-9066 (Bug Fix): Changes in LDAP and SSO role mappings are not applied to the
  existing sessions after running the "SYSADMIN.refreshLdapUserCache" procedure

• DVCORE-9055 (Bug Fix): VIEW with a column named as a reserved word cannot be created

• DVCORE-9033 (Bug Fix): SELECT INTO statement works incorrectly inside anonymous
  procedure blocks

• DVCORE-9022 (Bug Fix): Temporary table creation with the same name incorrectly succeeds in both outer and inner blocks if using SELECT INTO statement

• DVCORE-9014 (Bug Fix): SFTP: "saveFile" procedure calls hang in some cases

• DVCORE-8981 (Bug Fix): DATE_TRUNC function returns the wrong data type

• DVCORE-8913 (Bug Fix): Snowflake: "historyUpdate" procedure does not update pseudo-
  infinite date of deleted rows

• DVCORE-8798 (Bug Fix): A temporary table name cannot be resolved in a SELECT INTO
  statement with a CTE

• DVCORE-8745 (Bug Fix): CData Virtuality REST API: error message in XML response is empty or has incorrect format

• DVCORE-8701 (Bug Fix): Git Integration: strings are not properly escaped on storing to the
  repository

• DVCORE-7594 (Bug Fix): Error Handler: Multi-line errors displayed wrongly

• DVCORE-8297 (Bug Fix): OData: reading "users" table for "microsoft.graph" namespace fails

• DVCORE-8062 (Bug Fix): Git Integration: "SYSADMIN.purgeObjects" procedure throws an error with 'jobs' item listed multiple times if an incorrect filter value is passed

Studio

•  DVCORE-8990 (Improvement): Add "Old password" field to wizard to change the password

Exporter

• DVCORE-8666 (Bug Fix): CData Virtuality Exporter: ODBC settings are not exported

Connectors

• SQL-1123 (New Feature): TikTok Shop: create a connector

• SQL-1138 (Improvement): Bing Ads: identify duplicate rows with automation

• SQL-1129 (Improvement): Google Search Console: identify duplicate data with automation

• SQL-1128 (Improvement): Google Sheets: identify duplicate data with automation

• SQL-1126 (Improvement): Google Ads API : identify duplicate rows with automation

• SQL-1117 (Improvement): Adjust : identify duplicate rows from procedures with automation

• SQL-1115 (Improvement): Target Plus: orders proc does not update data properly removing the initial_date parameter

• SQL-1112 (Improvement): Impact: identify duplicate data of some procedures with automation

• SQL-1109 (Improvement): Walmart: support authentication with RefreshToken

• SQL-1119 (Bug Fix): Google Ads API: Performance_Campaign procedure returns Internal
  error

• SQL-1086 (Bug Fix): Adjust connector: creates duplicate rows when calling the Reports
  procedure removing all rejected_* dimensions from Report procedure

• SQL-930 (Bug Fix): Collibra: relationships are not imported