Data Integration

Hey CData Community!Moving large volumes of data into Snowflake just got a major performance upgrade. With the release of CData Sync version 24.3, high-volume data replication is now significantly faster and more efficient, cutting job completion times nearly in half for optimized processes.In our recent article, we looked at the enhancements made to CData Sync’s processing engine and explored how these improvements play out across datasets ranging from 1 million to 10 million records.Learn More: https://www.cdata.com/kb/articles/sync-snowflake-performance.rst

OAuth 2.0 Overview In today’s standards, OAuth 2.0 is becoming increasingly popular as a means to authorize users who are attempting to access a resource, be it a mail server, API endpoint or database. A common example of this shift is the deprecation of basic authentication in Exchange Online (Microsoft’s mail server for Outlook), which can be read about in more detail in Microsoft’s article, here.OAuth 2.0 is an authorization protocol and not an authentication protocol; a means of granting users access to a set of resources - in the case of what is discussed here, that resource would be a mail server.At a very high level, OAuth 2.0 can be summarized in the following steps: The connecting client acquires both a client ID and a client secre

Hi All,Sorry for the late response and recreating the  topic. The solution provide did to work as the heading info is no been added to the file. Not sure if this is because of the way I created the flows.   The file are received/collected from the FTP server and place into the folder on my server. Please would some assist with a possible solution. Thank you 

Hi,I have Sync version 24.3.9120.0 and I’m using Apache Kafka as source. It’s working fine, except little irritation. When I run task it will replicate only first 150 000 rows and stops after that. Next time I run the task, it replicates 179 000 rows and so on. Row count will increase. Seem to be randomI can run job multiple times and it will continue where it was stopped earlier. Is there way to change this behavior, so that all rows will be replicated at once?

The first CData Arc update of 2025 is here! The Q1 2025 release brings powerful new capabilities to streamline trading partner management, expand connectivity, and enhance security. Here’s what’s new: 🔹 Enhanced trading partner management – Test workflows end-to-end before deployment to minimize integration errors.🔹 Expanded connectivity & formats – New XML Group connector and ZPL label support for seamless shipping label generation.🔹 Enterprise-grade security – Built-in 2FA and extended Arc Key Vault integration for enhanced data protection.🔹 EU compliance & hosting – Support for AS4, ENTSOG V4, ZUGFeRD, and GDPR-compliant EU hosting.🔹 Migration utili

Since the update to Sync version 25.1.9207 we are getting some strange errors from different tasks.All these tasks worked without problems before the update.So far i have not found any difference (except for the tables) between tasks that work and that won’t work. It even fails on small datasets (19 in this example, a 16.000.000+ tasks works without problem)And the log also is not showing a lot of info:2025-03-25T12:34:16.494+0100    2    [ |Q-Id| T-Id]    [ |Q-Id]    [SQL |Bulk  ] 15 rows are being committed.2025-03-25T12:34:16.494+0100    5    [    2|    0| 1164]    [CORE] [RealDB:SQLServer]Starting WriteToServer2025-03-25T12:34:16.494+0100    5    [    2|    0| 1164]    [CORE] [RealDB:SQLServer]Invoke WriteToServer2025-03-25T12:34:16.508+0100    1    [    2|    0| 1164]    [CORE] [RealDB:S

Hi, We have an XML file that contains a node called /Items/delivery_header/delivery_line_item/serial_numbers and in that node we have a list of serial numbers separated by commas if there is more than one (e.g. 1 serial would just be XYZABC but 2 or more would be XYZABC,XYZDEF). Is there a way on the destination XML to create a loop that can iterate through the comma separated values and create XML nodes like the following: for a single serial number SerialNumber      ManufacturerVersion = XYZABC or for multiple serial number SerialNumber      ManufacturerVersion = XYZABCSerialNumber      ManufacturerVersion = XYZDEF I have got as far as the below in the XML Map designer, I assume that I need to use a Loop with the nodes under it, but not sure what to use to split the values out, populate the ManufacturerVersion and then repeat with the next value: 

Spring is here, bringing fresh updates to CData Sync! Our latest enhancements are designed to streamline your data integration and management, helping you bloom with efficiency this season: 📂 Workspaces for Better Organization: Easily group Jobs, Connections, and Transformations under Workspaces for improved management and streamlined workflows. Users now have greater control over the processes and data, ensuring a more structured and efficient Sync experience. ⚡ Enhanced CDC Catalog Expansion: You asked, we delivered—now supporting DB2 LUW and Informix for even broader data replication capabilities! 🔄 Reverse ETL Catalog Expansion: We are adding Oracle as source and destinations such as Kintone, HubSpot and Zoho CRM to enhance your reverse ETL capabilities! 🛠️ Destination Performance Improvements: Experience faster data processing with increased performance for our Databricks & Redsh

Hi All,I’m try to setup the flow that receives file from a FTP server in multiply folders  and then send an email with the subject containing the sub folder name. The flow I've create is as per  below In the  Navis, a script, connector I've added the below script however I keep receiving the below messageFormatter [ subfolder_path ] failed in the evaluation of [subfolder_path]. The error was: The value of the attribute could not be accessed: The attribute does not exist.please can anyone guid me to a better solution or why I’m receiving the message.<arc:script>  <arc:set attr="output.header:subfolder" value="[subfolder_path]" />  <arc:set attr="output.filepath" value="[filepath]" />  <arc:set attr="output.header:email" value="[email_address]" />

CData Sync supports the execution of ELT processes flexibly using dbt Core or dbt Cloud. This integration gives users complete control over defining and executing complex data transformation workflows in dbt, while CData Sync provides a pipeline for users to run those transformations in your chosen destination. Read more about using dbt Core or dbt Cloud with CData Sync here:dbt Core: https://www.cdata.com/kb/articles/sync-dbt-core.rstdbt Cloud: https://www.cdata.com/kb/articles/sync-dbt-cloud.rst

Hi there.Is there a way of replicating data in batches.We are looking at a lambda approach with small incremental replication, followed up with a weekly full reload to catch late-arriving data and physical/hard deletes.The challenge is that the full reload data volumes are too large for the server hosting our CData Sync. Even with the JAVA heap adjustments.I was wondering if we could do the reload/replication in a looping process.  The source is Snowflake the target is Azure ADLS2 parquet files.

I am working with an incoming EDI file where I use lookaheads to identify the correct loop. In my case, the EDI file contains the Service Level Loop (HL/HL03='SS') appearing twice. While I can successfully extract the first occurrence using the following XPath:xpath("HLLoop1[HL/HL03='SS']/MSG/MSG01")I attempted to retrieve the second occurrence using:xpath("HLLoop1[HL/HL03='SS'][2]/MSG/MSG01")Although this expression did not return an error, it did not extract the expected data either. Has anyone encountered a similar issue? What would be the correct XPath expression to extract both occurrences of the Service Level Loop? Any insights or alternative approaches would be greatly appreciated!

CData Sync can be deployed in a Kubernetes environment. This allows users to take advantage of Kubernetes’ orchestration functionality, including high availability (no downtime), scalability of loads, and backup-and-restore capabilities.Read more about this and follow our guide for this deployment here:https://www.cdata.com/kb/articles/sync-kubernetes.rst

CData Arc supports cloud deployment through the Azure cloud platform via Azure Virtual Machines (VMs) and Azure Web Apps. You can follow our guide for both deployment options here: https://arc.cdata.com/kb/articles/azure-deployment.rst

Hey CData Community!Slowly Changing Dimensions (SCDs) are essential in data warehousing to manage and track changes in dimension data over time.With CData Sync, users can easily implement these SCD models. In our recent article on how to implement SCD using CData Sync, we introduce the most common types—Types 1 to 3—and explain how CData Sync enables you to configure and automate the management of these SCD types without writing any code, making it accessible to users of all technical levels.Read here to know more: https://www.cdata.com/kb/articles/slowly-changing-dimensions.rstIf you have additional questions, please let us know!

CData Sync can be deployed to Microsoft Internet Information Services (IIS), which enables businesses to host Sync as an externally facing website. This allows organizations to utilize the advanced features of IIS, all while keeping the simplicity and functionality of CData Sync's embedded web server.Read more about the two methods you can use to deploy CData Sync in IIS here: Sync IIS Java deployment

What is the CData Arc system service level agreement for the API’s .  e.g. availability ratio, RTO/RPO

Table of Contents Introduction Overview of CData Sync Overview of Terraform Benefits of CData Sync automation with Terraform Prerequisites Terraform configurations VPC module EC2 module Parent module DEMO: CData Sync automatic deployment on AWS Conclusions Key Considerations IntroductionIn today's rapidl

For the Email Send is there a solution for controlling the To: field dynamically with data coming into it from the workflow?Seems that Subject: and From: fields can be, but not much else. 

CData Workday Drivers and Connectors now allow users to create tables that execute specific Workday Query Language (WQL) statements for Workday reports. The new stored procedure, CreateWQLSchema, processes the output from Workday's "Convert Report to WQL" task and retrieves the same report data using the WQL statement. Read more about this Workday Driver enhancement here:Workday: How to Access Live Data from the WQL for a Workday ReportIf you have additional questions, please let us know!

Hi, We recently renewed our license for another year, i have applied it to all dev and Test SQL servers, and the offline node in Prod, but our online node in prod gets the below error.  Can you please advise what to do to resolve this??  We only have a couple of days before our current license expires.ThanksAndrew

Hello,I am currently working on building certain JSON messages.For doing this I am starting with an XML message that has the desired structure and place the data into this XML. Then I use the JSON Connector to convert my XML into the JSON format.So far so good.Due to the fact that sometimes the data of a node does not exist, I would like to know, if there is an option in the JSON Connector that will automatically remove all nodes that are empty in the underlying XML.For example the following XML Structure shows that some fields have values and other do not:<?xml version="1.0" encoding="UTF-8"?><Items xmlns:json="http://arc.cdata.com/ns/jsonconnector" json:array="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">  <Item>    <field1>value1</field1>    <field2>value1</field2>    <field3></field3>    <su

Hi all,Our team is using CData Sync to extract and load data from SQL Server to Snowflake. The source MSSQL table is CDC-enabled, and we have created a CData Job with Change Data Capture enabled. We started testing the task with smaller tables (containing 100K - 300K records), and the task was successful without any issues. However, when trying with larger tables (containing 10M - 35M records), we faced failures every time.I have attached the error output for your reference. Has anyone come across this scenario? Any replies or help would be much appreciated. Thanks in advance.Error output: "2024-11-14T02:25:19.555+0000 2 [32|Q-Id] Failed to cache row [501097] The query was canceled."We tried running the same task with a few manipulations, like reducing the source table records to 1M over 10M but had no luck. We are not facing any other issues besides "Failed to cache row." What does caching mean in a CData

Hey CData Community!We're excited to share the latest CData Arc update for Q1 2024, bringing innovative features that manyof you have been anticipating. Let's dive into what's new and start some discussions around these updates:  Enhanced User Role Granularity & Access Policies: We've upgraded user roles and access policies for better management and security. How do you see these changes impacting your workflow?   Service-Level Agreements (SLAs) for Connectors: With SLAs for connectors, you can now monitor transaction volumes more effectively. What are your thoughts on utilizing SLAs to maintain partner relationships?  

Hi,I am working on EDI 856(CSV to X12).Please help me on this issue.Wanted to generate multiple transactions set in single X12 fileEXAMPLE:ISA*...GS*...ST......SE...ST.....SE...GE...IEA...