CData Sync

Hi!I am working with a client who uses NetSuite ERP. We are trying to get Sales reports for them, which we are doing by using the transaction table from SuiteQL schema.We also need the transactions’ status, for which we need the transactionStatus table to load through Sync. On doing so, I am getting the following error :“[0] [INVALID_PARAMETER] Invalid search query. Detailed unprocessed description follows. Search error occurred: Record 'transactionStatus' was not found. Bad Request.” Please help me load this table.​​​​​​​

You can use post-job Events to run some arbitrary SQL (like a stored procedure) at the end of your Job.The documentation says you have to make your own connection, as below.  But I’d really rather not embed cleartext credentials in multiple jobs.  Is there a way to specify a CData connection (which we’ve already established at this point), in the Event?What I’d really like is the ability to run arbitrary SQL commands against the target table, after each Task completes, not just at the end of the Job.  Seems like this could be a parameter that we supply as we define each Task.<api:set attr="db.driver" value="System.Data.SqlClient"/> <api:set attr="db.conn" value="Data Source=sqlserver;Initial Catalog=Test;User ID=user;Password=pass;"/> <api:set attr="db.query">INSERT INTO MyLog (JobName, RunDate, Results) VALUES ('[_input.JobName]', '[null | now()]', '[_input.JobStatus]')</api:set>

Using CData Sync Cross Platform with Workday WQL connection. Some queries work successfully, but a bunch fail with this message:[0] Received error from Workday: Invalid WQL query: !**! This error occurred during a request to calculate the field !**!, for instance !**!.. Cause: com.workday.exceptions.ExternalFieldException: The Increment or Decrement Date field, !**!, has encountered a blank date..  Thanks.

Trying to connect to Min.io on-prem S3 bucket. Confirmed API is working:  However, I don’t know how to point to https://nas2.piorl.us:9000/bucket1/ in CDataSync Cross-Platform Edition. I’ve tried defining the full path in the Bucket field as well as “Path” in advanced options, but neither want to work. There is a reference to a “Custom URL” field in the advanced connection properties, but I can’t find the variable anywhere:  Thanks.

Hi, So I have a system with CDataSync to sync my data from SQL Server to Azure Synapse with below detail: Old System:CDataSync Version: 22.0.8342.0CData AzureSynapse Provider Version: 22.0.8326.0 New System:CDataSync Version: 23.3.8704.0CData AzureSynapse (23.0.8616.0) I backup the old configuration and restore it to new configuration. However, it keeps getting an error like this Error: The SQL Error Number is 103010, State is 1, Severity is 16, Error is 'Parse error at line: 1, column: 126: Incorrect syntax near '.'.', Server name is server1, Line number is 1. The Token Type is 170. I found that the task is successful when I use this conditions:I dont use a new scheme for the destination, but I checked the Drop Table/Truncate Table I use a new scheme, but I unchecked the Drop Table/Truncate TableIs there a new query or options that changed in the new version? What can I do to fix the error? Thank you

I’m attempting to add a task to an existing job and get an error:“The job <job-name> contains an unlicensed connection: Only 10 tasks are allowed in the Job” I also tried creating a job from this existing job and got an error:The connection [Salesforce_abc] is unlicensed: Unlicensed connections cannot be shared across multiple jobs. We have an Enterprise license, and my job is using SalesForce as a source and Azure SQL as a destination. I tried adding a task to a different job that uses Salesforce as source and Azure SQL as destination, and it worked.

Hello CDATA Community,I hope this message finds you well. I am currently facing an issue with the CDATA Sync tool and would appreciate any insights or assistance from the community.Problem Description: I have configured a job with a single task, where the source is Salesforce, and the destination is an SQL database. Initially, I ran the task without incremental replication, and it successfully synchronized all records for a specific Salesforce object. Later, I implemented incremental replication and verified that it worked as expected. However, when I decided to remove incremental replication and attempted to sync the data again, the job seems to persist in using incremental replication, and not all data is being synchronized. Steps Performed:Created a job with Salesforce as the source and SQL database as the destination. Ran the task without incremental replication, successfully syncing all records. Implemented incremental replication and verified its functionality. Removed incrementa

When bulk loading to Snowflake, data is uploaded to a temp table (named by CData) and then insert/merged into the final target table.Is there a way to specify, either at the Connection level, or the Job level, what Database or Schema should be used to store those tables?  By default it just puts them in the target schema, but it would be nice to split those out so we can more quickly clean anything up after an aborted job.

I now use CdataSync to transfer ActianZen database data to Bigquery through Btrieve Connector.1. Even after the data transfer is completed, if you view it in ActianZen's Monitor, it still exists in ActiveFile, that is to say, the resources have not been released.2. The open mode of the file whose resources have not been released is not the readonly mode, but the normal mode.Because of the above problems, table lock occurs on the system operation page using the ActianZen database.If you know how to solve it, or how I should try to solve it, please tell me, thank you very much.

is it possible for us to configure CData Sync to use more RAM of the VM? For example, when I do a initial replication of 1mil records from Redshift (RA3.16xlarge) to Snowflake (medium wh).. it takes roughly 3 mins.. I noticed Cdata is using only ¼ of the total 32GB RAM available. 

Hello,When creating temp files to be bulk loaded up to Snowflake, CData Sync appears to be creating the files in this folder on our Windows server:file://C:\\Windows\\SERVIC~1\\NETWOR~1\\AppData\\Local\\Temp\\*Is there a way to direct CData to use a different folder, globally, for all temporary workspace?Thank you!

Hello,Has anyone experimented with different batch sizes while sending data into Snowflake?  It looks like the default is 100000 rows.  I am wondering if there are any downsides to trying a higher number.Thanks!

Hi guys, I cannot get the companies and deals table to execute in the data ingestion phase. Anyone has experienced this before? The error I keep getting is:Executing table [HubSpot].[Companies]:failed with error:System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out 

📢 Exciting Updates from CData Sync! Q4 2023 Release Highlights 🌟 We're thrilled to announce the latest features in the Q4 2023 CData Sync release! Here's a quick snapshot:  API Connector - Seamlessly connect with any API without coding. Perfect for users of all technical levels.   Continuous Replication - Quicker, point-and-click replication options for near real-time data access. No more complex setups!  Integration with Databricks Unity Catalog - Embrace unified data governance by managing assets across all platforms efficiently.  🔗 Dive deeper into these updates in our Q4-2023 Sync Release Blog. 🎥 Want a visual guide on the API connector? Check out our instructional video here. 📄 For an in-depth look at all new features, head over to our Current Release Documentation. Ready to experience these features?  🆓 Try them out with a Free Trial.  Already a CData user? Upgrade now and elevate your data management game!  Cheers to continuous innovation! 🚀 The CData Sync Team

Hello, couple of CSV source related questions…How do I indicate, for a CSV-based source is, what the primary key column is for each table, so that an upsert can be properly done?   Is it possible to move the files to another folder (in the operating system) after Sync has ingested them?  We would want to archive the files so they don’t get re-processed the next time.  I can do this outside of Sync if necessary, but it would be cleaner if Sync could do this somehow (or call out to a batch file on the OS and run that).   In my case, I have many source files with a timestamp, like MYFILE_XYZ_20231006.TXT.  I would like to use the file Aggregation so that I can just pull them all into the same target table.  However when I do that, is there a way to pass along the individual filename being processed, into a column in the target table?Thank you for any assistance you can provide! 

Open Source helps to collaborate, build networks, help a wide range of people, develop skills, and gain real-world experience. But what makes open source so popular and trendy today? It is the technology community that is willing to share knowledge and allow people across the world to contribute. We are here to talk about 5 popular open-source tools that are widely used in the world of data science, data engineering, and analytics and how we are utilizing them in the CData Sync app. DBT is an open-source data build tool used to transform data where it lives, and since the CData Sync app version 22 you can directly run transformation models within the app making sure that users have an enhanced experience. PostgreSQL is an advanced open-source RDBMS. It is one of the most popular RDBMS among data engineers due to its stability, features, and how it is constantly improving its performance. In the CData Sync app, you can use PostgresSQL as a source and destination both. We support modern

PurposeSteps to create Oracle Destination Connection, which is the native Oracle Managed Data Access Client (Oracle.ManagedDataAccess.dll) in CData Sync .Net Edition.  Step 1: Download ODP.NET_Managed_ODAC.zip from 64-bit Oracle Data Access Components (ODAC) for Windows  Step 2: Copy the Oracle.ManagedDataAccess.dll from D:\Downloads\ODP.NET_Managed_ODAC122cR1.zip\odp.net\managed\common Step 3: Put it in www/bin folder Step 4: Add provider in web.config file<remove invariant="Oracle.ManagedDataAccess.Client" /><add name="ODP.NET, Managed Driver" invariant="Oracle.ManagedDataAccess.Client" description="Oracle Data Provider for .NET, Managed Driver" type="Oracle.ManagedDataAccess.Client.OracleClientFactory, Oracle.ManagedDataAccess, Version=4.122.1.0, Culture=neutral, PublicKeyToken=89b483f429c47342"/> Step 5: Check the Version, Culture and PublicKeyToken to match your DLL’s. Steps to check Version, Culture and PublicKeyToken: Open Visual Studio’s Developer Command Prompt

Hello, Snowflake seems to be case-sensitive with table names, sort of.  If the incoming table name from Sync gets created with lower or mixed case, it gets created that way in Snowflake, and you have to reference them with quotes later on when querying.  Like    select * from “my_table".  Whereas if it were created as MY_TABLE, you wouldn’t need to bother adding quotes.Some data sources (a SQL server database we have, Active Directory, etc.) have lower or mixed case tables as exposed in CData.Is there a way to uppercase these either on the source as the metadata is read, or, when created in the destination?thank you!

Change Data Capture (CDC) is a crucial technique in modern data management that allows you to track and capture changes made to a database. It provides real-time insights into data modifications, enabling businesses to react swiftly to evolving trends and requirements. In this article, we'll explore what CDC is and how to implement it using CData Sync.Understanding Change Data Capture (CDC)CDC revolves around capturing data changes at the source using a log file, eliminating the need for constant querying. Instead, CData Sync reads the log file for events (Insert, Update, or Delete) and extracts these changes in near real time, ensuring accurate replication for future use.Supported CDC Sources:MySQL: Utilizes binary logs. Oracle: Leverages Oracle Flashback. PostgreSQL: Employs logical replication. SQL Server: Uses either change tracking or change data capture.Creating a CDC Job in CData SyncTo initiate a job in CData Sync, ensure you have pre-configured source and destination connectio

INFO: Get app data directory form /Users/mac/Downloads/CDataSync /sync.properties file. nullAug 18, 2023 3:51:40 PM cdata.sync.SyncServer aINFO: The server log directory is /Users/mac/Downloads/CDataSync /logs/httpserverAug 18, 2023 3:51:40 PM b.d qINFO: Init Jetty log with default settings.Aug 18, 2023 3:51:40 PM b.d bINFO: Starting server from /Users/mac/Downloads/CDataSync /sync.properties configuration.Aug 18, 2023 3:51:40 PM d.a aINFO: The properties deserializes success.Aug 18, 2023 3:51:41 PM b.d cINFO: Enabling embedded Jsp support...Aug 18, 2023 3:51:41 PM b.d cINFO: Successfully enabled embedded Jsp support.2023-08-18 15:51:41.719:INFO :oejs.Server:main: jetty-10.0.14; built: 2023-02-22T23:12:32.272Z; git: 976721d0f3e903a243584d47870ad2f2c1bf9e55; jvm 17.0.8+9-LTS-2112023-08-18 15:51:49.684:WARN :oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@7d3a22a9{/,null,STOPPED}{/Users/mac/Downloads/CDataSync /webapp/sync.war}java.util.zip.ZipException: invalid

Hi Community, new to CData Sync here.When am trying to extract a certain table from Workday through the inbuilt Workday source connection, an getting the error - [0] Cannot invoke "java.util.List.iterator()" because "" is null Could anyone help with the error please. Thanks in advance.  

Dear all, apologizes in advance for the probably silly question, but I am new at this and I could not find any examplea customer of mine is sending me a file always with the same filename, and I need to be able to save it and be sure that it will not be overwrittenfor example adding a timestamp would be perfect (but preserving the extension)EUC2313.xml → EUC2312_20230828101211.xmlEUC2313.xml → EUC2312_20230828101313.xmlhow could I do this ? I can not use filepath or filename

Hi Everyone!The latest release of CData Sync has arrived! Take a closer look at what's new and how it can benefit you. Visit these resources to learn more:BlogWhat’s New in the Q3 Sync releaseKnowledge Base ArticlesEnterprise features Transformation librariesExperience what’s new yourself!Start a 30 day trial  Upgrade to the latest release And of course, let us know what you think!All the best,The CData Sync Team

Executing SF_MirrorAll results in an error as indicated below.  It occurs both with FullCopy or DeltaCopy.  I have performed various searches but haven’t found a solution.  --- Starting SF_Mirror for WorkStepStatus V8304 [SQLSTATE 01000]01:53:03: Parameters: SALESFORCE_Med_ARCH_FULL WorkStepStatus  Version: V8304 [SQLSTATE 01000]01:53:04: DBAmpAZ 22.0.8304.0 Copyright c 2021 CData Software, Inc. [SQLSTATE 01000]01:53:04: Obtaining Connection Props [SQLSTATE 01000]01:53:04: Obtaining User Information [SQLSTATE 01000]01:53:04: Using settings from Registry. [SQLSTATE 01000]01:53:05: Using FullCopy because the WorkStepStatus table does not exist. [SQLSTATE 01000]01:53:05: Server: WUIT-S-14261, Database: MedARCH_SF_FULL, Linked Server: SALESFORCE_Med_ARCH_FULL [SQLSTATE 01000]01:53:05: Using the Salesforce BulkAPI. [SQLSTATE 01000]01:53:06: Create WorkStepStatus_Previous20230828015305925 with new structure. [SQLSTATE 01000]01:53:06: Error: Entity 'WorkStepStatus' is not supported by the Bul

Hi, I’ve been successfully using Sync to replicate data from Eleasticsearch index to Azure Data Lake. Now I have faced an issue one of index that is bigger than others (not much, but obviously enough bigger). In replication log there are following errors: [META|Cache ] Cache failed: Updating Row is failed. ERROR: Invalid HTTP response! Cannot upload PostData: System error: An existing connection was forcibly closed by the remote host., AffectedCount: 5200000 [SQL |Messag] Query Failed: [CACHE [uspa/spaiderloki/arkisto/2023] SELECT [_id], [Aika], [Atunnus], [Caller], [Callsite], [cdata_timestamp], [Duration], [Exception], [Filter], [guid], [ID], [kafka_partition], [Logger], [LogLevel], [Message], [Thread], [Version] FROM [Elasticsearch].[uspa-spaiderloki_arkisto_20233x]]. Error: Updating Row is failed. ERROR: Invalid HTTP response! Cannot upload PostData: System error: An existing connection was forcibly closed by the remote host., AffectedCount: 5200000 Updating Row is failed. ERROR: I