CData Sync

When bulk loading to Snowflake, data is uploaded to a temp table (named by CData) and then insert/merged into the final target table.Is there a way to specify, either at the Connection level, or the Job level, what Database or Schema should be used to store those tables?  By default it just puts them in the target schema, but it would be nice to split those out so we can more quickly clean anything up after an aborted job.

I now use CdataSync to transfer ActianZen database data to Bigquery through Btrieve Connector.1. Even after the data transfer is completed, if you view it in ActianZen's Monitor, it still exists in ActiveFile, that is to say, the resources have not been released.2. The open mode of the file whose resources have not been released is not the readonly mode, but the normal mode.Because of the above problems, table lock occurs on the system operation page using the ActianZen database.If you know how to solve it, or how I should try to solve it, please tell me, thank you very much.

is it possible for us to configure CData Sync to use more RAM of the VM? For example, when I do a initial replication of 1mil records from Redshift (RA3.16xlarge) to Snowflake (medium wh).. it takes roughly 3 mins.. I noticed Cdata is using only ¼ of the total 32GB RAM available. 

Hello,When creating temp files to be bulk loaded up to Snowflake, CData Sync appears to be creating the files in this folder on our Windows server:file://C:\\Windows\\SERVIC~1\\NETWOR~1\\AppData\\Local\\Temp\\*Is there a way to direct CData to use a different folder, globally, for all temporary workspace?Thank you!

Hello,Has anyone experimented with different batch sizes while sending data into Snowflake?  It looks like the default is 100000 rows.  I am wondering if there are any downsides to trying a higher number.Thanks!

Hi guys, I cannot get the companies and deals table to execute in the data ingestion phase. Anyone has experienced this before? The error I keep getting is:Executing table [HubSpot].[Companies]:failed with error:System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out 

📢 Exciting Updates from CData Sync! Q4 2023 Release Highlights 🌟 We're thrilled to announce the latest features in the Q4 2023 CData Sync release! Here's a quick snapshot:  API Connector - Seamlessly connect with any API without coding. Perfect for users of all technical levels.   Continuous Replication - Quicker, point-and-click replication options for near real-time data access. No more complex setups!  Integration with Databricks Unity Catalog - Embrace unified data governance by managing assets across all platforms efficiently.  🔗 Dive deeper into these updates in our Q4-2023 Sync Release Blog. 🎥 Want a visual guide on the API connector? Check out our instructional

Hello, couple of CSV source related questions…How do I indicate, for a CSV-based source is, what the primary key column is for each table, so that an upsert can be properly done?   Is it possible to move the files to another folder (in the operating system) after Sync has ingested them?  We would want to archive the files so they don’t get re-processed the next time.  I can do this outside of Sync if necessary, but it would be cleaner if Sync could do this somehow (or call out to a batch file on the OS and run that).   In my case, I have many source files with a timestamp, like MYFILE_XYZ_20231006.TXT.  I would like to use the file Aggregation so that I can just pull them all into the same target table.  However when I do that, is there a way to pass along the individual filename being processed, into a column in the target table?Thank you for any assistance you can provide! 

Open Source helps to collaborate, build networks, help a wide range of people, develop skills, and gain real-world experience. But what makes open source so popular and trendy today? It is the technology community that is willing to share knowledge and allow people across the world to contribute. We are here to talk about 5 popular open-source tools that are widely used in the world of data science, data engineering, and analytics and how we are utilizing them in the CData Sync app. DBT is an open-source data build tool used to transform data where it lives, and since the CData Sync app version 22 you can directly run transformation models within the app making sure that users have an enhanced experience. PostgreSQL is an advanced open-source RDBMS. It is one of the most popular RDBMS among data engineers due to its stability, features, and how it is constantly improving its performance. In the

PurposeSteps to create Oracle Destination Connection, which is the native Oracle Managed Data Access Client (Oracle.ManagedDataAccess.dll) in CData Sync .Net Edition.  Step 1: Download ODP.NET_Managed_ODAC.zip from 64-bit Oracle Data Access Components (ODAC) for Windows  Step 2: Copy the Oracle.ManagedDataAccess.dll from D:\Downloads\ODP.NET_Managed_ODAC122cR1.zip\odp.net\mana

Hello, Snowflake seems to be case-sensitive with table names, sort of.  If the incoming table name from Sync gets created with lower or mixed case, it gets created that way in Snowflake, and you have to reference them with quotes later on when querying.  Like    select * from “my_table".  Whereas if it were created as MY_TABLE, you wouldn’t need to bother adding quotes.Some data sources (a SQL server database we have, Active Directory, etc.) have lower or mixed case tables as exposed in CData.Is there a way to uppercase these either on the source as the metadata is read, or, when created in the destination?thank you!

Change Data Capture (CDC) is a crucial technique in modern data management that allows you to track and capture changes made to a database. It provides real-time insights into data modifications, enabling businesses to react swiftly to evolving trends and requirements. In this article, we'll explore what CDC is and how to implement it using CData Sync.Understanding Change Data Capture (CDC)CDC revolves around capturing data changes at the source using a log file, eliminating the need for constant querying. Instead, CData Sync reads the log file for events (Insert, Update, or Delete) and extracts these changes in near real time, ensuring accurate replication for future use.Supported CDC Sources:MySQL: Utilizes binary logs. Oracle: Leverages Oracle Flashback. PostgreSQL: Employs logical replication. SQL Server: Uses either change tracking or change data capture.

INFO: Get app data directory form /Users/mac/Downloads/CDataSync /sync.properties file. nullAug 18, 2023 3:51:40 PM cdata.sync.SyncServer aINFO: The server log directory is /Users/mac/Downloads/CDataSync /logs/httpserverAug 18, 2023 3:51:40 PM b.d qINFO: Init Jetty log with default settings.Aug 18, 2023 3:51:40 PM b.d bINFO: Starting server from /Users/mac/Downloads/CDataSync /sync.properties configuration.Aug 18, 2023 3:51:40 PM d.a aINFO: The properties deserializes success.Aug 18, 2023 3:51:41 PM b.d cINFO: Enabling embedded Jsp support...Aug 18, 2023 3:51:41 PM b.d cINFO: Successfully enabled embedded Jsp support.2023-08-18 15:51:41.719:INFO :oejs.Server:main: jetty-10.0.14; built: 2023-02-22T23:12:32.272Z; git: 976721d0f3e903a243584d47870ad2f2c1bf9e55; jvm 17.0.8+9-LTS-2112023-08-18 15:51:49.684:WARN :oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@7d3a22a9

Hi Community, new to CData Sync here.When am trying to extract a certain table from Workday through the inbuilt Workday source connection, an getting the error - [0] Cannot invoke "java.util.List.iterator()" because "" is null Could anyone help with the error please. Thanks in advance.  

Dear all, apologizes in advance for the probably silly question, but I am new at this and I could not find any examplea customer of mine is sending me a file always with the same filename, and I need to be able to save it and be sure that it will not be overwrittenfor example adding a timestamp would be perfect (but preserving the extension)EUC2313.xml → EUC2312_20230828101211.xmlEUC2313.xml → EUC2312_20230828101313.xmlhow could I do this ? I can not use filepath or filename

Hi Everyone!The latest release of CData Sync has arrived! Take a closer look at what's new and how it can benefit you. Visit these resources to learn more:BlogWhat’s New in the Q3 Sync releaseKnowledge Base ArticlesEnterprise features Transformation librariesExperience what’s new yourself!Start a 30 day trial 

Executing SF_MirrorAll results in an error as indicated below.  It occurs both with FullCopy or DeltaCopy.  I have performed various searches but haven’t found a solution.  --- Starting SF_Mirror for WorkStepStatus V8304 [SQLSTATE 01000]01:53:03: Parameters: SALESFORCE_Med_ARCH_FULL WorkStepStatus  Version: V8304 [SQLSTATE 01000]01:53:04: DBAmpAZ 22.0.8304.0 Copyright c 2021 CData Software, Inc. [SQLSTATE 01000]01:53:04: Obtaining Connection Props [SQLSTATE 01000]01:53:04: Obtaining User Information [SQLSTATE 01000]01:53:04: Using settings from Registry. [SQLSTATE 01000]01:53:05: Using FullCopy because the WorkStepStatus table does not exist. [SQLSTATE 01000]01:53:05: Server: WUIT-S-14261, Database: MedARCH_SF_FULL, Linked Server: SALESFORCE_Med_ARCH_FULL [SQLSTATE 01000]01:53:05: Using the Salesforce BulkAPI. [SQLSTATE 01000]01:53:06: Create WorkStepStatus_Previous20230828015305925 with new structure. [SQLSTATE 010

Hi, I’ve been successfully using Sync to replicate data from Eleasticsearch index to Azure Data Lake. Now I have faced an issue one of index that is bigger than others (not much, but obviously enough bigger). In replication log there are following errors: [META|Cache ] Cache failed: Updating Row is failed. ERROR: Invalid HTTP response! Cannot upload PostData: System error: An existing connection was forcibly closed by the remote host., AffectedCount: 5200000 [SQL |Messag] Query Failed: [CACHE [uspa/spaiderloki/arkisto/2023] SELECT [_id], [Aika], [Atunnus], [Caller], [Callsite], [cdata_timestamp], [Duration], [Exception], [Filter], [guid], [ID], [kafka_partition], [Logger], [LogLevel], [Message], [Thread], [Version] FROM [Elasticsearch].[uspa-spaiderloki_arkisto_20233x]]. Error: Updating Row is failed. ERROR: Invalid HTTP response! Cannot upload PostData: System error: An existing connection was forcibly closed by the remote host., AffectedCo

Can i get the Twitter Connector API details for the Cdata platform ? 

When I attempt to replicate ContentDocumentLink with CData Sync from Salesforce to SQL, I am getting this error. [0] [500] Could not execute the specified command: [sf:MALFORMED_QUERY] MALFORMED_QUERY: Implementation restriction: ContentDocumentLink requires a filter by a single Id on ContentDocumentId or LinkedEntityId using the equals operator or multiple Id's using the IN operator. The tool that I am looking to replace with CData Sync can do it.  Is there a workaround to get this object to replicate? Thanks

Does anyone here have any experience with the CData Sync email connector? I want to know what type of functionality it has - as in, can it read attached files and replicate those, etc?

Hi all, Was wondering if anyone has any ideas regarding a NetSuite connector that will allow me to add the connection but displays no tasks when adding a new job.  The connector is using an OAuth token to auth against SuiteQL. It can connect sucessfully, although I can’t create any tasks as it says “No Tasks found. Please verify your connection has access to tables”. The permissions in NetSuite were set according to “https://cdn.cdata.com/help/DNH/cloud/default.htm#pg_permissions”. Any help would be appreciated!

Has anyone ever used the REST API Source to get to MS Teams Auto Attendant and/or Call Queue data?Any help is appreciated.

I am using cdata sync and getting the error “EXCEEDED_ID_LIMIT: there are '600000' or more records deleted in the specified time period. Please retry getDeleted() call using a shorter time period.”I have seen other posts say to use a Full vs partial copy but I am not sure that is something that Cdata sync does.   Is there a way go get beyond this error in cdata or salesforce?

Our newest Sync release is now generally available! In this release, as part of our ongoing work to improve the Sync user experience, we employed a Voice of the Customer methodology, soliciting and listening to customer feedback about navigation flows, tasks that would benefit from a more streamlined approach, and the need for improved schema support for common use cases. Read the blog to learn more.