CData Sync

Share product questions and get support for CData Sync

  • 57 Topics
  • 103 Replies

57 Topics

W
worldwidehowApprentice
asked in CData Sync

Make CData Sync go even faster

is it possible for us to configure CData Sync to use more RAM of the VM? For example, when I do a initial replication of 1mil records from Redshift (RA3.16xlarge) to Snowflake (medium wh).. it takes roughly 3 mins.. I noticed Cdata is using only ¼ of the total 32GB RAM available. 

4
W
5 months ago
D
DougN NouriaEnthusiast
asked in CData Sync

Directory for local file creation cache for Snowflake

Hello,When creating temp files to be bulk loaded up to Snowflake, CData Sync appears to be creating the files in this folder on our Windows server:file://C:\\Windows\\SERVIC~1\\NETWOR~1\\AppData\\Local\\Temp\\*Is there a way to direct CData to use a different folder, globally, for all temporary workspace?Thank you!

3
Ethem Q
5 months ago
D
DougN NouriaEnthusiast
asked in CData Sync

Batch size / Commit size for Snowflake target

Hello,Has anyone experimented with different batch sizes while sending data into Snowflake?  It looks like the default is 100000 rows.  I am wondering if there are any downsides to trying a higher number.Thanks!

4
Ankit Singh
5 months ago
M
mmaxellApprentice
asked in CData Sync

HubSpot ado.net connector

Hi guys, I cannot get the companies and deals table to execute in the data ingestion phase. Anyone has experienced this before? The error I keep getting is:Executing table [HubSpot].[Companies]:failed with error:System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out 

3
Ethem Q
6 months ago
Jaclyn
JaclynEmployee
posted in CData Sync

CData Sync Q4-2023 Release

📢 Exciting Updates from CData Sync! Q4 2023 Release Highlights 🌟 We're thrilled to announce the latest features in the Q4 2023 CData Sync release! Here's a quick snapshot:  API Connector - Seamlessly connect with any API without coding. Perfect for users of all technical levels.   Continuous Replication - Quicker, point-and-click replication options for near real-time data access. No more complex setups!  Integration with Databricks Unity Catalog - Embrace unified data governance by managing assets across all platforms efficiently.  🔗 Dive deeper into these updates in our Q4-2023 Sync Release Blog. 🎥 Want a visual guide on the API connector? Check out our instructional video here. 📄 For an in-depth look at all new features, head over to our Current Release Documentation. Ready to experience these features?  🆓 Try them out with a Free Trial.  Already a CData user? Upgrade now and elevate your data management game!  Cheers to continuous innovation! 🚀 The CData Sync Team

0
Jaclyn
6 months ago
D
DougN NouriaEnthusiast
asked in CData Sync

CData Sync - CSV Source file questions

Hello, couple of CSV source related questions…How do I indicate, for a CSV-based source is, what the primary key column is for each table, so that an upsert can be properly done?   Is it possible to move the files to another folder (in the operating system) after Sync has ingested them?  We would want to archive the files so they don’t get re-processed the next time.  I can do this outside of Sync if necessary, but it would be cleaner if Sync could do this somehow (or call out to a batch file on the OS and run that).   In my case, I have many source files with a timestamp, like MYFILE_XYZ_20231006.TXT.  I would like to use the file Aggregation so that I can just pull them all into the same target table.  However when I do that, is there a way to pass along the individual filename being processed, into a column in the target table?Thank you for any assistance you can provide! 

2
D
6 months ago
N
Nandini BaggaEmployee
posted in CData Sync

CData Sync app and open-source tools going strong

Open Source helps to collaborate, build networks, help a wide range of people, develop skills, and gain real-world experience. But what makes open source so popular and trendy today? It is the technology community that is willing to share knowledge and allow people across the world to contribute. We are here to talk about 5 popular open-source tools that are widely used in the world of data science, data engineering, and analytics and how we are utilizing them in the CData Sync app. DBT is an open-source data build tool used to transform data where it lives, and since the CData Sync app version 22 you can directly run transformation models within the app making sure that users have an enhanced experience. PostgreSQL is an advanced open-source RDBMS. It is one of the most popular RDBMS among data engineers due to its stability, features, and how it is constantly improving its performance. In the CData Sync app, you can use PostgresSQL as a source and destination both. We support modern

0
N
6 months ago
N
Nandini BaggaEmployee
posted in CData Sync

How to create Oracle Destination Connection in CData Sync - .NET Edition ?

PurposeSteps to create Oracle Destination Connection, which is the native Oracle Managed Data Access Client (Oracle.ManagedDataAccess.dll) in CData Sync .Net Edition.  Step 1: Download ODP.NET_Managed_ODAC.zip from 64-bit Oracle Data Access Components (ODAC) for Windows  Step 2: Copy the Oracle.ManagedDataAccess.dll from D:\Downloads\ODP.NET_Managed_ODAC122cR1.zip\odp.net\managed\common Step 3: Put it in www/bin folder Step 4: Add provider in web.config file<remove invariant="Oracle.ManagedDataAccess.Client" /><add name="ODP.NET, Managed Driver" invariant="Oracle.ManagedDataAccess.Client" description="Oracle Data Provider for .NET, Managed Driver" type="Oracle.ManagedDataAccess.Client.OracleClientFactory, Oracle.ManagedDataAccess, Version=4.122.1.0, Culture=neutral, PublicKeyToken=89b483f429c47342"/> Step 5: Check the Version, Culture and PublicKeyToken to match your DLL’s. Steps to check Version, Culture and PublicKeyToken: Open Visual Studio’s Developer Command Prompt

0
N
6 months ago
D
DougN NouriaEnthusiast
asked in CData Sync

Uppercasing the destination table name in Snowflake

Hello, Snowflake seems to be case-sensitive with table names, sort of.  If the incoming table name from Sync gets created with lower or mixed case, it gets created that way in Snowflake, and you have to reference them with quotes later on when querying.  Like    select * from “my_table".  Whereas if it were created as MY_TABLE, you wouldn’t need to bother adding quotes.Some data sources (a SQL server database we have, Active Directory, etc.) have lower or mixed case tables as exposed in CData.Is there a way to uppercase these either on the source as the metadata is read, or, when created in the destination?thank you!

2
D
7 months ago
E
eneopEmployee
posted in CData Sync

Understanding Change Data Capture and Creating CDC job in CData Sync

Change Data Capture (CDC) is a crucial technique in modern data management that allows you to track and capture changes made to a database. It provides real-time insights into data modifications, enabling businesses to react swiftly to evolving trends and requirements. In this article, we'll explore what CDC is and how to implement it using CData Sync.Understanding Change Data Capture (CDC)CDC revolves around capturing data changes at the source using a log file, eliminating the need for constant querying. Instead, CData Sync reads the log file for events (Insert, Update, or Delete) and extracts these changes in near real time, ensuring accurate replication for future use.Supported CDC Sources:MySQL: Utilizes binary logs. Oracle: Leverages Oracle Flashback. PostgreSQL: Employs logical replication. SQL Server: Uses either change tracking or change data capture.Creating a CDC Job in CData SyncTo initiate a job in CData Sync, ensure you have pre-configured source and destination connectio

1
Ankit Singh
7 months ago
A
abdullah028Apprentice
asked in CData Sync

HTTP ERROR 404 Not Found java.util.zip.ZipException: invalid entry size (expected 5623126 but got 5623165 bytes) while installing CData synch using java -jar sync.jar on macos

INFO: Get app data directory form /Users/mac/Downloads/CDataSync /sync.properties file. nullAug 18, 2023 3:51:40 PM cdata.sync.SyncServer aINFO: The server log directory is /Users/mac/Downloads/CDataSync /logs/httpserverAug 18, 2023 3:51:40 PM b.d qINFO: Init Jetty log with default settings.Aug 18, 2023 3:51:40 PM b.d bINFO: Starting server from /Users/mac/Downloads/CDataSync /sync.properties configuration.Aug 18, 2023 3:51:40 PM d.a aINFO: The properties deserializes success.Aug 18, 2023 3:51:41 PM b.d cINFO: Enabling embedded Jsp support...Aug 18, 2023 3:51:41 PM b.d cINFO: Successfully enabled embedded Jsp support.2023-08-18 15:51:41.719:INFO :oejs.Server:main: jetty-10.0.14; built: 2023-02-22T23:12:32.272Z; git: 976721d0f3e903a243584d47870ad2f2c1bf9e55; jvm 17.0.8+9-LTS-2112023-08-18 15:51:49.684:WARN :oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@7d3a22a9{/,null,STOPPED}{/Users/mac/Downloads/CDataSync /webapp/sync.war}java.util.zip.ZipException: invalid

1
A
7 months ago
B
bkilariApprentice
asked in CData Sync

Cannot invoke "java.util.List.iterator()" because "" is null

Hi Community, new to CData Sync here.When am trying to extract a certain table from Workday through the inbuilt Workday source connection, an getting the error - [0] Cannot invoke "java.util.List.iterator()" because "" is null Could anyone help with the error please. Thanks in advance.  

1
Ethem Q
7 months ago
B
bvbert1969Apprentice
asked in CData Sync

rename file after receiving

Dear all, apologizes in advance for the probably silly question, but I am new at this and I could not find any examplea customer of mine is sending me a file always with the same filename, and I need to be able to save it and be sure that it will not be overwrittenfor example adding a timestamp would be perfect (but preserving the extension)EUC2313.xml → EUC2312_20230828101211.xmlEUC2313.xml → EUC2312_20230828101313.xmlhow could I do this ? I can not use filepath or filename

6
James B
8 months ago
D
Deb Cobb_PMMEmployee
posted in CData Sync

CData Sync Q3 2023 Release

Hi Everyone!The latest release of CData Sync has arrived! Take a closer look at what's new and how it can benefit you. Visit these resources to learn more:BlogWhat’s New in the Q3 Sync releaseKnowledge Base ArticlesEnterprise features Transformation librariesExperience what’s new yourself!Start a 30 day trial  Upgrade to the latest release And of course, let us know what you think!All the best,The CData Sync Team

0
D
8 months ago
R
RonMooApprentice
asked in CData Sync

Error: Entity 'WorkStepStatus' is not supported by the Bulk API.

Executing SF_MirrorAll results in an error as indicated below.  It occurs both with FullCopy or DeltaCopy.  I have performed various searches but haven’t found a solution.  --- Starting SF_Mirror for WorkStepStatus V8304 [SQLSTATE 01000]01:53:03: Parameters: SALESFORCE_Med_ARCH_FULL WorkStepStatus  Version: V8304 [SQLSTATE 01000]01:53:04: DBAmpAZ 22.0.8304.0 Copyright c 2021 CData Software, Inc. [SQLSTATE 01000]01:53:04: Obtaining Connection Props [SQLSTATE 01000]01:53:04: Obtaining User Information [SQLSTATE 01000]01:53:04: Using settings from Registry. [SQLSTATE 01000]01:53:05: Using FullCopy because the WorkStepStatus table does not exist. [SQLSTATE 01000]01:53:05: Server: WUIT-S-14261, Database: MedARCH_SF_FULL, Linked Server: SALESFORCE_Med_ARCH_FULL [SQLSTATE 01000]01:53:05: Using the Salesforce BulkAPI. [SQLSTATE 01000]01:53:06: Create WorkStepStatus_Previous20230828015305925 with new structure. [SQLSTATE 01000]01:53:06: Error: Entity 'WorkStepStatus' is not supported by the Bul

1
R
8 months ago
J
JTapioApprentice
asked in CData Sync

[META|Cache ] Cache failed: Updating Row is failed. ERROR: Invalid HTTP response!

Hi, I’ve been successfully using Sync to replicate data from Eleasticsearch index to Azure Data Lake. Now I have faced an issue one of index that is bigger than others (not much, but obviously enough bigger). In replication log there are following errors: [META|Cache ] Cache failed: Updating Row is failed. ERROR: Invalid HTTP response! Cannot upload PostData: System error: An existing connection was forcibly closed by the remote host., AffectedCount: 5200000 [SQL |Messag] Query Failed: [CACHE [uspa/spaiderloki/arkisto/2023] SELECT [_id], [Aika], [Atunnus], [Caller], [Callsite], [cdata_timestamp], [Duration], [Exception], [Filter], [guid], [ID], [kafka_partition], [Logger], [LogLevel], [Message], [Thread], [Version] FROM [Elasticsearch].[uspa-spaiderloki_arkisto_20233x]]. Error: Updating Row is failed. ERROR: Invalid HTTP response! Cannot upload PostData: System error: An existing connection was forcibly closed by the remote host., AffectedCount: 5200000 Updating Row is failed. ERROR: I

1
P
9 months ago
D
DevApprentice
asked in CData Sync

Twitter API Details

Can i get the Twitter Connector API details for the Cdata platform ? 

2
D
9 months ago
T
tlallyApprentice
asked in CData Sync

Need to replicate ContentDocumentLink from Salesforce

When I attempt to replicate ContentDocumentLink with CData Sync from Salesforce to SQL, I am getting this error. [0] [500] Could not execute the specified command: [sf:MALFORMED_QUERY] MALFORMED_QUERY: Implementation restriction: ContentDocumentLink requires a filter by a single Id on ContentDocumentId or LinkedEntityId using the equals operator or multiple Id's using the IN operator. The tool that I am looking to replace with CData Sync can do it.  Is there a workaround to get this object to replicate? Thanks

1
A
9 months ago
A
ashleypaceRSFApprentice
asked in CData Sync

CData Sync Email Connector

Does anyone here have any experience with the CData Sync email connector? I want to know what type of functionality it has - as in, can it read attached files and replicate those, etc?

1
E
10 months ago
F
FishXVApprentice
asked in CData Sync

NetSuite Connector - No Tasks Found

Hi all, Was wondering if anyone has any ideas regarding a NetSuite connector that will allow me to add the connection but displays no tasks when adding a new job.  The connector is using an OAuth token to auth against SuiteQL. It can connect sucessfully, although I can’t create any tasks as it says “No Tasks found. Please verify your connection has access to tables”. The permissions in NetSuite were set according to “https://cdn.cdata.com/help/DNH/cloud/default.htm#pg_permissions”. Any help would be appreciated!

2
F
10 months ago
B
bweatherfordApprentice
asked in CData Sync

CData Sync and MS Teams REST API

Has anyone ever used the REST API Source to get to MS Teams Auto Attendant and/or Call Queue data?Any help is appreciated.

4
Ankit Singh
10 months ago
A
AmbroseApprentice
asked in CData Sync

EXCEEDED_ID_LIMIT: there are '600000' or more records deleted in the specified time period. Please retry getDeleted() call using a shorter time period.

I am using cdata sync and getting the error “EXCEEDED_ID_LIMIT: there are '600000' or more records deleted in the specified time period. Please retry getDeleted() call using a shorter time period.”I have seen other posts say to use a Full vs partial copy but I am not sure that is something that Cdata sync does.   Is there a way go get beyond this error in cdata or salesforce?

1
E
11 months ago
D
Deb Cobb_PMMEmployee
posted in CData Sync

New Sync release - modern UI and even better hybrid cloud connectivity

Our newest Sync release is now generally available! In this release, as part of our ongoing work to improve the Sync user experience, we employed a Voice of the Customer methodology, soliciting and listening to customer feedback about navigation flows, tasks that would benefit from a more streamlined approach, and the need for improved schema support for common use cases. Read the blog to learn more.

0
D
11 months ago
D
Deb Cobb_PMMEmployee
posted in CData Sync

CData Sync flyer datasheet

Take a look at CData Sync capabilities.

1
John C
1 year ago
James T
James TEmployee
asked in CData Sync

Why don't I see any records returned for my Sage Intacct tables?

I know I have data in my source but there a 0 records returned in my destination!

1
James T
1 year ago
Virtual Event: Data Virtualization, Reimagined Ma”arch 14, 2024 style=

Badge winners

  • Product Expert
    jussihihas earned the badge Product Expert
  • Product Expert
    TonyLhas earned the badge Product Expert
  • Product Expert
    Sagehas earned the badge Product Expert
  • Product Expert
    Ruppiehas earned the badge Product Expert
  • Product Expert
    ekanshdhas earned the badge Product Expert
Show all badges
Terms and ConditionsCookie settings