Skip to main content

Data Integration

Ask questions, get answers, and engage in discussions about data integration solutions from CData.

97 Topics
D
DLIApprentice
asked in CData DBAmp
Bad performance of DBAmp V22

Why DBAmp V22 SF_MirrorAll takes much more time (couple hours) than V5.1.9 SF_RefreshAll? even I skipped many tables, like %History, %Share... Also there are many error messages: Error occurred creating primary key for table.

3
D
10 months ago
D
DougN NouriaInfluencer
asked in CData Sync
Batch size / Commit size for Snowflake target

Hello,Has anyone experimented with different batch sizes while sending data into Snowflake?  It looks like the default is 100000 rows.  I am wondering if there are any downsides to trying a higher number.Thanks!

4
Ankit Singh
10 months ago
M
mmaxellApprentice
asked in CData Sync
HubSpot ado.net connector

Hi guys, I cannot get the companies and deals table to execute in the data ingestion phase. Anyone has experienced this before? The error I keep getting is:Executing table [HubSpot].[Companies]:failed with error:System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out 

3
Ethem Q
10 months ago
Jaclyn
JaclynEmployee
posted in CData Sync
CData Sync Q4-2023 Release

📢 Exciting Updates from CData Sync! Q4 2023 Release Highlights 🌟 We're thrilled to announce the latest features in the Q4 2023 CData Sync release! Here's a quick snapshot:  API Connector - Seamlessly connect with any API without coding. Perfect for users of all technical levels.   Continuous Replication - Quicker, point-and-click replication options for near real-time data access. No more complex setups!  Integration with Databricks Unity Catalog - Embrace unified data governance by managing assets across all platforms efficiently.  🔗 Dive deeper into these updates in our Q4-2023 Sync Release Blog. 🎥 Want a visual guide on the API connector? Check out our instructional video here. 📄 For an in-depth look at all new features, head over to our Current Release Documentation. Ready to experience these features?  🆓 Try them out with a Free Trial.  Already a CData user? Upgrade now and elevate your data management game!  Cheers to continuous innovation! 🚀 The CData Sync Team

0
Jaclyn
10 months ago
D
DougN NouriaInfluencer
asked in CData Sync
CData Sync - CSV Source file questions

Hello, couple of CSV source related questions…How do I indicate, for a CSV-based source is, what the primary key column is for each table, so that an upsert can be properly done?   Is it possible to move the files to another folder (in the operating system) after Sync has ingested them?  We would want to archive the files so they don’t get re-processed the next time.  I can do this outside of Sync if necessary, but it would be cleaner if Sync could do this somehow (or call out to a batch file on the OS and run that).   In my case, I have many source files with a timestamp, like MYFILE_XYZ_20231006.TXT.  I would like to use the file Aggregation so that I can just pull them all into the same target table.  However when I do that, is there a way to pass along the individual filename being processed, into a column in the target table?Thank you for any assistance you can provide! 

2
D
10 months ago
N
Nandini BaggaEmployee
posted in CData Sync
CData Sync app and open-source tools going strong

Open Source helps to collaborate, build networks, help a wide range of people, develop skills, and gain real-world experience. But what makes open source so popular and trendy today? It is the technology community that is willing to share knowledge and allow people across the world to contribute. We are here to talk about 5 popular open-source tools that are widely used in the world of data science, data engineering, and analytics and how we are utilizing them in the CData Sync app. DBT is an open-source data build tool used to transform data where it lives, and since the CData Sync app version 22 you can directly run transformation models within the app making sure that users have an enhanced experience. PostgreSQL is an advanced open-source RDBMS. It is one of the most popular RDBMS among data engineers due to its stability, features, and how it is constantly improving its performance. In the CData Sync app, you can use PostgresSQL as a source and destination both. We support modern

0
N
11 months ago
N
Nandini BaggaEmployee
posted in CData Sync
How to create Oracle Destination Connection in CData Sync - .NET Edition ?

PurposeSteps to create Oracle Destination Connection, which is the native Oracle Managed Data Access Client (Oracle.ManagedDataAccess.dll) in CData Sync .Net Edition.  Step 1: Download ODP.NET_Managed_ODAC.zip from 64-bit Oracle Data Access Components (ODAC) for Windows  Step 2: Copy the Oracle.ManagedDataAccess.dll from D:\Downloads\ODP.NET_Managed_ODAC122cR1.zip\odp.net\managed\common Step 3: Put it in www/bin folder Step 4: Add provider in web.config file<remove invariant="Oracle.ManagedDataAccess.Client" /><add name="ODP.NET, Managed Driver" invariant="Oracle.ManagedDataAccess.Client" description="Oracle Data Provider for .NET, Managed Driver" type="Oracle.ManagedDataAccess.Client.OracleClientFactory, Oracle.ManagedDataAccess, Version=4.122.1.0, Culture=neutral, PublicKeyToken=89b483f429c47342"/> Step 5: Check the Version, Culture and PublicKeyToken to match your DLL’s. Steps to check Version, Culture and PublicKeyToken: Open Visual Studio’s Developer Command Prompt

0
N
11 months ago
D
DougN NouriaInfluencer
asked in CData Sync
Uppercasing the destination table name in Snowflake

Hello, Snowflake seems to be case-sensitive with table names, sort of.  If the incoming table name from Sync gets created with lower or mixed case, it gets created that way in Snowflake, and you have to reference them with quotes later on when querying.  Like    select * from “my_table".  Whereas if it were created as MY_TABLE, you wouldn’t need to bother adding quotes.Some data sources (a SQL server database we have, Active Directory, etc.) have lower or mixed case tables as exposed in CData.Is there a way to uppercase these either on the source as the metadata is read, or, when created in the destination?thank you!

2
D
11 months ago
E
eneopEmployee
posted in CData Sync
Understanding Change Data Capture and Creating CDC job in CData Sync

Change Data Capture (CDC) is a crucial technique in modern data management that allows you to track and capture changes made to a database. It provides real-time insights into data modifications, enabling businesses to react swiftly to evolving trends and requirements. In this article, we'll explore what CDC is and how to implement it using CData Sync.Understanding Change Data Capture (CDC)CDC revolves around capturing data changes at the source using a log file, eliminating the need for constant querying. Instead, CData Sync reads the log file for events (Insert, Update, or Delete) and extracts these changes in near real time, ensuring accurate replication for future use.Supported CDC Sources:MySQL: Utilizes binary logs. Oracle: Leverages Oracle Flashback. PostgreSQL: Employs logical replication. SQL Server: Uses either change tracking or change data capture.Creating a CDC Job in CData SyncTo initiate a job in CData Sync, ensure you have pre-configured source and destination connectio

1
Ankit Singh
11 months ago
A
abdullah028Apprentice
asked in CData Sync
HTTP ERROR 404 Not Found java.util.zip.ZipException: invalid entry size (expected 5623126 but got 5623165 bytes) while installing CData synch using java -jar sync.jar on macos

INFO: Get app data directory form /Users/mac/Downloads/CDataSync /sync.properties file. nullAug 18, 2023 3:51:40 PM cdata.sync.SyncServer aINFO: The server log directory is /Users/mac/Downloads/CDataSync /logs/httpserverAug 18, 2023 3:51:40 PM b.d qINFO: Init Jetty log with default settings.Aug 18, 2023 3:51:40 PM b.d bINFO: Starting server from /Users/mac/Downloads/CDataSync /sync.properties configuration.Aug 18, 2023 3:51:40 PM d.a aINFO: The properties deserializes success.Aug 18, 2023 3:51:41 PM b.d cINFO: Enabling embedded Jsp support...Aug 18, 2023 3:51:41 PM b.d cINFO: Successfully enabled embedded Jsp support.2023-08-18 15:51:41.719:INFO :oejs.Server:main: jetty-10.0.14; built: 2023-02-22T23:12:32.272Z; git: 976721d0f3e903a243584d47870ad2f2c1bf9e55; jvm 17.0.8+9-LTS-2112023-08-18 15:51:49.684:WARN :oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@7d3a22a9{/,null,STOPPED}{/Users/mac/Downloads/CDataSync /webapp/sync.war}java.util.zip.ZipException: invalid

1
A
1 year ago
B
bkilariApprentice
asked in CData Sync
Cannot invoke "java.util.List.iterator()" because "" is null

Hi Community, new to CData Sync here.When am trying to extract a certain table from Workday through the inbuilt Workday source connection, an getting the error - [0] Cannot invoke "java.util.List.iterator()" because "" is null Could anyone help with the error please. Thanks in advance.  

1
Ethem Q
1 year ago
B
bvbert1969Apprentice
asked in CData Sync
rename file after receiving

Dear all, apologizes in advance for the probably silly question, but I am new at this and I could not find any examplea customer of mine is sending me a file always with the same filename, and I need to be able to save it and be sure that it will not be overwrittenfor example adding a timestamp would be perfect (but preserving the extension)EUC2313.xml → EUC2312_20230828101211.xmlEUC2313.xml → EUC2312_20230828101313.xmlhow could I do this ? I can not use filepath or filename

6
James B
1 year ago
D
Deb Cobb_PMMEmployee
posted in CData Sync
CData Sync Q3 2023 Release

Hi Everyone!The latest release of CData Sync has arrived! Take a closer look at what's new and how it can benefit you. Visit these resources to learn more:BlogWhat’s New in the Q3 Sync releaseKnowledge Base ArticlesEnterprise features Transformation librariesExperience what’s new yourself!Start a 30 day trial  Upgrade to the latest release And of course, let us know what you think!All the best,The CData Sync Team

0
D
1 year ago
R
RonMooApprentice
asked in CData Sync
Error: Entity 'WorkStepStatus' is not supported by the Bulk API.

Executing SF_MirrorAll results in an error as indicated below.  It occurs both with FullCopy or DeltaCopy.  I have performed various searches but haven’t found a solution.  --- Starting SF_Mirror for WorkStepStatus V8304 [SQLSTATE 01000]01:53:03: Parameters: SALESFORCE_Med_ARCH_FULL WorkStepStatus  Version: V8304 [SQLSTATE 01000]01:53:04: DBAmpAZ 22.0.8304.0 Copyright c 2021 CData Software, Inc. [SQLSTATE 01000]01:53:04: Obtaining Connection Props [SQLSTATE 01000]01:53:04: Obtaining User Information [SQLSTATE 01000]01:53:04: Using settings from Registry. [SQLSTATE 01000]01:53:05: Using FullCopy because the WorkStepStatus table does not exist. [SQLSTATE 01000]01:53:05: Server: WUIT-S-14261, Database: MedARCH_SF_FULL, Linked Server: SALESFORCE_Med_ARCH_FULL [SQLSTATE 01000]01:53:05: Using the Salesforce BulkAPI. [SQLSTATE 01000]01:53:06: Create WorkStepStatus_Previous20230828015305925 with new structure. [SQLSTATE 01000]01:53:06: Error: Entity 'WorkStepStatus' is not supported by the Bul

1
R
1 year ago
B
bdempseyApprentice
asked in CData DBAmp
DBAmp 2022 Fails to Update License if not installed on C: Drive (with workaround)

My DBAMP version is 22.0.8354.0I have DMAMP installed on a non OS Drive.   M:\Program Files\Cdata\CData DBAmpTrying to upgrade the license using the Config Utility.  Help → Licensing ->Update LicenseGetting following error.  The keys.reg file does not exist on my installation. Work Around Create the folder + File the error was complaining about. It can be an empty text file.“c:\CData\CData DBAmp\keys.reg”Re-Run the Update License command and seems to work fine now. Question Is this fixed in a later version of dbamp?  I am aware I dont have the most recent “patch”.

2
D
1 year ago
J
JTapioApprentice
asked in CData Sync
[META|Cache ] Cache failed: Updating Row is failed. ERROR: Invalid HTTP response!

Hi, I’ve been successfully using Sync to replicate data from Eleasticsearch index to Azure Data Lake. Now I have faced an issue one of index that is bigger than others (not much, but obviously enough bigger). In replication log there are following errors: [META|Cache ] Cache failed: Updating Row is failed. ERROR: Invalid HTTP response! Cannot upload PostData: System error: An existing connection was forcibly closed by the remote host., AffectedCount: 5200000 [SQL |Messag] Query Failed: [CACHE [uspa/spaiderloki/arkisto/2023] SELECT [_id], [Aika], [Atunnus], [Caller], [Callsite], [cdata_timestamp], [Duration], [Exception], [Filter], [guid], [ID], [kafka_partition], [Logger], [LogLevel], [Message], [Thread], [Version] FROM [Elasticsearch].[uspa-spaiderloki_arkisto_20233x]]. Error: Updating Row is failed. ERROR: Invalid HTTP response! Cannot upload PostData: System error: An existing connection was forcibly closed by the remote host., AffectedCount: 5200000 Updating Row is failed. ERROR: I

1
P
1 year ago
S
shreyApprentice
asked in CData DBAmp
Upgraded DBAmp and the SOQL queries don't work

We upgraded DBAMp version on our SQL Server.Followed the instructions and created a new Linked Server. Stored procedures are running fine, but when I execute a SOQL query; we get the following error An invalid schema or catalog was specified for the provider "MSOLEDBSQL" for linked server "SalesForce". Example SOQL queriesSelect * from SalesForce...user__cSelect * from SalesForce...Account

2
S
1 year ago
D
DevApprentice
asked in CData Sync
Twitter API Details

Can i get the Twitter Connector API details for the Cdata platform ? 

2
D
1 year ago
S
shreyApprentice
asked in CData DBAmp
DBamp Stored procedures failing after upgrade to V2022: Unable to retrieve settings from sys_connection_props.

We upgraded to DBAmp V22 version. Have upgraded the Stored Procedures.Regular SOQL queries work from the SQL server, but when try using SF_REPLICATE or SF_REFRESH we get the following error. --- Starting SF_Replicate for Account V830409:56:45: Parameters: SALESFORCE Account  Version: V830409:56:45: DBAmpAZ 22.0.8304.0 Copyright c 2021 CData Software, Inc.09:56:45: Using options specified in Options parameter: soap09:56:45: Obtaining Connection Props09:56:46: Error: Unable to retrieve settings from sys_connection_props.09:56:46: System.Data.SqlClient.SqlException (0x80131904): Cannot get the column information from OLE DB provider "DBAmp.DBAmp" for linked server "SALESFORCE".   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)   at System.Data.SqlClient.TdsParser.TryRun(

1
D
1 year ago
A
andrew.wilsonApprentice
asked in CData DBAmp
DBAmp v2022 Collation issues

Hi, We are migrating to new SQL 2022 servers.  I have installed v2022 of DBAmp on our new dev server, we use SF_Replication/SF_Refresh, changing to SF_Mirror is not yet on the table.  My default server collation is Latin1_General_CI_AS, and the DBAmp replicant also becomes this collation.  This is the exact same setup we run in production, with version 5.1.8.0 of dbamp.   Both environments, current prod and new dev have “Collation Compatible” and “Use Remote Collation” set to true.I have run a full SF_Replicate, and when i start to run the daily SF_Refresh i am getting the below error:--- Starting SF_Refresh for Contact V830416:12:31: Using Schema Error Action of subsetdelete16:12:37: Local table created at: 2023-05-17 16:29:0016:12:37: Using last run time of 2023-05-17 16:29:00Msg 468, Level 16, State 9, Line 1Cannot resolve the collation conflict between "SQL_Latin1_General_CP1_CI_AS" and "Latin1_General_CI_AS" in the equal to operation.--- Ending SF_Refresh. Operation FAILED.Msg 500

1
D
1 year ago
T
tlallyApprentice
asked in CData Sync
Need to replicate ContentDocumentLink from Salesforce

When I attempt to replicate ContentDocumentLink with CData Sync from Salesforce to SQL, I am getting this error. [0] [500] Could not execute the specified command: [sf:MALFORMED_QUERY] MALFORMED_QUERY: Implementation restriction: ContentDocumentLink requires a filter by a single Id on ContentDocumentId or LinkedEntityId using the equals operator or multiple Id's using the IN operator. The tool that I am looking to replace with CData Sync can do it.  Is there a workaround to get this object to replicate? Thanks

1
A
1 year ago
A
ashleypaceRSFApprentice
asked in CData Sync
CData Sync Email Connector

Does anyone here have any experience with the CData Sync email connector? I want to know what type of functionality it has - as in, can it read attached files and replicate those, etc?

1
E
1 year ago
F
FishXVApprentice
asked in CData Sync
NetSuite Connector - No Tasks Found

Hi all, Was wondering if anyone has any ideas regarding a NetSuite connector that will allow me to add the connection but displays no tasks when adding a new job.  The connector is using an OAuth token to auth against SuiteQL. It can connect sucessfully, although I can’t create any tasks as it says “No Tasks found. Please verify your connection has access to tables”. The permissions in NetSuite were set according to “https://cdn.cdata.com/help/DNH/cloud/default.htm#pg_permissions”. Any help would be appreciated!

2
F
1 year ago
A
AndreQApprentice
asked in CData DBAmp
Weekly SF_ReplicateAll failing

There are a host of errors, mostly “Error occurred creating primary key for table.” in tables AppDefinition, AuraDefinitionBundleInfo, AuraDefinitionInfo, FormulaFunction, FormulaFunctionAllowedType, FormulaFunctionAllowedType and other. but also “Using FullCopy because the FieldChangeSnapshot table does not exist.”threw a SOAP error: Error System.Web.Services.Protocols.SoapException: EXTERNAL_OBJECT_UNSUPPORTED_EXCEPTION: Enter a valid ParentID.    at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)   at DBAmpAZ.sforce.SforceService.query(String queryString)    at afo220l.Uh.w() Should I wipe out all the tables and start again? Please advise.RegardsAndre

1
Ankit Singh
1 year ago
B
bweatherfordApprentice
asked in CData Sync
CData Sync and MS Teams REST API

Has anyone ever used the REST API Source to get to MS Teams Auto Attendant and/or Call Queue data?Any help is appreciated.

4
Ankit Singh
1 year ago
Virtual Event: Data Virtualization, Reimagined Ma”arch 14, 2024 style=

Badge winners

Show all badges
Terms and ConditionsCookie settings