We are looking at a lambda approach with small incremental replication, followed up with a weekly full reload to catch late-arriving data and physical/hard deletes.
The challenge is that the full reload data volumes are too large for the server hosting our CData Sync. Even with the JAVA heap adjustments.
I was wondering if we could do the reload/replication in a looping process.
The source is Snowflake the target is Azure ADLS2 parquet files.
You can certainly configure CData Sync to work with large volumes of data by using replication intervals to ensure that the data is broken up into smaller batches. You can read more on how this is configured in the documentation here.
As for the weekly reload that you mentioned: you could configure a job to run separately from your other jobs and schedule the job to run on a weekly basis. You can find more information on how to schedule your jobs to run like this in the documentation here.
This website stores cookies on your computer. These cookies are used to collect information about how you interact with our website and allow us to remember you. We use this information in order to improve and customize your browsing experience and for analytics and metrics about our visitors both on this website and other media.
Review section III of our Privacy Policy for more information on cookies.
To opt out of “sale” or “sharing” of non-cookie data, visit our Privacy Rights Request Form.