Skip to main content
Solved

Sync Kafka source and max rows

  • April 8, 2025
  • 7 replies
  • 126 views
J
Forum|alt.badge.img

Hi,
I have Sync version 24.3.9120.0 and I’m using Apache Kafka as source. It’s working fine, except little irritation. When I run task it will replicate only first 150 000 rows and stops after that. Next time I run the task, it replicates 179 000 rows and so on. Row count will increase. Seem to be random

I can run job multiple times and it will continue where it was stopped earlier. Is there way to change this behavior, so that all rows will be replicated at once?

Best answer by Dibyendu

Hi ​@JTapio, just to clarify, whenever you perform full replication in CData Sync, the destination data is dropped, and a fresh set of data is fetched from the source.

Since the issue is persistent, I suggest that you contact our support team to help resolve your query.. CData Support: https://www.cdata.com/support/submit.aspx

    This topic has been closed for replies.

    7 replies

    D
    Forum|alt.badge.img+1
    • Dibyendu
    • Employee
    • April 9, 2025

    Hi ​@JTapio, could you please share the logs so we can investigate this issue further and assist? To do this, please go into the Job settings, set the Logfile Verbosity to "Verbose", run the job again, and share the resulting logs with us.

      J
      Forum|alt.badge.img
      • JTapio
      • Author
      • Collaborator
      • April 9, 2025

      Here is one example. Job finished successfully, but didn’t replicate all rows.

        D
        Forum|alt.badge.img+1
        • Dibyendu
        • Employee
        • April 9, 2025

        Hi ​@JTapio, In the source log file, we can see only 33,000 rows being fetched (screenshot attached). However, all these fetched rows are replicating successfully, so from the replication standpoint, it appears to be working correctly. The most probable cause I can identify relates to the source logs: the page size is set to 1000, and I suspect this size might be insufficient to fetch all the data.

        Additionally, please try using the incremental replication feature. This might work, as this issue shouldn't typically occur during full replication.

          J
          Forum|alt.badge.img
          • JTapio
          • Author
          • Collaborator
          • April 9, 2025

          I don’t think I can configure incremental replication, because source is Kafka. Kafka consumer offset value keeps track of incremental replication by default (Records can be consumed only once)

            J
            Forum|alt.badge.img
            • JTapio
            • Author
            • Collaborator
            • April 9, 2025

            Source Information edit button is grayed-out

             

              D
              Forum|alt.badge.img+1
              • Dibyendu
              • Employee
              • Answer
              • April 9, 2025

              Hi ​@JTapio, just to clarify, whenever you perform full replication in CData Sync, the destination data is dropped, and a fresh set of data is fetched from the source.

              Since the issue is persistent, I suggest that you contact our support team to help resolve your query.. CData Support: https://www.cdata.com/support/submit.aspx

                J
                Forum|alt.badge.img
                • JTapio
                • Author
                • Collaborator
                • April 10, 2025

                ok, I’ll do it.

                If I count rows from the source and from the destination, values are very close, so I don’t think that all the data is fetched every time replication occurs

                  State of AI Data Connectivity Report: 2026 Outlook
                  Terms and ConditionsCookie settingsAccessibility statement