Understanding Log Files – CData Sync

About CData Sync log files

The CData Sync application keeps track of its operations through a comprehensive Job History table, documenting the outcomes of each executed task. Beyond just maintaining a job history, Sync records intricate details of job executions in a detailed format across multiple files. This process is augmented by the flexibility to configure logs at different levels of verbosity. The degree of verbosity chosen helps pinpoint connectivity issues and troubleshoot various challenges encountered during job execution. This multifaceted logging system not only ensures a detailed record of operations but also empowers users with the insights needed for effective error tracking and problem resolution within the transformation and replication process.

How to generate and find the log files

CData Sync is designed to produce log files that encapsulate a comprehensive record of the various processes it executes like Job runs, Transformations, Application requests, Access requests, and Audit configuration changes. Each of these categories of activities is logged to provide a thorough overview of the application's performance.

The “Logs” page serves as a centralized hub for accessing and managing these logs. This page is structured with individual tabs dedicated to each specific log type. This organizational approach allows users to navigate through the logs on different aspects of the application's functionality. Importantly, each log type is stored as a distinct table in the application database, ensuring a clear and organized repository of information. This systematic arrangement not only facilitates easy retrieval and analysis of logs but also contributes to maintaining the integrity and structure of the logged data within the application.

In the Logs page, the Job History tab consolidates logs from all job runs in a single location. You have the option to download individual log files by accessing the ellipsis menu on the right side of each job run, as depicted above. Furthermore, you can obtain logs for all historical job runs by utilizing the 'Export History' feature, as illustrated below.

Types of log files in CData Sync  

CData Sync generates various categories of log files depending on the tasks it carries out.

1. Job logs

Job logs are created when a job is run on Sync. A job run can produce three separate logfiles:

1. Source: This file logs any communication with the data source. Communications include connection issues, metadata retrieval, and errors related to raw data retrieval.

2. Destination: This file logs any communication with the data destination. Communications include connection issues, metadata retrieval, and errors related to bulk data loading.

3. Replicate Engine: This file logs operations that are managed by the engine. Operations include communication with the CDATA_SYNC_STATUS table and the generated queries that are sent to the source and destination.

2. Transformation History logs

Transformations involve processing raw data from your source, refining, reshaping, and organizing it to align with your specific requirements within your data pipeline. The Transformation History tab consists of logs that capture the operations performed at the destination.

3. Application logs

The Application tab records any application-level errors that occur during processing and any requests made to application resources. The error messages in the Application tab can help provide context to errors for specific transactions. The level of information in these logs is determined by verbosity levels (discussed in the later section).

4. Access logs

Access Logs record details about users who have utilized the sync application within your organization, particularly relevant when multiple users are involved. The Access tab records entries for requests directed at the provided endpoints.

5. Audit logs

The Audit tab stores each change made to the application configuration. This includes all actions like configuring connectors, updating profile settings, etc. It displays the following details:

• Date/Time: Timestamp for the configuration change.
• User: User who made the change.
• Method: The type of change that was made. The options are: GET, POST, PUT, MERGE, and PATCH
• Message: The text describing the change

You can query the Audit tab to perform audits of the application configuration.

NOTE: All the above-mentioned logs can be downloaded from the top-right corner using the ‘Download Logs’ feature on the Logs page.

Characteristics of logging in CData Sync

The following sections explain how to enable logging, view your job history, download job logs, and archive logs.

1. Enabling logging

Follow the given steps to enable logging:

1.  Click the job for which you want to enable logging. This action opens the Jobs/YourJobName page.

2.  Click the Overview tab. In the Settings category, click the edit icon (For additional help, click the documentation icon.)

3.  Enter a schema in the Destination Schema field.

4.  Select the verbosity level that you want from the Logfile Verbosity list. Setting this property determines the amount of information included in logs for a job. The available verbosity levels are None, Error, Info, Transfer, and Verbose (Discussed in the next section). Setting the verbosity level to Transfer or Verbose exposes the raw data that is transferred between the source applications and the destination database. Although connection properties (such as passwords) are masked, you should review logs at this level for sensitive information before you share them outside of your organization.

5.  Select Enable under Parallel Processing if you want to run multiple tasks within the job in parallel.

6.  Click Save to apply the settings.

After you save your logging configuration, the Settings category on the Overview tab displays the settings that you defined.

2. Log verbosity

The table below explains logged information based on their verbosity levels.

3. Application log verbosity levels

Sometimes the Sync application itself may throw some errors. These errors are registered and can be obtained from Application logs. On the Settings page of the CData Sync admin console, the Advanced tab provides options for configuring application logs. These logs are created in the folder specified in the Logs Folder field whenever the conditions for the selected value of the Application Log Level are met. These options are outlined below:

NOTE: Logging levels Debug, Trace, and All might log sensitive information, including file information and SSL certificates. Although sensitive details such as passwords are masked, you should review logs of this level for sensitive information before sharing them outside of your organization.

4. Viewing the job history

After each attempt of a job run, Sync adds an entry into the Job History table that contains the results of the run. Click the Job History tab to view the previous history of a job. Expanding a row in the table displays the results of each separate task that ran.

5. Downloading job logs

To download a job log:

1.  Navigate to the Job History table.

2.  Locate the run date of the log file that you want to download and review. Then, click the ellipsis that appears in the far-right column of that row. (Note: The ellipsis is not visible until you position your cursor over the column.) The ellipsis provides two options:

a.  Download Log File: This option opens a Downloads dialog box that enables you to log files that are associated with your job. Click the Open file for the specific log that you want to download and review.

b.  Run Details: This option opens a dialog box that displays two tabs:

• Details: This tab displays the following details: Run Date, Status, Runtime, and Rows Affected.
• Logs: This tab lists the logs that are available for your job. You can download individual logs here by clicking the download icon to the far right of the log name. You can also download all logs by clicking Download All Log Files.

To see a table of log files with a specific run status (All Status, SUCCESS, FAILED, or ERROR), click Show Filter. This action displays the All Status menu at the top of the logs table.

6. Archiving logs

Sync enables you to archive logs, either in the same instance of Sync that is running (local disk) or in an Amazon S3 bucket.

To archive your logs:

1.  Open Sync and navigate to the Clean-up Settings category on the Advanced tab. (Settings > Advanced).

2.  Click the edit icon to set the properties that are listed below. (The edit icon is next to the documentation icon at the top right of that category.)

• Type: Select Archive.
• Retention Period (Days): Set the number of days that you want the log files to be held in the original directory. When this number expires, Sync archives these files.
• Daily Clean-up Time: Specify the time that you want to run the cleanup job each day. If you want to run the cleanup job immediately, return to the Clean-up Settings category on the Advanced tab and click Run Clean-up.
• Archive Type: Select either LocalDisk or S3, depending on where you want the archived logs to be stored.
• If you select Local Disk, you need to specify where to store the files. In the Local Archive Folder field, specify the path on the server where you want to place the archived files.
• If you select S3, specify the following settings:

• S3 Bucket: Specify the name of your S3 bucket.
• S3 Access Key: Specify the access key for your S3 bucket.
• S3 Secret Key: Specify the secret key for your S3 bucket.
• S3 Region: Specify the region where Amazon stores your bucket.

After you specify these settings, click Test Connection to verify that your S3 connection works.

3. Click Save to save your settings and return to the Advanced tab.

Understanding the different types of CData Sync log files 

In this segment, we aim to comprehend the various essential components that constitute the diverse logs generated by CData Sync.

1. Job logs

As outlined in the 'Types of log files in CData Sync' section above, a Job run generates three distinct versions of log files: Source, Destination, and Replicate Engine. In this section, we delve into each of them to grasp their unique characteristics.

a.  As part of the demonstration, a Job is established in CData Sync, with Salesforce serving as the source and MySQL as the destination database. In the Settings subsection within the Overview tab, we configured the Logfile Verbosity to 'Transfer' (as shown below).

b.  After making the necessary configurations, a task is created to replicate the table ‘Customer’ from Salesforce into MySQL and the job is run. Once the job is complete, we navigate to the “Job History” tab and select the completed job.

c.  We select the Logs tab and download all three logs generated (Replicate, Source, and Destination).

NOTE: The verbosity of the logs is set to ‘Transfer’ which is ‘Verbosity=3’.

I. Source Log: 

This file records all interactions with the source connector, encompassing connection problems, metadata retrieval, and errors associated with the retrieval of raw data.

II. Destination Log: 

This file records all interactions with the destination connector, encompassing connection challenges, metadata retrieval, and errors associated with bulk data loading.

III. Replicate Log: 

This file documents tasks managed by the replication engine, such as interactions with the CDATA_SYNC_STATUS table and the generated queries transmitted to both the source and destination.

2. Application logs

The Application logs document errors at the application level that may arise during processing, as well as any requests made to resources within the application.

3. Audit logs

The Audit log preserves every modification made to the application configuration, including actions such as configuring connectors, updating profile settings, and more.

Server Responses

Typically, the status of an HTTP server response can be summarized by its HTTP status code. The following table lists HTTP status codes that Sync can send, along with when you can expect these status codes.

Log File Checklist

A log file contains a wealth of valuable information. Here are some initial recommendations for checking if the solution is not immediately evident:

1.  Inspect the log for any error messages in the HTTP responses.

2.  Check and test the connections for each connector.

3.  Verify the validity of the transformation queries provided.

4.  Check the driver's version; if it's an older version, contact our customer support.

Try CData Sync Today

To effortlessly integrate live and on-demand data with incredible speed and accuracy. Sign up for a 30-day free trial of CData Sync.