Solved

Max File processed by a connector

  • 12 March 2024
  • 4 replies
  • 36 views
P
Userlevel 2
Badge

Hi Folks,

 

Please help to advice what will be the max number of files / max Workers a connector will be utilized in a time, since many time i could see files getting stagged up for few hours not getting processed. Then manually sending the files also take time to process, but the server CPU utilization is in optimal level. 

This causes the business team unable to receive files in time and causing issues, 

 

above is the example from today 9.27 AM MYT files got queued up and not processed for nearly 1.5 hrs. Now i am manually processing them. This is simple Copy Connector. 

icon

Best answer by lohith 19 March 2024, 07:02

View original

4 replies

lohith
Userlevel 1

Hi Pradeep,

Adjusting Max Workers per Connector and Max Files per Connector settings can optimize Arc performance, especially for connectors with varying resource needs. Increasing max workers for bottleneck connectors assigns more resources for processing, but if done for many connectors, it can deplete the Worker Pool, causing delays as the application waits for thread recycling. 

Raising Max Files per Connector for a specific connector reduces file processing delays, particularly useful for high-volume connectors. However, it may prolong thread assignment, potentially leading to a thread shortage.

Optimizing performance depends on the environment and use case. Connectors handling numerous files should have more workers and process more files before recycling, while others should have fewer workers and process fewer files to avoid thread pool exhaustion.

More details at: https://cdn.cdata.com/help/AZJ/mft/Architecture.html#maximizing-performance

 

Lohith
P
Userlevel 2
Badge

Hi Lohith,

value of Max workers per connector is “20” and what is the max numbers of Max files in a connector to be processed at time. 

lohith
Userlevel 1

Hi Pradeep,

Setting higher values for Max Workers and Max Files parameters can indeed help allocate more resources for processing. It's worth noting that values set at the individual connector level override those set at the global Profile level. For example, configuring the busy connector with 10 Max Workers and 100 Max Files would allow it to process a larger number of files simultaneously, thereby reducing wait times for files in its Input folder.

Conversely, for low-priority or seldom-used connectors, reducing their Max Workers to 1 and their Max Files to 5 would free up resources for more critical connectors.

However, optimizing performance is a complex process without a one-size-fits-all solution. Variables such as traffic volume, file size, processing types, number of connectors, and available computing resources all influence performance. It's crucial to experiment with different settings and monitor performance to find the optimal configuration for your system.

Lohith
P
Userlevel 2
Badge

Thanks Lohith, by filling more in details . 

Reply


Virtual Event: Data Virtualization, Reimagined Ma”arch 14, 2024 style=
Terms and ConditionsCookie settings