Data engines

Data Engines are the workers that pick up and process jobs from the queue. When multiple jobs are queued simultaneously, Data Engines use priority to determine which job to process first. Navigate to More → System → Queue to monitor active jobs and processing times.

Job priority

Priority is set per Scheduler job and has three levels: High, Medium, and Low. If two jobs are queued at the same time, the one with the higher priority is picked up first. A lower-priority job that is already running will complete before a higher-priority job is picked up. Disabled Schedulers are ignored by Data Engines.

Common issues and solutions

Scheduler runs overlap This happens when a process takes longer to complete than the interval between runs. For example, an Incoming configuration that takes 10 minutes to complete but is scheduled every 5 minutes will cause a backlog in the queue. Check the Total throughput time column on the Queue page to see how long a process actually takes, then adjust the Scheduler interval to be longer than the process duration. Process runs out of memory The default memory limit per process is 384 MB. Common causes and solutions:

Cause	Solution
Large files from a filesystem	Configure the deserializer to read the file incrementally rather than loading it all at once.
Large API responses	If the endpoint supports pagination, fetch one page per run using the progress tracker storage pattern instead of fetching all records in a single call.
Real-time processing	Real-time Routes run both Incoming and Outgoing in a single process, doubling memory usage. Disable real-time processing and create a separate Scheduler for the Route.
Too many tasks are exported per run	Reduce the Number of tasks to export settings on the Route Scheduler until the process completes without failing.

If none of these solutions resolves the issue, a memory limit upgrade can be requested via your Customer Success Manager.

Getting started

Integrations

Connections

Data transformation

Placeholders and patterns

Data processing

Storages

Monitoring and logs

Administration and settings

Job priority

Common issues and solutions

​Job priority

​Common issues and solutions

Job priority

Common issues and solutions