Warehouse Master Data Query Sync in Practice: Building a Robust Pipeline from Source ERP to the Integration Platform

What This Strategy Solves

A retail client was running multiple sales channels, and warehouse master data originally lived only in the source ERP. After the finance and inventory accounting system was rolled out, the two sides had to stay consistent. Warehouses are few but change over time—new sites open, sites close, codes get adjusted—and manual syncing drifts within weeks. We built a single QUERY-style strategy that paginates warehouse records from the source ERP into the Qeasy data integration platform, where a downstream strategy then fans them out. In effect, the warehouse master data becomes a near-real-time baseline owned by the platform.

Data Flow and Field Mapping

The overall shape of the pipeline is: source system → middleware (Qeasy) → target system. This strategy covers only the first half: pulling warehouse records from the source ERP into the platform.

Key field mapping:

The source API is the POST method warehouse_query. Pagination starts from 0, and the maximum page size is 100. The middleware node uses an EXECUTE effect labeled "write no-op," whose purpose is not to call any third-party API but to persist the current page into the platform's staging table so that downstream strategies can perform idempotent writes by warehouse_id.

How to Configure It on Qeasy

Source-side configuration essentials:

• API selection: warehouse_query, POST method, effect = QUERY.
• Pagination params: bind page_no to {{PAGINATION_START_PAGE}} and page_size to {{PAGINATION_PAGE_SIZE}}. A value between 50 and 100 is recommended.
• Idempotency: enable idCheck=true and use warehouse_id as the unique key to prevent duplicate staging rows.
• Response modeling: enable both autoFillResponse=true and buildModel=true. The platform auto-models fields such as warehouse_id, warehouse_no, and name, sparing engineers from manual table creation.
• Incremental + full dual-track: a common pattern on customer sites is to prioritize modified_time-based incremental sync, with a monthly full reconciliation job at night. Since the source payload here does not expose a timestamp field, the practical pattern is full overwrite combined with an incremental flag bit.

Target-side configuration essentials:

• The effect is EXECUTE, but the API is set to "write no-op," with both number and id set to "0," meaning this node only receives data and does not dispatch.
• The advantage of this "dummy node" approach is that the source strategy and the downstream write strategy are decoupled. If you later need to switch target systems, only the target node changes; the source stays untouched.

For code mappings, customers typically centralize enumerations—such as warehouse type or status codes—in Qeasy's code-mapping module rather than scattering them into individual strategies, so a single edit propagates globally.

Implementation Steps

We recommend the following sequence:

1. Incremental baseline alignment: run an initial full load to seed the platform and mark sync_flag=full; then switch to incremental mode and pull only new or changed warehouses.
2. Full reload trigger: schedule a manual full reload at 2 a.m. on the 1st of each month as a reconciliation baseline. The source crontab can be set to 3 2 * * * (02:03) for the source query.
3. Downstream dispatch: once data lands in the staging table, a subsequent SYNC strategy pushes it to the finance/inventory system by warehouse_id. This is outside the scope of this strategy but must be configured first, or you risk "staging table piling up while downstream is empty."
4. Schedule frequency: since warehouse changes are infrequent, running the main pipeline once per day is sufficient. If business users complain about lag on new warehouses, add a lightweight verification job every six hours during business hours.

Pitfalls from the Field

1. Pagination starts at 0, not 1: the source explicitly states "defaults to page 0 if no value is passed," but many pagination helpers default to 1, causing the first page to be fetched twice. The safe approach is to hard-code start=0 in Qeasy's pagination parameter.
2. idCheck left disabled: in early runs we skipped idempotency for speed and ended up with three rows of the same warehouse_id in the staging table, breaking downstream writes. Even with small datasets, idempotency is non-negotiable.
3. Warehouse code renames: the source ERP allows editing warehouse_no while keeping warehouse_id stable. If the business side truly changes a code, any downstream that joins on warehouse_no will produce "orphan records." Always join downstream on warehouse_id; treat warehouse_no as display only.
4. "Write no-op" mistaken for a bug: a teammate once saw effect=EXECUTE with no actual API call and assumed the config was broken. In fact, this is the explicit "dummy node" pattern—a way to tell the platform, "this stage only receives data."
5. Cron time collisions: with the source scheduled at 02:03 and downstream dispatch also in the 2 a.m. window, jobs can collide. Keep the source in the 2 a.m. window and downstream in the 3 a.m. window, leaving a one-hour buffer for staging to settle.

When to Use It and When Not To

Use it when: warehouse master data must stay consistent across multiple systems, the source ERP exposes a paginated query API (as in this case), and the warehouse count ranges from dozens to a few thousand with low change frequency.

Do not use it when: warehouses change so frequently that second-level sync is required (switch to change-notification plus a message queue), or when the source ERP exposes no stable query interface and the only option is file export and re-import.