使用轻易云进行ETL转换并写入金蝶云的技术案例

  • 轻易云集成顾问-钟家寿
### R入库单(退货)-其他入库单系统对接案例:易仓数据集成金蝶云星空 在本文中,我们将深入探讨如何通过轻易云数据集成平台,将易仓的R入库单数据无缝对接至金蝶云星空,实现高效、透明且稳定的数据流转。此方案主要聚焦于处理API接口调用、数据格式转换以及异常处理等关键技术问题。 首先,我们需要确保从易仓获取的R入库单不会出现漏单现象。这是通过定时可靠地抓取`getPutAwayList`接口的数据实现的,使用轻易云平台内置的调度功能,可以精准控制每个时间周期内的数据拉取,从而保证了数据完整性和及时性。在面对大量数据写入到金蝶云星空时,利用其提供的批量保存接口`batchSave`, 可显著提高传输效率,同时减轻网络压力。 其次,为了保证系统能够应对海量数据交互过程中的分页与限流问题,在设计这个集成方案时,通过合理设置请求频率及分批次处理策略,有效提高了API调用性能并减少了服务器负载。此外,还需特别注意两者之间的数据格式差异,尤其是在自定义字段映射方面,需要针对不同业务场景进行定制化处理,以适配各自系统要求。 在整个流程中,不可避免会遇到各种异常情况。因此,重试机制和错误日志记录至关重要。借助轻易云强大的实时监控功能,可随时捕获并记录异常事件,并根据预设规则自动触发重试操作,大大提升系统稳定性与容错能力。这不仅简化了运维工作,也为后续排查潜在问题提供有力支持。 ![如何对接钉钉API接口](https://pic.qeasy.cloud/D32.png~tplv-syqr462i7n-qeasy.image) ### 调用易仓接口getPutAwayList获取并加工数据的技术实现 在轻易云数据集成平台中,调用源系统接口是数据处理生命周期的第一步。本文将详细介绍如何通过调用易仓接口`getPutAwayList`获取并加工数据,以实现R入库单(退货)-其他入库单的集成方案。 #### 接口配置与请求参数 首先,我们需要配置接口和请求参数。根据提供的元数据配置,`getPutAwayList`接口使用POST方法进行调用,以下是请求参数的详细配置: ```json { "api": "getPutAwayList", "method": "POST", "number": "{random}", "id": "{random}", "idCheck": true, "beatFlat": ["dataList"], "request": [ {"field":"page","label":"页码","type":"Int","describe":"页码"}, {"field":"pageSize","label":"每页显示数量","type":"Int","describe":"每页显示数量(默认:20,不能超过50)"}, {"field":"pdStatus","label":"上架单状态","type":"Int","describe":"上架单状态(取值:0.未上架 1.已完成)","value":"1"}, {"field":"pdType","label":"上架单类型","type":"Int","describe":"上架单类型(取值:0.ASN 1.QC 2.BOX )","value":"1"}, {"field":"warehouseCode","label":"仓库代码","type":"String","describe":"仓库代码"}, {"field":"receivingType","label":"入库类型","type":"Int","describe":"入库类型(取值:0.标准 1.订单退货 2.指定产品退件 3.中转 4.采购 5.其它 6.借用归还入库 7.初始化库存 8.不良品入库 9.盘盈 10.调拨 11.换货 12.同步服务商库存 13.出库单入库 14.同步添加库存 15.赠品入库)", "value": "1"}, {"field": "productSku", "label": "产品SKU", "type": "String", "describe": "产品SKU"}, {"field": "pdLocation", "label": "临时库位", "type": "String", "describe": "临时库位"}, {"field": "pdPutawayCar", "label": "上架小车", "type": "String", "describe": "上架小车"}, {"field": "orderType", "label": "单据类型", "type": "Int", "describe": "单据类型(取值:1.质检单 2.上架单 3.入库单 4.产品代码 5.采购单 6.单据模糊查询 ,默认:取值1,即质检单)", "value": "3"}, {"field": "orderCode", "label": "单据编号", "type": "String", "describe": "单据编号" }, {"field": "supplierCode", "label": "供应商代码", "type": "String", "describe": "供应商代码" }, {"field": "dateType", "label": "日期类型", "type": "Int", "describe": "日期类型(取值:1. 创建时间 2. 上架时间 3. 生产日期 4. 质检时间 5. 预检时间,默认取值1,即创建时间)", "value": " 2 " }, {"field": " startDate ", " label ": " 开始日期, ", " type ": " String ", " describe ": " 开始日期, 格式: 2020-12-17 ", "value": "_function date_format(CURRENT_DATE()-5,'%Y-%m-%d')" }, {"field": " endDate ", " label ": " 截止日期, ", "type": " String ", "description: 截止日期, 格式: 2020-12-17 ,value: _function date_format(CURRENT_DATE(),'%Y-%m-%d')"} , {"field: operatorCode,label:上架员代码,type:String,description:上架员代码(即登录系统的账号)"} ] } ``` #### 数据请求与清洗 在实际操作中,首先要确保请求参数正确无误,并根据业务需求设置具体参数值。例如: ```json { "pageSize":50, "pdStatus":1, "pdType":1, "warehouseCode":"", "receivingType":"", ... } ``` 发送POST请求后,平台会返回包含`dataList`字段的数据。接下来,我们需要对返回的数据进行清洗和处理。 #### 数据转换与写入 在数据清洗完成后,需要对数据进行转换,以符合目标系统的数据格式要求。以下是一个简单的数据转换示例: ```python def transform_data(data): transformed_data = [] for item in data['dataList']: transformed_item = { 'order_id': item['orderCode'], 'sku': item['productSku'], 'quantity': item['quantity'], 'warehouse': item['warehouseCode'], ... } transformed_data.append(transformed_item) return transformed_data # 假设我们已经获得了原始数据 raw_data cleaned_data = transform_data(raw_data) ``` 最后,将转换后的数据写入目标系统。轻易云平台支持多种异构系统的无缝对接,可以通过API、数据库连接等方式将数据写入目标系统。 #### 实时监控与日志记录 为了确保数据处理过程的透明度和可靠性,轻易云平台提供了实时监控和日志记录功能。可以实时查看数据流动情况,并记录每一步操作日志,以便于问题排查和性能优化。 通过以上步骤,我们完成了从调用源系统接口获取数据到清洗、转换并写入目标系统的全过程。这不仅提升了业务效率,还确保了数据处理的准确性和透明度。 ![如何对接用友BIP接口](https://pic.qeasy.cloud/S29.png~tplv-syqr462i7n-qeasy.image) ### 使用轻易云数据集成平台进行ETL转换并写入金蝶云星空API接口的技术案例 在数据集成生命周期的第二步,我们需要将已经集成的源平台数据进行ETL转换,使其符合目标平台金蝶云星空API接口的格式,最终将数据写入目标平台。本文将详细探讨如何利用元数据配置完成这一过程。 #### 数据请求与清洗 在开始ETL转换之前,我们首先需要确保从源平台获取的数据已经经过清洗和初步处理。这一步骤通常包括数据的去重、格式化和基本验证。假设我们已经完成了这些前期工作,现在进入数据转换与写入阶段。 #### 元数据配置解析 元数据配置是ETL转换的重要部分,它定义了如何将源数据映射到目标平台所需的数据结构中。以下是我们需要使用的元数据配置: ```json { "api": "batchSave", "method": "POST", "idCheck": true, "operation": { "rowsKey": "array", "rows": 1, "method": "batchArraySave" }, "request": [ {"field":"FBillNo","label":"单据编号","type":"string","describe":"单据编号","value":"{dataList_receivingCode}_{dataList_productBarcode}"}, {"field":"FBillTypeID","label":"单据类型","type":"string","describe":"单据类型","parser":{"name":"ConvertObjectParser","params":"FNumber"},"value":"QTRKD02_SYS"}, {"field":"FStockOrgId","label":"库存组织","type":"string","describe":"组织","parser":{"name":"ConvertObjectParser","params":"FNumber"},"value":"{dataList_warehouseCode}","mapping":"63688a45a23a2e0fa5271b19","mappingDirection":"positive"}, {"field":"FStockDirect","label":"库存方向","type":"string","describe":"下拉列表"}, {"field":"FDate","label":"日期","type":"string","describe":"日期","value":"{dataList_pdPutawayTime}"}, {"field":"FDEPTID","label":"部门","type":"string","describe":"基础资料","parser":{"name":"ConvertObjectParser","params":"FNumber"},"value":"BM000013"}, {"field":"FOwnerTypeIdHead","label":"货主类型","type":"string","describe":"多类别基础资料列表","value":"BD_OwnerOrg"}, {"field":"FOwnerIdHead","label":"货主","type":"","describe":"","parser":{"name":"","params":""},"value":"","mapping":"","mappingDirection":""}, {"field":"","label":"","type":"","describe":"","parser":{"name":"","params":""},"value":"","mapping":"","mappingDirection":""}, {"field":"","label":"","type":"","describe":"","parser":{"name":"","params":""},"value":"","mapping":"","mappingDirection":""}, {"field":"","label":"","type":"","describe":"","parser":{"name":"","params":""},"value":"","mapping":"","mappingDirection":""}, {"field","","","","","","","",""} ], "otherRequest":[ {"field" : "", "label" : "", "type" : "", "describe" : "", "value" : ""}, {"field" : "", "label" : "", "type" : "", "describe" : "", "value" : ""}, {"field" : "", "label" : "", "type" : "", "describe" : "", "value" : ""} ] } ``` #### 数据字段映射与转换 根据上述元数据配置,我们需要将源数据字段映射到金蝶云星空API所需的字段。以下是几个关键字段的映射和转换过程: 1. **单据编号(FBillNo)**: - 源字段:`dataList_receivingCode` 和 `dataList_productBarcode` - 转换规则:拼接这两个字段作为单据编号 - 配置示例:`{"field": "FBillNo", "value": "{dataList_receivingCode}_{dataList_productBarcode}"}` 2. **单据类型(FBillTypeID)**: - 固定值:`QTRKD02_SYS` - 配置示例:`{"field": "FBillTypeID", "value": "QTRKD02_SYS", "parser": {“name”: “ConvertObjectParser”, “params”: “FNumber”}}` 3. **库存组织(FStockOrgId)**: - 源字段:`dataList_warehouseCode` - 转换规则:使用 `ConvertObjectParser` 将仓库代码转换为目标系统识别的编码 - 配置示例:`{"field": “FStockOrgId”, “value”: “{dataList_warehouseCode}”, “parser”: {“name”: “ConvertObjectParser”, “params”: “FNumber”}, “mapping”: “63688a45a23a2e0fa5271b19”, “mappingDirection”: “positive”}` 4. **日期(FDate)**: - 源字段:`dataList_pdPutawayTime` - 配置示例:`{"field": “FDate”, “value”: “{dataList_pdPutawayTime}”}` 5. **明细信息(FEntity)**: - 包含多个子字段,如物料编码、收货仓库、实收数量等 - 每个子字段都需要进行相应的映射和转换 - 配置示例: ```json { “field”: “FEntity”, “children”: [ {“field”: “FMATERIALID”, “value”: "_findCollection find FNumber from 32df639a-9c45-3823-8a92-1e2ceb30649e where FOldNumber={dataList_productBarcode}", “parser”: {“name”: “ConvertObjectParser”, “params”: “”}}, {“field”: “FSTOCKID”, “value”: "{dataList_warehouseCode}", “parser”: {“name”: ConvertObjectParser, params: FNumber}}, {“field”: FQty, value: "{dataList_pdQuantity}"} ] } ``` #### 调用API接口 在完成所有字段的映射和转换后,我们可以使用HTTP POST方法调用金蝶云星空的 `batchSave` API 接口,将处理后的数据发送至目标平台。 ```python import requests import json url = 'https://api.kingdee.com/batchSave' headers = {'Content-Type': 'application/json'} payload = { # 根据元数据配置生成的请求体 } response = requests.post(url, headers=headers, data=json.dumps(payload)) if response.status_code == 200: print("Data successfully written to Kingdee Cloud.") else: print("Failed to write data:", response.text) ``` 通过上述步骤,我们成功地将源平台的数据经过ETL转换后写入到了金蝶云星空API接口中。在实际操作中,需要根据具体业务需求调整元数据配置,以确保所有必要的数据都能正确映射和传输。 ![如何对接用友BIP接口](https://pic.qeasy.cloud/T1.png~tplv-syqr462i7n-qeasy.image)