使用轻易云平台实现数据ETL并写入金蝶云星辰V2

  • 轻易云集成顾问-李国敏
### 汤臣倍健营销云数据集成到金蝶云星辰V2技术案例 在此次系统对接项目中,我们将重点分享如何将汤臣倍健营销云的数据高效可靠地集成到金蝶云星辰V2。为了确保订单及出库单数据的无缝传输,我们使用了新版订单-出库单-厦门益天阁方案,以下是具体实现的一些关键技术细节和实践经验。 首先,为保证从汤臣倍健营销云获取完整且准确的数据,我们调用了其开放API接口`/api/openapi/v1/erp/order/honour/agreement/header`。我们采取定时抓取机制,实时监控并记录每次调用的状态日志,以确保数据不漏单。在处理大量数据时,通过优化分页处理以及应对接口限流策略,有效避免了因请求过多导致的服务性能问题。同时,将获取的数据与金蝶云星辰V2所需格式进行转换,以适配不同系统间的数据差异。 另一方面,在写入阶段,我们利用金蝶云星辰V2提供的API `/jdy/v2/scm/sal_out_bound`批量插入数据。这不仅提升了数据传输效率,也通过异常处理和错误重试机制保障了整个集成过程的稳定性。此外,针对特定业务需求,对部分关键字段进行了定制化映射,从而实现更加精准有效的数据同步。 下一步,让我们详细探讨该方案各环节中的重要技术点及具体实施步骤,着重展示实际操作中的解决方法和最佳实践。 ![数据集成平台API接口配置](https://pic.qeasy.cloud/D38.png~tplv-syqr462i7n-qeasy.image) ### 调用汤臣倍健营销云接口获取并加工数据的技术案例 在数据集成生命周期的第一步中,调用源系统接口并获取数据是至关重要的环节。本文将深入探讨如何通过轻易云数据集成平台调用汤臣倍健营销云的接口 `/api/openapi/v1/erp/order/honour/agreement/header`,并对获取的数据进行加工处理。 #### 接口调用配置 首先,我们需要配置元数据,以便正确调用汤臣倍健营销云的API接口。以下是该接口的元数据配置: ```json { "api": "/api/openapi/v1/erp/order/honour/agreement/header", "effect": "QUERY", "method": "POST", "number": "no", "id": "id", "idCheck": true, "request": [ {"field":"orgId","label":"组织ID","type":"string","describe":"111","value":"d85a4f7230ae4ed3a12c211a80cf8f3c"}, {"field":"page","label":"页码","type":"string","describe":"111","value":"1"}, {"field":"id","label":"订单id","type":"string","describe":"111"}, {"field":"applyerId","label":"要货方id","type":"string","describe":"111"}, {"field":"supplierId","label":"供货方id","type":"string","describe":"111"}, {"field":"no","label":"订单号","type":"string","describe":"111"}, {"field":"distributionType","label":"分销类型","type":"string","describe":"111"}, {"field":"distributorId","label":"分销商id","type":"string","describe":"111"}, {"field":"orderStatus","label":"订单状态","type":"string","describe":"111", "value": "WAIT_DELIVERY,PART_DELIVERY,ALL_DELIVERY,WAIT_OUT_STORAGE,PART_OUT_STORAGE,ALL_OUT_STORAGE,PART_CONFIRM,CONFIRM,AUDIT_SUCCESS,FINISH"}, {"field":"createDt","label":"创建时间","type":"string","describe":"111"}, {"field":"orderTypeCode","label":"订单类型,如普通订单、直运销售","type":"string","describe":"111"}, {"field": "isDeliveryFreezed", "label": "是否暂停发货", "type": "string", "describe": "111"}, {"field": "nature", "label": "单据类型1订单、2退货", "type": "string", "describe": "111", "value": "1"}, {"field": "relatedApplyerId", "label": "关联交易经销商id", "type": "string", "describe": "111"}, {"field": "saleDistribution", "label": "销售渠道", "type": "string", "describe": "111"}, {"field": "disApplyerId", "label": ![系统集成平台API接口配置](https://pic.qeasy.cloud/S2.png~tplv-syqr462i7n-qeasy.image) ### 使用轻易云数据集成平台将源数据转换并写入金蝶云星辰V2API接口 在数据集成的生命周期中,ETL(提取、转换、加载)过程是至关重要的一环。本文将深入探讨如何使用轻易云数据集成平台,将已经集成的源平台数据进行ETL转换,并最终通过金蝶云星辰V2API接口写入目标平台。 #### 配置API接口元数据 在配置过程中,我们需要根据提供的元数据配置,确保每个字段都正确映射到目标平台所需的格式。以下是具体的元数据配置: ```json { "api": "/jdy/v2/scm/sal_out_bound", "effect": "EXECUTE", "method": "POST", "number": "id", "id": "id", "name": "id", "idCheck": true, "request": [ {"field":"bill_source","label":"单据来源,固定传入ISV","type":"string","describe":"单据来源,固定传入APP","value":"ISV"}, {"field":"bill_date","label":"出库日期,格式:2019-01-01","type":"datetime","describe":"出库日期,格式:2019-01-01","value":"{{financeApproveDt|date}}"}, {"field":"customer_id","label":"客户id","type":"string","describe":"客户信息","value":"_findCollection find id from 617d3b33-162f-3e9f-be59-14e1953bc4a0 where number={clientAppNo}"}, {"field":"remark","label":"单据备注","type":"string","value":"{remark}"}, {"field":"contact_linkman","label":"联系信息-联系人","type":"string","value":"{contacts}"}, {"field":"contact_phone","label":"联系信息-联系方式","type":"string","value":"{phone}"}, {"field":"contact_address","label":"联系信息-详细地址","type":"string","value":"{shippingAddress}"}, {"field":"custom_field","label":"自定义字段","type":"object", "children":[{"field": "custom_field__1__2xietiob41d0jia2", "label": "营销云单号", "type": "string", "value": "{no}"}]}, { "field": "material_entity", "label": "商品分录", "type": "array", "describe": "商品分录", "value": "details", "children":[ {"field": "material_number", "label": "商品", "type": "string", "describe": "商品", "value": "{{details.easNo}}"}, {"field": "stock_id", "label": "仓库", "type": "string", "describe": "仓库", "value": "1277527712973493248"}, {"field": "qty", "label": "数量", "type": "string", "describe": "数量", "value": "{{details.quantity}}"}, {"field": "unit_id", "label": "单位id", "type": "string", "describe": "单位(只能填商品关联的单位,来源于商品单位查询接口或商品详情中的baseunitid_id、fixunit1_id等)", "value": "_findCollection find base_unit_id from 72029be6-c558-39a7-8084-4b75796fcba1 where number={{details.easNo}}"}, {"field": ... } ] } ] } ``` #### 数据请求与清洗 首先,我们从源系统中提取原始数据,并对其进行必要的清洗和预处理。此过程包括去除冗余数据、修正错误记录以及标准化数据格式,以确保后续转换过程的顺利进行。 例如,对于`bill_date`字段,我们需要将日期格式标准化为`YYYY-MM-DD`。可以使用以下代码进行处理: ```python import datetime def standardize_date(date_str): return datetime.datetime.strptime(date_str, '%Y-%m-%d').strftime('%Y-%m-%d') # 示例调用 standardized_date = standardize_date('2023-10-05') ``` #### 数据转换与映射 接下来,我们需要将清洗后的数据按照目标平台的要求进行转换和映射。根据元数据配置中的字段定义,我们可以使用模板引擎或脚本语言(如Python)来实现这一过程。 例如,对于`customer_id`字段,需要通过查询映射表获取对应的客户ID: ```python def get_customer_id(client_app_no): # 模拟查询操作 customer_mapping = { 'clientAppNo1': 'customerId1', 'clientAppNo2': 'customerId2' } return customer_mapping.get(client_app_no, '') # 示例调用 customer_id = get_customer_id('clientAppNo1') ``` 对于复杂对象,如`material_entity`数组中的各个子字段,需要逐一进行映射和转换: ```python def transform_material_entity(details): transformed_details = [] for detail in details: transformed_detail = { 'material_number': detail['easNo'], 'stock_id': '1277527712973493248', 'qty': detail['quantity'], 'unit_id': get_unit_id(detail['easNo']), 'price': detail['price'], # 更多字段... } transformed_details.append(transformed_detail) return transformed_details # 示例调用 transformed_materials = transform_material_entity(details) ``` #### 数据写入目标平台 完成所有转换后,我们将最终的数据结构通过API接口写入金蝶云星辰V2。具体实现如下: ```python import requests def write_to_kingdee(data): url = 'https://api.kingdee.com/jdy/v2/scm/sal_out_bound' headers = {'Content-Type': 'application/json'} response = requests.post(url, json=data, headers=headers) if response.status_code == 200: print('Data successfully written to Kingdee.') else: print(f'Failed to write data: {response.text}') # 示例调用 data_to_write = { 'bill_source': 'ISV', 'bill_date': standardized_date, 'customer_id': customer_id, # 更多字段... } write_to_kingdee(data_to_write) ``` 通过上述步骤,可以实现从源系统到金蝶云星辰V2API接口的数据无缝对接,确保每个环节的数据准确性和完整性。这一过程不仅提升了业务效率,还保证了数据处理的透明度和可追溯性。 ![金蝶与外部系统打通接口](https://pic.qeasy.cloud/T30.png~tplv-syqr462i7n-qeasy.image)