Not able to add custom fields on datahub offloading job

[vachaspati_Diwevedi1 (vachaspati Diwevedi)]

What product/components do you use and which version/fix level are you on?

Cumulocity datahub, version 10.13

Is your question related to the free trial, or to a production (customer) instance?

Production customer

What are you trying to achieve? Please describe it in detail.

I am trying to add additional fields while configuring the data offloading job for events.
Below event have been stored in the cumulocity.
{
“creationTime”: “2022-09-16T05:54:16.684Z”,
“source”: {
“name”: “ktg_pms_reader_exit”,
“self”: “”,
“id”: “83946990”
},
“type”: “ktg_exit”,
“lastUpdated”: “2022-09-16T05:54:16.684Z”,
“self”: “<>”,
“time”: “2022-09-16T05:54:16.684Z”,
“id”: “84100575”,
“text”: “exit”,
“VehType”: “car”,
“exittime”:“”,
“entrytime”: “”,
“seasontype”: “Season”,
“IU”: “U000001234567890”,
“payamt”: “0”,
“Plate”: “SG1234”
}

While configuring the datahub job for events, I do not see the custom event field auto-detected. When I am trying to add manually I am getting below error message.

I have tried the same without double quote as well as without “src”, I am getting the same error message.

Do you get any error messages? Please provide a full error message screenshot and log file.

Have you installed all the latest fixes for the products and systems you are using?

Yes

[Murat_Bayram]

The syntax in the Datahub offloading is a bit tricky, but @Tim_Doernemann1 can help out here

[Tim_Doernemann1 (Tim Doernemann)]

Hello,

I think the issue here is not the tricky syntax (IMHO, it can be a bit complex when you want to drill into nested structures, but for top-level JSON fields it is pretty straight forward). In other words: the syntax seems to be correct, but I guess the problem is a known issue: the field “VehType” is not present in all events, but it was added after some time, i.e. there is a non-neglectable number of events without “VehType” and then new events with the field.

Dremio (the internal engine in CDH) has learned the MongoDB schema without “VehType” and thus it is required to trigger schema learning again. Unfortunately, this is not a function you can run from the UI, but you need to run an external script (Sign in to your account) and need an admin account for Dremio. The script is run like that (documentation is contained in the ZIP file):

python datahub_fix_table_rest.py --host <host> --tenant <tenant_name> --user <admin_user> --password <password>

On which system is the problem occurring? If one of the Public Cloud instances is affected, I can assist.

Thanks, Tim