Can't get logfiles and configuration files using thinedge

[screwdriver_logfile]

Hello everyone,

I try to get logfiles and configuration files using thinedge to my Cumulocity Edge on k8s instance.
Cumulocity Edge k8s instance version : 10.18
thin edge version : 1.1.1
The VM which host thin edge doesn’t has internet connection.
Thin-Edges was updated from 1.1.0 to 1.1.1 manually with all the packages .deb
The client thin-edge is well connected to the platform.

But I can’t get logfiles and configuration files, more details please check the following pictures.

Can someone help me ?

Thank you everybody

juin 06 15:21:43 VM tedge-agent[153170]: 2024-06-06T13:21:43.545753179Z  INFO tedge_config_manager::actor: Config Snapshot received: ConfigSnapshotCmdPayload { status: Init, tedge_url: Some("http://127.0.0.1:8000/tedge/file-transfer/ppbdiot01/config_snapshot/tedge-configuration-plugin-c8y-mapper-2504"), config_type: "tedge-configuration-plugin", path: None, log_path: None }
juin 06 15:21:43 VM tedge-agent[153170]: 2024-06-06T13:21:43.54588981Z  INFO tedge_agent::tedge_operation_converter::actor: Ignoring config_snapshot operation which is not registered
juin 06 15:21:43 VM tedge-agent[153170]: 2024-06-06T13:21:43.588623098Z  INFO tedge_config_manager::actor: Awaiting upload of config type: tedge-configuration-plugin to url: http://127.0.0.1:8000/tedge/file-transfer/ppbdiot01/config_snapshot/tedge-configuration-plugin-c8y-mapper-2504
juin 06 15:21:43 VM tedge-agent[153170]: 2024-06-06T13:21:43.588801365Z  INFO tedge_agent::tedge_operation_converter::actor: Ignoring config_snapshot operation which is not registered
juin 06 15:21:43 VM tedge-agent[153170]: 2024-06-06T13:21:43.588950381Z  INFO tedge_uploader_ext::actor: Uploading from /etc/tedge/plugins/tedge-configuration-plugin.toml to url: http://127.0.0.1:8000/tedge/file-transfer/ppbdiot01/config_snapshot/tedge-configuration-plugin-c8y-mapper-2504
juin 06 15:21:43 VM tedge-agent[153170]: 2024-06-06T13:21:43.613684035Z  INFO tedge_config_manager::actor: Config Snapshot request processed for config type: tedge-configuration-plugin.
juin 06 15:21:43 VM tedge-agent[153170]: 2024-06-06T13:21:43.61577998Z  INFO tedge_agent::tedge_operation_converter::actor: Ignoring config_snapshot operation which is not registered
juin 06 15:21:43 VM tedge-agent[153170]: 2024-06-06T13:21:43.832153828Z  INFO tedge_agent::tedge_operation_converter::actor: Ignoring config_snapshot operation which is not registered
juin 06 15:21:47 VM tedge-agent[153170]: 2024-06-06T13:21:47.266278022Z  INFO tedge_agent::tedge_operation_converter::actor: Ignoring config_update operation which is not registered
juin 06 15:21:47 VM tedge-agent[153170]: 2024-06-06T13:21:47.266302357Z  INFO tedge_config_manager::actor: Config Update received: ConfigUpdateCmdPayload { status: Init, tedge_url: None, remote_url: "http://127.0.0.1:8001/c8y/inventory/binaries/2345", config_type: "tedge-configuration-plugin", path: None, log_path: None }
juin 06 15:21:47 VM tedge-agent[153170]: 2024-06-06T13:21:47.268179226Z  INFO tedge_agent::operation_file_cache: Downloading config file from `http://127.0.0.1:8001/c8y/inventory/binaries/2345` to cache
juin 06 15:21:47 VM tedge-agent[153170]: 2024-06-06T13:21:47.268253381Z  INFO tedge_agent::tedge_operation_converter::actor: Ignoring config_update operation which is not registered
juin 06 15:21:47 VM tedge-agent[153170]: 2024-06-06T13:21:47.268439003Z  INFO tedge_downloader_ext::actor: Downloading from url http://127.0.0.1:8001/c8y/inventory/binaries/2345 to location /var/tedge/cache/41da61d495f9f7a5f7efcaa373ddf2a08f88439da5be4211c21e09baa44e98a4
juin 06 15:21:47 VM tedge-agent[153170]: 2024-06-06T13:21:47.321238189Z ERROR tedge_agent::operation_file_cache: tedge-agent failed downloading a file: HTTP status client error (400 Bad Request) for url (http://127.0.0.1:8001/c8y/inventory/binaries/2345)
juin 06 15:21:47 VM tedge-agent[153170]: 2024-06-06T13:21:47.323921969Z  INFO tedge_agent::tedge_operation_converter::actor: Ignoring config_update operation which is not registered
juin 06 15:21:47 VM tedge-agent[153170]: 2024-06-06T13:21:47.40089661Z  WARN tedge_agent::operation_file_cache: Received config update, but payload is malformed: EOF while parsing a value at line 1 column 0
juin 06 15:21:47 VM tedge-agent[153170]: 2024-06-06T13:21:47.401042626Z  INFO tedge_agent::tedge_operation_converter::actor: Ignoring config_update operation which is not registered

[yiz (Yishu Zhong)]

Here are the other screenshots:

[didier.wenzek]

Hi,

thin-edge use a local HTTP proxy to access the Cumulocity IoT instance (per default on https://127.0.0.1:8001).
Here, this proxy is responding but fails to interpret the request.

As suggested by Yishu, the command curl https://127.0.0.1:8001/c8y/tenant/currentTenant should reach the Cumulocity tenant and return a JSON response.

• Were log update and config management working before the update to thin-edge 1.1.1 ?
• What is the configuration of the c8y proxy (tedge config list | grep c8y.proxy)?
• What is the configuration of the file transfer service (tedge config list | grep http)?
• If the mapper and the agent are not deployed on the same box, can you give the tedge config settings on both?

[didier.wenzek]

Noticing that the response for curl https://127.0.0.1:8001/c8y/tenant/currentTenant is produced by nginx, i.e. coming from Cumulocity and not a thin-edge component, I would discard a thin-edge configuration issue.

• Were log update and config management working before the update to thin-edge 1.1.1 ?
• What is the response of the tenant via another tool such as go-c8y-cli?

[screwdriver_logfile]

• Were log update and config management working before the update to thin-edge 1.1.1 ?
  => In fact I have two instances of cumulocity, one edge instance on an appliance and one edge instance on k8s. The log update and config management used to works with the tedge 1.1.0 linked to the appliance. Now, I try to do the same on the k8s edge instance and it’s not working, with 1.1.0 or 1.1.1. I just retry with 1.1.1 on edge applicance and it’s working …

• What is the response of the tenant via another tool such as go-c8y-cli?
  => I haven’t access to the internet, even if I download the file from github, it tries to do a curl, which can’t work.

[Korbinian_Butz]

You could try:

• doing a test request: curl -u "{username}:{password}" 'https://{your-tenant-url}/tenant/currentTenant' and
• download your file via wget: wget --user "{username}" --password "{password}" -O your-file-name.txt https://{your-tenant-url}/inventory/binaries/{your binary id}

Use the url of your Edge instance, not the local proxy of thin-edge. From your description I would think these request will also fail - so the problem has to do with Edge-K8S and network configuration, not with thin-edge in the first place.

[screwdriver_logfile]

Hi,

I just try these commands from the VM which host tedge, they run successfully.

curl -u “{username}:{password}” ‘https://url-VM_instance/tenant/currentTenant’

curl -u “{username}:{password}” ‘https://url-k8s_instance/tenant/currentTenant’

What else can I test?

[Reuben_Miller2 (Reuben Miller)]

@screwdriver_logfile

url-k8s_instance

It might also be helpful to show the thin-edge.io configuration, so that other settings can be checked:

tedge config list

[didier.wenzek]

Can you check that the thin-edge settings for c8y.url are correctly set on for both one edge instances (appliance and k8s)?
.i.e. that you are using the same url-VM_instance and url-k8s_instance as for the manual connection.

Here are the relevant tedge config settings on my device:

$ tedge config list | grep c8y      
c8y.url=didier.latest.stage.c8y.io

c8y.http=didier.latest.stage.c8y.io:443
c8y.mqtt=didier.latest.stage.c8y.io:8883

c8y.proxy.bind.address=127.0.0.1
c8y.proxy.bind.port=8001
c8y.proxy.client.host=127.0.0.1
c8y.proxy.client.port=8001

[screwdriver_logfile]

Currently, my tedge is connected to the k8s instance, my config is :
. c8y.http=url-k8s_instance:443
. c8y.mqtt=url-k8s_instance:8883

I have already the issue of :

me@VM:/home/directory# curl http://127.0.0.1:8001/c8y/tenant/currentTenant
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

[screwdriver_logfile]

Hi everyone,

Watching the logfiles of the ingress controller, it seems that the issue comes from a duplication of the host header.
This issue is solved by the 1.1.2~120+gf53fecb version of thin-edge.

Thank you for you help, I close the topics.

Bye !

[Reuben_Miller2 (Reuben Miller)]

Just for other users, it was solved by this PR in the thin-edge.io repository, fix: Remove unwanted host header before forwarding request to Cumulocity by jarhodes314 · Pull Request #2946 · thin-edge/thin-edge.io · GitHub.

The bug only appears if there is an ingress controller which has strict HTTP header validation which rejects HTTP requests which have a duplicate HTTP header. Cloud Cumulocity IoT and thick edge users not using an ingress controller are NOT affected.

The fix will be included in the next official release, and can already be used by installing from the main branch, e.g.

wget -O - thin-edge.io/install.sh | sh -s -- --channel main

[didier.wenzek]

The root cause was a duplicated HTTP header sent by thin-edge c8y proxy. This header was rejected by Kubernetes Ingress between Thin-edge and Cumulocity IoT Edge.

This has been fixed by thin-edge version 1.1.2~122+gf53fecb and will be released with 1.1.2.