Opcua-agent (thin-edge.io setup) - no free space on disk

Hi Community,
please see my question below.

What product/components do you use and which version/fix level?

Cumulocity
Backend: 1013.0.260
UI: 1011.0.12

Are you using a free trial or a product with a customer license?

License

What are trying to achieve? Please describe in detail.

I have a problem in my EDGE-Device (Secunet Gateway).

I run 3 docker images

thin-edge.io (opcua-solution-custom_tedge:latest 110MB)
Opcua agent (y509177/thin-edge.io-opcua-solution_gateway:latest 352MB)
hmic-service (use-case specific tasks, connecting to mqtt-bridge)

I know this is a very complex situation, and I can provide tons of data - for the moment I only dont know which data is helpful.

The problem is that after some time it happens that Opcua agent stops working, exitcode = 1, details about exit and status of opcua-agent are below.

I can see that somehow all free space (6.2 GB) is used of one file system (see below).

I did not see this before, but the last time it happen exactly at midnight 00:00:00, which may be or may not be important.

opcua agent wants to write, no disk space left - exit.

My question for the moment: did anybody have already a similar situation?

Thank you
BR Manfred

[
    {
        "Id": "6196a6ce88987284dca3e1b03142b9cebb5af37fdc4e0946029fe3543b9acd86",
        "Created": "2022-11-18T10:12:07.294543835Z",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "/usr/bin/java -Dlogging.config=file:./logging.xml -Dspring.profiles.active=default,tenant -jar opcua-device-gateway.jar -Dspring.config.location=file:./application-tenant.yaml"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 1,
            "Error": "write /bps/docker/var/lib/containers/6196a6ce88987284dca3e1b03142b9cebb5af37fdc4e0946029fe3543b9acd86/hash239207119: no space left on device",
            "StartedAt": "2022-11-18T10:12:09.357620988Z",
            "FinishedAt": "2022-11-20T14:45:37.229380429Z"
        },
        "Image": "sha256:202d746563e7331cb2165362bce2a9f0f665895c967896715c5c497ca27a31fb",
        "ResolvConfPath": "/bps/docker/var/lib/containers/6196a6ce88987284dca3e1b03142b9cebb5af37fdc4e0946029fe3543b9acd86/resolv.conf",
        "HostnamePath": "/bps/docker/var/lib/containers/6196a6ce88987284dca3e1b03142b9cebb5af37fdc4e0946029fe3543b9acd86/hostname",
        "HostsPath": "/bps/docker/var/lib/containers/6196a6ce88987284dca3e1b03142b9cebb5af37fdc4e0946029fe3543b9acd86/hosts",
        "LogPath": "",
        "Name": "/OPCUA",
        "RestartCount": 1,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/bps/docker/cfg/images/202d746563e7331cb2165362bce2a9f0f665895c967896715c5c497ca27a31fb-data:/var/edge",
                "/var/hostdir:/var/hostdir:ro",
                "/bps/dockerdata:/var/dockerdata"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "syslog",
                "Config": {
                    "syslog-address": "tcp://localhost:514",
                    "tag": "edge_docker_container_{{.ID}}"
                }
            },
            "NetworkMode": "tedge",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [
                "edge.internal:host-gateway"
            ],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/bps/docker/var/lib/overlay2/8f0a93f6b22b422774d0c806e48a4f1e7707edb1ca9a0b498afb1f1386419049-init/diff:/bps/docker/var/lib/overlay2/4bfffc952870e1180e3b45bf8709154d71083b557b45271963ace4b907f95ea7/diff:/bps/docker/var/lib/overlay2/701024eaf72534f38b02d4b11e8b526dc553d545c04926df69bd8ac2a4a00590/diff:/bps/docker/var/lib/overlay2/fdb57dac960522e3b975d46dd55c0fd118c8b457bd5ae88f555e26460d2a88a4/diff:/bps/docker/var/lib/overlay2/30d11379828b570705cf4df46fee9014aec372b273ecd3301d99d272f6fb1ba3/diff:/bps/docker/var/lib/overlay2/db85e15a953229d5eadf6e059926a430d4dfde25f5b5a7bdc5cfe29e809eb4e6/diff:/bps/docker/var/lib/overlay2/6b18c1a7d2d50f860ed890c51ef050f9968bf31ff1109289d7838802026cca00/diff",
                "MergedDir": "/bps/docker/var/lib/overlay2/8f0a93f6b22b422774d0c806e48a4f1e7707edb1ca9a0b498afb1f1386419049/merged",
                "UpperDir": "/bps/docker/var/lib/overlay2/8f0a93f6b22b422774d0c806e48a4f1e7707edb1ca9a0b498afb1f1386419049/diff",
                "WorkDir": "/bps/docker/var/lib/overlay2/8f0a93f6b22b422774d0c806e48a4f1e7707edb1ca9a0b498afb1f1386419049/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/bps/docker/cfg/images/202d746563e7331cb2165362bce2a9f0f665895c967896715c5c497ca27a31fb-data",
                "Destination": "/var/edge",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/hostdir",
                "Destination": "/var/hostdir",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/bps/dockerdata",
                "Destination": "/var/dockerdata",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "6196a6ce8898",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "mqtt.clientId=99000002",
                "hmidataProvider.baseUrl=http://192.168.11.1:10282",
                "deviceID=99000002",
                "gatewayIdentifier=GWID_99000002",
                "gatewayName=GWNA_99000002",
                "baseUrl=fronius-xxx.adamos.com",
                "tenantId=xxx",
                "component.cache.path=/home/spring",
                "download.cache.path=/home/spring",
                "logging.level.com.fronius.hmicservice.application.service.ProvideMachineComponents=debug",
                "logging.level.com.fronius.hmicservice.adapter.hmidataprovider=debug",
                "mqtt.broker=tcp://tedge:8883",
                "hmidataProvider.username=blabla",
                "hmidataProvider.password=blablabla",
                "c8y.baseUrl=https://fronius-xxx.adamos.com",
                "COMPONENT_LIST_SCHEDULER_CRON_EXPRESSION=0 */1 * * * *",
                "EDGE_ID=SE00010540D482",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "/usr/bin/java -Dlogging.config=file:./logging.xml -Dspring.profiles.active=default,tenant -jar opcua-device-gateway.jar -Dspring.config.location=file:./application-tenant.yaml"
            ],
            "Image": "202d746563e7331cb2165362bce2a9f0f665895c967896715c5c497ca27a31fb",
            "Volumes": null,
            "WorkingDir": "/app",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "00565135375e1d66289a1b65eefeb4018f17dac9bfb4a0e7c876297eccb60641",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/00565135375e",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "tedge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "6196a6ce8898"
                    ],
                    "NetworkID": "6d33dd36dea7ad02efde2461bdc2e3ce7feb970bc367454d947add36724a7269",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

Docker limits the size of a containers via “default docker storage” on docker/var/lib/

Link 1

Link 2

You can increase that, however I did not check about why docker sets that limit. This might have side effects.

Hi Manfred,

could you please provide stack trace of this no disk space left error? The OPC UA Gateway is writing a local db file, to persist the address space. I could imaging that this file is becoming to big for your configured volume. Another problem could be the log files, if file appender is configured.

Best regards,
Alex

1 Like

Hi,

I made many tests and could pin the problem to the opcua container.

First it looked like there is a link to the deadband settings, but after all it seems not.

I could now see that it is indeed a logging problem.

Inside the container:
the folder /root/.opcua/log was > 9GB, with log files and it was increasing in size rapidly.

In “best times” every some minutes 33 MB. This is the increase of the virtual disk of WSL-2

I deleted all, now it seems it is not logging at all anymore…

I can see the same (but really less data) with mosquitto. also the log of mosquitto is growing.

It seems nobody is taking care of cleaning up the log files, like some first in / first out on files…

How is this supposed to work at all?
What can I do to eleminate logging as - honestly - nobody will ever see this files inside a container.

Thank you.
BR Manfred

EDIT: additional info

While the space occupation of the logging is rather moderate at the moment, it was quite consuming before I removed all log-files and started the containers over.

See this table:
left row: increase in Byte (compare to last “du” command, 17 sec. earlier), in 17 seconds 1,5 MB

00:00:17	 Mon 	Dec
 872 	 5 824 710 	docker/overlay2/fa851d42da84b2ecc972fbafb51b04f06386e38b04966618dcb9e0a095bfd99c/merged/var/log/mosquitto
 872 	 5 874 009 	docker/overlay2/fa851d42da84b2ecc972fbafb51b04f06386e38b04966618dcb9e0a095bfd99c/merged/var/log
 872 	 8 254 299 	docker/overlay2/fa851d42da84b2ecc972fbafb51b04f06386e38b04966618dcb9e0a095bfd99c/merged/var
 872 	 67 363 751 	docker/overlay2/fa851d42da84b2ecc972fbafb51b04f06386e38b04966618dcb9e0a095bfd99c/merged
 872 	 5 824 710 	docker/overlay2/fa851d42da84b2ecc972fbafb51b04f06386e38b04966618dcb9e0a095bfd99c/diff/var/log/mosquitto
 872 	 5 874 009 	docker/overlay2/fa851d42da84b2ecc972fbafb51b04f06386e38b04966618dcb9e0a095bfd99c/diff/var/log
 872 	 5 878 105 	docker/overlay2/fa851d42da84b2ecc972fbafb51b04f06386e38b04966618dcb9e0a095bfd99c/diff/var
 872 	 6 021 356 	docker/overlay2/fa851d42da84b2ecc972fbafb51b04f06386e38b04966618dcb9e0a095bfd99c/diff
 1 744 	 73 398 928 	docker/overlay2/fa851d42da84b2ecc972fbafb51b04f06386e38b04966618dcb9e0a095bfd99c
 425 549 	 4 711 782 686 	docker/overlay2/51c2de4d4911da2047414c7eba77d210cd9bc4a6c8e2f98b3db3647c99b9143f/merged/root/.opcua/log
 425 549 	 4 711 786 782 	docker/overlay2/51c2de4d4911da2047414c7eba77d210cd9bc4a6c8e2f98b3db3647c99b9143f/merged/root/.opcua
 425 549 	 4 711 791 624 	docker/overlay2/51c2de4d4911da2047414c7eba77d210cd9bc4a6c8e2f98b3db3647c99b9143f/merged/root
 425 549 	 5 078 756 545 	docker/overlay2/51c2de4d4911da2047414c7eba77d210cd9bc4a6c8e2f98b3db3647c99b9143f/merged
 425 549 	 4 711 782 686 	docker/overlay2/51c2de4d4911da2047414c7eba77d210cd9bc4a6c8e2f98b3db3647c99b9143f/diff/root/.opcua/log
 425 549 	 4 711 786 782 	docker/overlay2/51c2de4d4911da2047414c7eba77d210cd9bc4a6c8e2f98b3db3647c99b9143f/diff/root/.opcua
 425 549 	 4 711 791 624 	docker/overlay2/51c2de4d4911da2047414c7eba77d210cd9bc4a6c8e2f98b3db3647c99b9143f/diff/root
 425 549 	 4 725 222 080 	docker/overlay2/51c2de4d4911da2047414c7eba77d210cd9bc4a6c8e2f98b3db3647c99b9143f/diff
 851 098 	 9 803 991 141 	docker/overlay2/51c2de4d4911da2047414c7eba77d210cd9bc4a6c8e2f98b3db3647c99b9143f
 852 842 	 11 334 650 163 	docker/overlay2
 727 268 	 8 099 267 854 	docker/containers/934931ea5a7fee594e5f1b278654cec5ecf2547e0e86d79c25ff3e51a5e71b4e
 727 268 	 8 116 957 956 	docker/containers
 1 580 110 	 19 453 162 895 	docker

–
Increase of virtual disk (WSL-2)


2022-12-05 10:42:55 __ Free Space   72096 ->   72064 diff  -32   vdisk =   17089 ->   17123  diff   33 
2022-12-05 10:51:15 __ Free Space   72056 ->   72024 diff  -32   vdisk =   17123 ->   17156  diff   33 
2022-12-05 10:59:45 __ Free Space   72016 ->   71984 diff  -32   vdisk =   17156 ->   17190  diff   33 
2022-12-05 11:07:40 __ Free Space   71988 ->   71956 diff  -32   vdisk =   17190 ->   17223  diff   33 
2022-12-05 11:16:15 __ Free Space   71946 ->   71914 diff  -32   vdisk =   17223 ->   17257  diff   33 
2022-12-05 11:24:10 __ Free Space   71899 ->   71867 diff  -32   vdisk =   17257 ->   17291  diff   33 
2022-12-05 11:32:50 __ Free Space   71860 ->   71828 diff  -32   vdisk =   17291 ->   17324  diff   33

Hi Manfred,

first of all I would basically mount the log directory to the host system, that way you can access the logs and the container itself does not get that big. -v PATH_OF_HOST:PATH_OF_CONTAINER_LOGS

However the logging.xml seemed to be configured properly, although it does not delete the files. You could adjust within the logging.xml an additional parameter (totalSizeCap) within the section.

500MB

That way all files together would not reach 500 MB and will be deleted. I will update the image on docker hub according to that.

Here is a guide describing it:

Hi Murat,
thank you for your answer.

first of all I would basically mount the log directory to the host system, that way you can access the logs and the container itself does not get that big. -v PATH_OF_HOST:PATH_OF_CONTAINER_LOGS

It does not make a big difference if the log files on my hard drive are consuming space or the virtual linux disk is increasing. Also I don’t know where to apply this switch you mention
-v PATH_OF_HOST:PATH_OF_CONTAINER_LOGS

About the new image you want to create I have these questions

  1. Would it be possible to use a newer version of Opc-ua agent as well? The one used now creates that big amount of exceptions, we had the same problem before we used thin-edge.io in the past. Back than we upgraded from
    opcua-device-gateway-1011.0.4.jar
    to
    opcua-device-gateway-1011.0.22.jar
    and that resolved the problem with exceptions.

  2. Would it somehow be possible to set some basic logging configuration at the moment of creating the container (e.g. environment file preferable). Like max size, logging level, …

  3. What timeline can you give until a new image is ready?

Thank you
Regards, Manfred

Hi Manfred,

  1. Don`t you use thin-edge.io anymore? That way you could also use the resources from here: Link. A newer thin-edge.io OPCUA Agent version should be released that year, currently I have no insights on any timeline here.

  2. As Support already mentioned, this is possible. The logging.xml can be mounted into the container via -v. The important parameters within the xml are “totalSizeCap” and “maxFileSize”. You can find more about that here: Link

  3. I can create a new image if a newer OPC UA Agent version with thin-edge.io support is available.

Hi Murat,
thank you for the answer.

Yes, we still use thin-edge.io, so we can’t use another OPC-UA Agent.

We will try to use our customized logging.xml and hopefully we can have a workaround with this.

BR Manfred

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.