Opcua-agent (thin-edge.io setup) - no free space on disk

Hi Community,
please see my question below.

What product/components do you use and which version/fix level?

Cumulocity
Backend: 1013.0.260
UI: 1011.0.12

Are you using a free trial or a product with a customer license?

License

What are trying to achieve? Please describe in detail.

I have a problem in my EDGE-Device (Secunet Gateway).

I run 3 docker images

thin-edge.io (opcua-solution-custom_tedge:latest 110MB)
Opcua agent (y509177/thin-edge.io-opcua-solution_gateway:latest 352MB)
hmic-service (use-case specific tasks, connecting to mqtt-bridge)

I know this is a very complex situation, and I can provide tons of data - for the moment I only dont know which data is helpful.

The problem is that after some time it happens that Opcua agent stops working, exitcode = 1, details about exit and status of opcua-agent are below.

I can see that somehow all free space (6.2 GB) is used of one file system (see below).

I did not see this before, but the last time it happen exactly at midnight 00:00:00, which may be or may not be important.

opcua agent wants to write, no disk space left - exit.

My question for the moment: did anybody have already a similar situation?

Thank you
BR Manfred

[
    {
        "Id": "6196a6ce88987284dca3e1b03142b9cebb5af37fdc4e0946029fe3543b9acd86",
        "Created": "2022-11-18T10:12:07.294543835Z",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "/usr/bin/java -Dlogging.config=file:./logging.xml -Dspring.profiles.active=default,tenant -jar opcua-device-gateway.jar -Dspring.config.location=file:./application-tenant.yaml"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 1,
            "Error": "write /bps/docker/var/lib/containers/6196a6ce88987284dca3e1b03142b9cebb5af37fdc4e0946029fe3543b9acd86/hash239207119: no space left on device",
            "StartedAt": "2022-11-18T10:12:09.357620988Z",
            "FinishedAt": "2022-11-20T14:45:37.229380429Z"
        },
        "Image": "sha256:202d746563e7331cb2165362bce2a9f0f665895c967896715c5c497ca27a31fb",
        "ResolvConfPath": "/bps/docker/var/lib/containers/6196a6ce88987284dca3e1b03142b9cebb5af37fdc4e0946029fe3543b9acd86/resolv.conf",
        "HostnamePath": "/bps/docker/var/lib/containers/6196a6ce88987284dca3e1b03142b9cebb5af37fdc4e0946029fe3543b9acd86/hostname",
        "HostsPath": "/bps/docker/var/lib/containers/6196a6ce88987284dca3e1b03142b9cebb5af37fdc4e0946029fe3543b9acd86/hosts",
        "LogPath": "",
        "Name": "/OPCUA",
        "RestartCount": 1,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/bps/docker/cfg/images/202d746563e7331cb2165362bce2a9f0f665895c967896715c5c497ca27a31fb-data:/var/edge",
                "/var/hostdir:/var/hostdir:ro",
                "/bps/dockerdata:/var/dockerdata"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "syslog",
                "Config": {
                    "syslog-address": "tcp://localhost:514",
                    "tag": "edge_docker_container_{{.ID}}"
                }
            },
            "NetworkMode": "tedge",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [
                "edge.internal:host-gateway"
            ],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/bps/docker/var/lib/overlay2/8f0a93f6b22b422774d0c806e48a4f1e7707edb1ca9a0b498afb1f1386419049-init/diff:/bps/docker/var/lib/overlay2/4bfffc952870e1180e3b45bf8709154d71083b557b45271963ace4b907f95ea7/diff:/bps/docker/var/lib/overlay2/701024eaf72534f38b02d4b11e8b526dc553d545c04926df69bd8ac2a4a00590/diff:/bps/docker/var/lib/overlay2/fdb57dac960522e3b975d46dd55c0fd118c8b457bd5ae88f555e26460d2a88a4/diff:/bps/docker/var/lib/overlay2/30d11379828b570705cf4df46fee9014aec372b273ecd3301d99d272f6fb1ba3/diff:/bps/docker/var/lib/overlay2/db85e15a953229d5eadf6e059926a430d4dfde25f5b5a7bdc5cfe29e809eb4e6/diff:/bps/docker/var/lib/overlay2/6b18c1a7d2d50f860ed890c51ef050f9968bf31ff1109289d7838802026cca00/diff",
                "MergedDir": "/bps/docker/var/lib/overlay2/8f0a93f6b22b422774d0c806e48a4f1e7707edb1ca9a0b498afb1f1386419049/merged",
                "UpperDir": "/bps/docker/var/lib/overlay2/8f0a93f6b22b422774d0c806e48a4f1e7707edb1ca9a0b498afb1f1386419049/diff",
                "WorkDir": "/bps/docker/var/lib/overlay2/8f0a93f6b22b422774d0c806e48a4f1e7707edb1ca9a0b498afb1f1386419049/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/bps/docker/cfg/images/202d746563e7331cb2165362bce2a9f0f665895c967896715c5c497ca27a31fb-data",
                "Destination": "/var/edge",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/hostdir",
                "Destination": "/var/hostdir",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/bps/dockerdata",
                "Destination": "/var/dockerdata",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "6196a6ce8898",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "mqtt.clientId=99000002",
                "hmidataProvider.baseUrl=http://192.168.11.1:10282",
                "deviceID=99000002",
                "gatewayIdentifier=GWID_99000002",
                "gatewayName=GWNA_99000002",
                "baseUrl=fronius-xxx.adamos.com",
                "tenantId=xxx",
                "component.cache.path=/home/spring",
                "download.cache.path=/home/spring",
                "logging.level.com.fronius.hmicservice.application.service.ProvideMachineComponents=debug",
                "logging.level.com.fronius.hmicservice.adapter.hmidataprovider=debug",
                "mqtt.broker=tcp://tedge:8883",
                "hmidataProvider.username=blabla",
                "hmidataProvider.password=blablabla",
                "c8y.baseUrl=https://fronius-xxx.adamos.com",
                "COMPONENT_LIST_SCHEDULER_CRON_EXPRESSION=0 */1 * * * *",
                "EDGE_ID=SE00010540D482",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "/usr/bin/java -Dlogging.config=file:./logging.xml -Dspring.profiles.active=default,tenant -jar opcua-device-gateway.jar -Dspring.config.location=file:./application-tenant.yaml"
            ],
            "Image": "202d746563e7331cb2165362bce2a9f0f665895c967896715c5c497ca27a31fb",
            "Volumes": null,
            "WorkingDir": "/app",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "00565135375e1d66289a1b65eefeb4018f17dac9bfb4a0e7c876297eccb60641",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/00565135375e",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "tedge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "6196a6ce8898"
                    ],
                    "NetworkID": "6d33dd36dea7ad02efde2461bdc2e3ce7feb970bc367454d947add36724a7269",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

Docker limits the size of a containers via “default docker storage” on docker/var/lib/

Link 1

Link 2

You can increase that, however I did not check about why docker sets that limit. This might have side effects.

Hi Manfred,

could you please provide stack trace of this no disk space left error? The OPC UA Gateway is writing a local db file, to persist the address space. I could imaging that this file is becoming to big for your configured volume. Another problem could be the log files, if file appender is configured.

Best regards,
Alex

1 Like