IOT Edge for Linux on Windows (eflow) - nested VM deployement - proxy settings - corporate firewall

Question

My team is working on achieving following:

1. Sending telemetry data from OPC Server to Azure Iot Hub.
2. The environment has restricted internet, so we opened required urls and ports (outbound).
3. OPC Server has no internet, so we made a Windows Virtual Machine on same environment.
4. This Windows Virtual Machine has IoT Edge for Linux on Windows (EFLOW) deployed on it.
5. We planned to send telemetry data using OPC Publisher Module to IoT Hub.

Helping Microsoft Article: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-configure-proxy-support?view=iotedge-1.4

The planned architecture is: enter image description here

Unfortunately, we are experiencing issues while achieving above. I am attaching logs. Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:26Z [INFO] - Starting Azure IoT Edge Daemon Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:26Z [INFO] - Version - 1.4.20 Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:26Z [INFO] - Obtaining Edge device provisioning data... Dec 26 18:19:26 WINDOWSVM-EFLOW systemd[1]: Started Azure IoT Identity Service. Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Starting service... Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Version - dev build Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Detected HTTPS proxy server http://123.45.67.89:8080/ Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Provisioning starting. Reason: Startup Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Updated device info for myEdgeDevice. Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Provisioning complete. Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Identity reconciliation started. Reason: Startup Dec 26 18:19:26 WINDOWSVM-EFLOW systemd[1]: Started Azure IoT Keys Service. Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Starting service... Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Version - dev build Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Loaded libaziot-keys with version 0x02010000 Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Starting server... Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "402"} Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: unexpected EOF Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: unexpected EOF Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [WARN] - Network not available for Identity reconciliation. Using offline backup from last run. Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - Starting server... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - <-- POST /identities/device?api-version=2020-09-01 {"content-type": "application/json", "host": "identityd.sock", "content-length": "16"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Device is myEdgeDevice on myIOThub.azure-devices.net Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Initializing module runtime... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Using runtime network id azure-iot-edge Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Successfully initialized module runtime Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- POST /keypair?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "61"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW systemd[1]: Started Azure IoT Certificates Service. Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - Starting service... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - Version - dev build Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - Starting server... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Using existing Edge CA certificate Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Certificate aziot-edged-ca will be auto-renewed. Next renewal at 2024-03-06T13:02:07+00:00. Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Stopping all modules... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - All modules stopped Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Detecting if device information has changed... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Device information has not changed Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting management API... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting workload API... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting image garbage collection task... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting watchdog with 60 second period... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Watchdog checking Edge runtime status Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - <-- PUT /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "identityd.sock", "content-length": "40"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "402"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: unexpected EOF Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: unexpected EOF Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - !!! Hub client error Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - !!! caused by: error trying to connect: unexpected EOF Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - !!! caused by: unexpected EOF Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - --> 404 {"content-type": "application/json"} Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:34Z [WARN] - Error in watchdog: Failed to update $edgeAgent identity: Hub client error Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-edged[14160]: caused by: error trying to connect: unexpected EOF Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-edged[14160]: caused by: unexpected EOF Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - <-- POST /identities/device/reprovision?api-version=2020-09-01 {"content-type": "application/json", "host": "identityd.sock", "content-length": "16"} Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Provisioning starting. Reason: Api Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Updated device info for myEdgeDevice. Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Provisioning complete. Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Identity reconciliation started. Reason: Api Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "402"} Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:36Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: unexpected EOF Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: unexpected EOF Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - !!! Hub client error Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - !!! caused by: error trying to connect: unexpected EOF Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - !!! caused by: unexpected EOF Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - --> 404 {"content-type": "application/json"}

We tried below:

1. Sending telemetry data from OPC Server to Azure Iot Hub.
2. The environment has restricted internet, so we opened required urls and ports (outbound).
3. OPC Server has no internet, so we made a Windows Virtual Machine on same environment.
4. This Windows Virtual Machine has IoT Edge for Linux on Windows (EFLOW) deployed on it.
5. We planned to send telemetry data using OPC Publisher Module to IoT Hub.

We expected stable connection between EFLOW and Iot Hub

Answer 1

It looks like there might be issues with the connectivity Firstly verify that the Windows Virtual Machine (WINDOWSVM-EFLOW) has the necessary outbound network connectivity to reach Azure IoT Hub. make sure that the required URLs and ports for Azure IoT Hub are open.

Since you mentioned a restricted internet environment with a proxy, make sure that the proxy configuration is correctly set up on the Windows Virtual Machine. check the proxu settings too, kindly double check these as these might be the cause,

Try to use debugging to find the issue

sudo iotedge check
sudo iotedge list
sudo iotedge logs <module-name>

Even after doing all of these still if it doesn't work, I recommend contacting azure support, they will assist this issue better. below link takes you directly to the support req page just fill in the details and hopefully it works.

https://portal.azure.com/#view/Microsoft_Azure_Support/NewSupportRequestV3Blade/callerWorkflowId/fa2e194f-ac47-4741-aad8-1cdd80fcfee1/callerName/Microsoft_Azure_Support%2FHelpAndSupportOverview.ReactView

Answer 2

Did you verify the Azure IoT Certificate you are using on your virtual machine is current? It's possible that an expired cert could be the issue.