Appear to be locking up under heavy traffic

Question

We have an AWS IoT based device using the W5500 chip. The device works flawlessly when it connects directly to a router or a switch. However, it seems to be locking up if it is plugged into a wall jack in a large commercial building. Here are some of our observations: My company has several buildings and each has its own network segment. If this device is plugged into a wall jack in a relatively small building, typical IP 10.66.xxx.xxx, we do see disconnect events at the AWS IoT application layer. However, the device can recover/reconnect without a problem. If the device is plugged into a wall jack in the large multi-story office building, typical IP 10.68.xxx.xxx, it appears to lock up from time to time. When this problem happens: There is no traffic coming out of W5500 when we put a network sniffer (TAP) on. The ICMP ping gets no response. We can still read/write all internal registers. We monitor the SIP register in case there is a reset on the W5500 side, but we haven’t seen one. Again, if we install a LAN switch between the device and the wall jack, the problem seems to go away. The W5500 runs at 25MHz and the circuit design follows the recommendations provided by WIZNET. The application firmware doesn’t generate much network traffic. Most of time it is sending MQTT Keep Alive message once every 30-seconds. Not sure if this matters - In the firmware we set to use 10BT Half-duplex with auto-negotiation disabled even though PMODE 0-2 pins are all pulled high. Any thoughts? Thanks.

Eugeny · Answer

acpie360: The device works flawlessly when it connects directly to a router or a switch. Router/switch filters the traffic, sending to W5500 only traffic designated to it and broadcasts. acpie360: However, it seems to be locking up if it is plugged into a wall jack in a large commercial building. I used W5100 (which has more or less the same engine), and empirically found out that W5x00 does not perform well under high load, in particular the issue, which I caught, is when TCP communication gets out of order - W5100 can not recover properly from it. But your case sounds different, as you say there’s nothing sent from the W5500 - in my case it sends some packets when its times out (can be up to 2 minutes). You already have a workaround for the problem - buy cheapest router, and put it before W5500. Ugly, but it should work While you performed initial troubleshooting, more steps are needed to find the cause. acpie360: There is no traffic coming out of W5500 when we put a network sniffer (TAP) on. I usually use Wireshark, but I guess your network sniffer is set up in correct location of the network (connected directly to W5500?) and can capture everything on the wire. Then, how much time did you wait to decide that there’s nothing in there? 30 seconds? 5 minutes? Remember W5500 works according to its timeout algorithm, which may cause it looking hung for minutes. acpie360: The ICMP ping gets no response. Needs more monitoring time to ensure there’s nothing on the wire for minutes after ping request is sent. acpie360: We can still read/write all internal registers. Dump all registers (common as well as socket for all sockets) to see anomalities. acpie360: We monitor the SIP register in case there is a reset on the W5500 side, but we haven’t seen one. That is a good way to detect reset. acpie360: The application firmware doesn’t generate much network traffic. Most of time it is sending MQTT Keep Alive message once every 30-seconds. Then logically, W5500 gets something…

acpie360 · Answer

Thanks for the reply. Remember W5500 works according to its timeout algorithm,… Can you tell me where to find this timeout algorithm you were referring to? I assume we are using the default settings at the moment since there is no code on our side that sets up anything related to timeout. Perhaps to say that nothing coming out of W5500 is a bit premature because I don’t think we knew what to look for at the time of system locking up or had waited long enough. The Wireshark capture, through the use of a network sniffer, was showing tons of seemingly unrelated traffic. FYI - This is the sniffer we are using: https://www.amazon.com/Dualcomm-1000Base-T-Gigabit-Ethernet-Network/dp/B004EWVFAY/ref=sr_1_3?ie=UTF8&qid=1543770322&sr=8-3&keywords=network+tap We also suspect that there might be electrical/wiring issues in this large building because when we put a network sniffer between the W5500 and the wall jack, we can’t seem to reproduce the problem. This makes the troubleshooting really difficult. We have to not use the sniffer first to make the problem happen, then insert the sniffer after the problem occurs. Of course, this causes the PHY layer to disconnect then reconnect.

acpie360 · Answer

Eugeny: Do you have only one W5500-based device under consideration? There is another product line being developed at the moment. However, the software on the cloud side isn’t as ready as the product I am currently testing. Eugeny: Can you build into your W5500 driver the procedure to reset PHY (PHYCFGR bit 7) so that you can execute this code when chip locks up and see if it helps? I’ll see if I can do this. Good idea. Thanks.