W5300 does not respond to SYN request from peer

Question

Hello everyone! Recently there is an Ethernet connection issue of our product which uses W5300. After analyzing the packets, we found when the issue happens, W5300 does not respond to SYN request from the peer. You can see several occasions in the attachment like packet ID 216, 256, 338, 369. They are SYN request from 10.54.53.240 (a remote peer) to 10.15.1.80 (W5300), but W5300 does not give SYN ACK. 2018-11-01%2010_38_06-Wiznet%20packets%20capture831×238 21.9 KB We believe there are still free sockets on W5300 and as we tested, even if all sockets are occupied, W5300 should send a RST instead of keeping silent. As I understand from the specification, the transition from SOCK_LISTEN through SOCK_SYNRECV to SOCK_ESTABLISHED should be controlled by W5300 itself not application. Interestingly, W5300 is still working when this issue happens because another remote site 10.10.28.240 is talking to it. It seems W5300 just cannot respond anything to new SYN request at that moment. A software RST to MR1 of W5300 can restore the situation and it can accept new SYN and talk to both sites correctly. Both remote sites (10.54.53.240 and 10.10.28.240) use the same MAC address. This is because two sites are connected to the local site via WAN and a router. The router is connected to one port of one local switch and 10.15.1.80 is connected to another port of the same switch. It should be a normal topology. Did anyone come across the similar issue? Thanks in advance! Wiznet packets capture.zip (11.2 KB)

becky · Answer

Hi This problem is due to your network configuration. 10.54.53.240 is in a different network band from 10.15.1.80, and it is judged that the packet transmitted from 10.54.53.240 does not reach 10.15.1.80.

Eugeny · Answer

It looks like really W5300 ignores the packets not responding to them, even to the retramsmitted one - thus it is not just a kind of “packet loss” - it is steady behavior in some W5300 state. sensiwood: We believe there are still free sockets on W5300 You must prove it - that when this happens, you have at least one socket in LISTEN state. If you have diagnostic terminal to W5300, then program its driver to print all common and socket registers into this diagnostic output (e.g. pressing a key on the diag terminal) so that you can see whole the configuration when issue happens. sensiwood: W5300 should send a RST instead of keeping silent I suspect it is not correct assumption. In my web server application, when client’s web browser issues more requests than sockets available, all new requests are “hung” - browser keeps waiting, until it timeouts. This means for me that chip (I use W5100) simply does not respond - otherwise if browser’s stack would receive RST, it stopped waiting immediately. sensiwood: As I understand from the specification, the transition Yes, it is controlled by the W5300 TCP/IP stack. Thus if something stops working, you first must look, carefully and thoroughly, into the current effective configuration of the chip at the time of the issue. sensiwood: Interestingly, W5300 is still working when this issue happens because another remote site 10.10.28.240 is talking to it. It confirms that chip is still operational, and there’s something else than faulty hardware. sensiwood: A software RST to MR1 of W5300 can restore the situation and it can accept new SYN and talk to both sites correctly. Also logical as you reset whole configuration of the chip. My conclusion - further troubleshooting is needed dumping all possible meaningful registers at the time after issue starts happening. I am sure we will find something interesting in the output.

sensiwood · Answer

Hi Eugeny, Sorry for the late reply. I appreciate very much for your detailed comments. And it is not possible at the moment to debug anything on the site so I was still trying to reproduce it on my own test bench. To dump all the registers is a good idea and I will try. Hope I can come back with some useful information in following days. Thanks again for your help!

sensiwood · Answer

Hi Eugeny, I did some tests on my test bench. The only way to reproduce the issue is, to set up two devices with the different IP addresses but the same MAC addresses. Device 1 (TCP client), IP 192.168.30.10, MAC 00:01:19:01:01:01 Device 2 (TCP client), IP 192.168.30.11, MAC 00:01:19:01:01:01 Device 3 (TCP server), IP 192.168.30.17, MAC 00:01:19:64:01:02 All devices are connected to one switch to form a LAN. Device 1 and 2 send the request to device 3 and device 3 responses. I let the system run for some time (10 minutes) and I found device 3 does not send any response to SYN request any more. The only way to recover the communication is to reset the W5300. This is the similar phenomenon as our customer reports. I reckon this is not desirable for LAN configuration because typically the MAC address over the LAN should be unique. But at the moment it is the only way to reproduce the issue stably. And even if there are two MAC addresses on the LAN, this will confuse the switch only. For ARP table of the device, I think it should be OK if several target IP addresses use the same MAC address. Correct me if I am wrong. When the issue happens, the register values I dumped from the firmware are: MR: 0x01 IR: 0x00 IMR: 0x00 SHAR: 00:01:19:64:01:02 GAR: 0x0000 SUBR: 0x0fff SIPR: 0xc0a81e11 RTR: 0xc350 RCR: 0x0008 TMSR: 0x0808080808080808 RMSR: 0x0808080808080808 MTYPER: 0x00ff UIPR: 0x00000000 UPORTR: 0x0000 FMTUR: 0x0000 SOCK0 MR21 CR0 SR14 SOCK1 MR21 CR0 SR14 SOCK2 MR21 CR0 SR14 SOCK3 MR21 CR0 SR14 SOCK4 MR21 CR0 SR14 SOCK5 MR21 CR0 SR14 SOCK6 MR21 CR0 SR14 SOCK7 MR21 CR0 SR14 I cannot see obvious abnormal values from above. All sockets are in LISTENING status. I also captured one situation that all sockets are in 0x17 status ESTABLISHED and the further SYN request will be responded with RST by w5300. It is reasonable. And during the test, I ever read 0x1e and 0x11 status value of the socket randomly but they are not described in the manual. What do they mean? Thanks!

sensiwood · Answer

Dear support team of Wiznet, I am still waiting for your response on my query. Our customer keeps having similar phenomenon every 2 weeks or so. Since the workaround provided by you does not work on my setup, I cannot positively reply anything to our customer yet. The situation is getting more pressured when time elapsed without progress. I hope you can understand it. I think I have provided all the details I tested and all the dumped register values when the issue happened. If you think that is not good enough for you to start working with, please let me know. I will provide more if possible. Thanks!

sensiwood · Answer

Dear support team, Thanks for your reply. I believe _SYN packet has been sent from the peer device and this SYN has also been captured by Wireshark using the port-mirroring switch, as I shared in my original message. I re-pasted below for your reference. 2018-11-01%2010_38_06-Wiznet%20packets%20capture831×238 21.9 KB And I am sure that when W5300 does not reply to new SYN packet, the existing established socket in W5300 still works. image1091×322 36.4 KB As illustrated above, you can identify that @216 packet, there is a new SYN request from 10.54.53.240 to 10.15.1.80, but 10.15.1.80 (W5300) does not reply. But one existing socket on W5300 still works because 10.15.1.80 can still reply to 10.10.28.240. About your second question about MAC address. Yes typically MAC address should be unique in one LAN. In this test, I only used this special configuration to reproduce the problem faster. In this configuration, W5300 opens and closes sockets very frequently which can produce the same phenomenon (W5300 not replying SYN request) in several minutes. I think the errata sheet you shared was helpful; I need a workable method to clear the buffer and close the socket safely. Using your suggested workaround, the socket will not change into UDP mode, so my test program is locked up here. I re-attached my original email for your reference, which also includes the dumped register value. Please support. If you need further verification or assistance from my side, please let me know. Re Ticket#2018111201000127 W5300 does not respond to SYN request from peer.zip (114.8 KB)

Eugeny · Answer

sensiwood: In this test, I only used this special configuration to reproduce the problem faster. In this setup, if routing device is intelligent, it will have to decide which route to send the packets. It may simply appear that SYN packets are sent to wrong wire. sensiwood: Wireshark using the port-mirroring switch I would do it differently. I would put PC (Win or Linux) with two LAN cards in bridge mode between switch and W5300, and set up Wireshark on it. In this setup you will be able to see PC’s both interfaces, and be sure that respecive packets are travelling in the wire up to the W5300’s magjack.

sensiwood · Answer

Eugeny: In this setup, if routing device is intelligent, it will have to decide which route to send the packets. It may simply appear that SYN packets are sent to wrong wire. sensiwood: I have to clarify that, on my setup, I don’t use router. I only used a switch. The diagram is below. image789×638 18.4 KB Let the system run about 10 minutes and then device 3 cannot respond to any SYN request from device 1 or device 2. I reckon that two devices with same MAC address on the same LAN will confuse the switch, but this is the only scenario I can reproduce the same phenomenon reported by customer. After I remove the duplicated MAC address device from the network and rebooted the switch, the device 3 cannot still reply to incoming SYN request. So I suppose W5300 run into an unexpected status. Write RST to MR1 register can restore W5300, but that will reset other working sockets on the same W5300. So I am querying if any method to prevent W5300 go into that status.

sensiwood · Answer

Eugeny: I would do it differently. I would put PC (Win or Linux) with two LAN cards in bridge mode between switch and W5300, and set up Wireshark on it. In this setup you will be able to see PC’s both interfaces, and be sure that respecive packets are travelling in the wire up to the W5300’s magjack. Thanks for sharing this good idea. I will try to figure out how to do it on my setup.

Eugeny · Answer

sensiwood: I don’t use router That does not guarantee that this switching device performs no packet filtering. sensiwood: Let the system run about 10 minutes and then device 3 cannot respond to any SYN request from device 1 or device 2. I suspect it is normal. After some time of operation intermediate network device decides on illegal devices and “disconnects” them. sensiwood: After I remove the duplicated MAC address device from the network and rebooted the switch, the device 3 cannot still reply to incoming SYN request. So I suppose W5300 run into an unexpected status. Depending on the device, rebooting it may not clear cache. I think you must manually perform commands to reinitializate the tables / clear them. Your diagram shows 3 W5300 devices, but you say that device 3 is affected. Clear everything, and swap switch ports of device 1 and device 2 - will another physical device become unresponsive?

sensiwood · Answer

I didn’t observe device 1 or 2 unresponsive because in this test, device 1 and 2 keep initiating requests to device 3. Device 3 replies the requests from device 1 and 2. When issue happened, device 1 and 2 still sent SYN request to device 3.

sensiwood · Answer

Dear support team, Thanks a lot for your reply and support. Actually I agree that duplicated MAC address within one LAN is not a correct configuration. And the duplicate MAC address will confuse the switch because the switch detects two ports have the the same MAC address connected so when it receives the packet targeting to that MAC, it does not know which port it should forward to. I totally understood the comments from you and Eugeny. However very unluckily, it is the only scenario I can reproduce the issue so far. I have uploaded the wireshark file in my original message of this topic. Please check. That is the original wireshark file captured on the real site and returned by my customer. In this capture file, 10.15.1.80 is W5300, normally it will receive the data from two IP addresses: 10.10.28.240 10.54.53.240 They are not in the same sub-net but that is OK because the routers have been configured correct to bridge and forward the packets correctly. When the problem happened, 10.15.1.80 cannot reply SYN request from 10.54.53.240 anymore. The only way to restore the communication is, rebooting our device which will reboot W5300 from hardware level as well. This is the origin the whole story and issue. It is not possible for me to set up the same network as our customer, so I used a simpler system to reproduce the same phenomenon faster (I mean using that duplicate MAC scenario). I will double check if W5300 can get back when I switch off the switch and wait enough long time. As well I will check my coding issues as you suggested. I appreciate you pointed that out.

Eugeny · Answer

sensiwood: However very unluckily, it is the only scenario I can reproduce the issue so far. Symptoms may look like the same as at your customer site, but the cause might be different. sensiwood: The only way to restore the communication is, rebooting our device which will reboot W5300 from hardware level as well. Try simply disconnecting and reconnecting network jack from W5300 for a minute - causing switch’s port going down and then up.

sensiwood · Answer

Hi, I have attached a diagram of the network we were talking about. You can find that 10.15.1.80 is the master device which sits in the control center. 10.54.53.240 is a port of WAUCHOPE, 10.10.28.240 is a port of TELEGRAPH. Both of them are remote stations and connected to the master via public network. There is one router, BMD_SIGS_RTR1, as the interface between the control room and WAN. You can see from the wireshark capture file that, 10.54.53.240 and 10.10.28.240 share the same MAC but I think that is correct because they go through the same router so that MAC address should be of the LAN port of the router. That LAN port of router is connected to the switch in the control room thus connected to the master device. 2018-12-14%2015_30_09-DWG-N-NCA-GEN-NCA5_KF_Communications1472×937 160 KB All these devices use W5300. It is not easy to capture the data exactly when the event happens. The phenomenon occurs every couple of weeks and the customer cannot set up Wireshark for that long period of time. But I will try to find opportunities. And I confirmed again on my setup that, after the issue happened, W5300 cannot be restored automatically. I removed the devices with duplicate MAC, waited at least 24 hours and even factory reset my switch. W5300 still refuses replying SYN request, until I restarted the system.

sensiwood · Answer

midnightcow: As you mentioned, You didn’t use the same mac address in real world, did use for just test. But your diagram is not. In your environment, It is possible that the communication doesn’t work well. Sorry, I cannot understand well about your comment. My customer’ setup can be simplified as below. image665×529 16.4 KB Remote device 1 and device 2 are reaching device 3 via the public network. One Cisco router acts as the gateway between the local network and the public network. So on the local network (where device 3 is), device 1 and device 2 both use the MAC address (00:24:97:8a:7e:58) of the LAN Port of the router because that is the only access point to the local network. For switch’s ARP table, this MAC address will always be mapped to its port 1. It is a very typical scenario I think, and I will not (cannot) change that. Just think about one example that you configured two IP addresses on the single network interface of your computer, which is connected to the local network via a switch. So two IP addresses share the same MAC address. It is a correct configuration. MAC is only for LAN. I hope my explanation makes sense for you. This is different to my own test setup. On my own test setup, the switch has duplicated MAC address on two ports of it. I have reckoned this is not the typical scenario, and I only used to reproduce the issue. I am not sure if you can set up the same as I did, reproduce the issue and have a look what status the chip has run into and how to prevent it if possible. Sorry I cannot test further of code change due to last busy week. I will let you know when I got progress. Thanks for your support.

sensiwood · Answer

Thanks for capturing a typo of IP address in the diagram. Now it is updated as below. image1472×937 160 KB . What you understood is basically correct. 10.15.1.80 is one Ethernet port of MASTER RTU 15. All the network interfaces behind each IP address of RTU devices (RTU 15, 125, 127, 128, 129) are W5300. I appreciate your suggestion about network bridging. Firstly I cannot do it on customer’s site unfortunately; actually it is not easy to do anything on customer site at the moment. On my setup, I can do it; but maybe at a later time. I am pretty sure that SYN request has sent to W5300, and W5300 did not reply it. Whether the SYN request is not received by W5300 or not processed, I don’t know. That is why I posted the question here. I ever mailed the dumped register values when I reproduced it on my own setup. I attached them again here. It shows when W5300 did not answer SYN request, all the sockets were free.dumped W5300 registers in error situation.zip (491 Bytes)

sensiwood · Answer

Hello midnightcow, Happy new year to you! I am just back to office from the holiday and will look into/try your suggestions in following days. Thanks for your support!

sensiwood · Answer

Hi team, Sorry for coming back so late on this thread. Actually I did some work as my background task in the past month. Now based on your suggestions, I think I have got a progress to bypass the issue. Two key changes I have made were: Before closing the socket, always check if the send buffer has been emptied. Below are the functions I used. static void closeSocket(CPU_PORT port, int s) { T_SOCK *sock = GET_SOCK(port, s); int closing = waitForTxDone(port, s); if(closing == FAIL) applyErratum1(port, s); sock->timeWaitTimeout = 0; sock->sourcePort = 0; sock->remoteRtuAddress = 0; } static int waitForTxDone(CPU_PORT port, int s) { T_SOCK *sock = GET_SOCK(port, s); USHORT loop; if(sock->txDone) return OK; loop = 1000; // approx 50 ms. while((w5300IndirectReadWord(port, Sn_IR(s)) & Sn_IR_SEND_OK) == 0) { if(--loop == 0) return FAIL; } w5300IndirectWriteWord(port, Sn_IR(s), Sn_IR_SEND_OK); return OK; } That means, every time when I close the socket, I will check if the last sending operation has been done. If the sending activity is not finished within certain period of time (50ms), I will use following function to purge the output buffer and close the socket forcefully. void applyErratum1(CPU_PORT port, int s) { BYTE dummyData[2] = { 0, 0 }; union IPV4_ADDR ipAddr; ipAddr.addrVal = 1; w5300_open_socket_mode(port, s, MIN_PORT_NUM - 1, Sn_MR_UDP); w5300_set_dipaddr(port, s, ipAddr, TCP_KINGFISHER); w5300_write_tx_data(port, s, dummyData, 1); closeSocketWithoutWait(port, s); } After changing the socket type to UDP, I do not wait the socket status to be SOCK_UDP in a dead loop; instead I just wait a period of time and then directly start to send the rubbish bytes in output buffer to a void target IP address. As tested on my own setup, the new firmware doe not have the permanent socket lock up issue any more. Sometimes, within certain period of time, one socket still cannot respond to SYN request as I observed. But the situation can be restored automatically. Anyway, the…