W5300 SENDOK and Reserved Status Byte

Question

I’m having a recurring issue sending data with the W5300. I have 5 test units, and each unit is using a W5300 and sending data at a very high speed, about 80Mbits/second total, mostly evenly divided among 4 sockets. They ran for hours, and then eventually they all failed at different times. The failure mode is that I am waiting for the SENDOK bit to be set in Sn_IR, and it never gets set for one of the sockets. The Sn_SSR remains good, 0x17 socket established. I do notice that the Reserved byte of the status register (for example 0x288 for socket 2) is different. For the sockets that are working, I see 0x01 in this, but for the frozen socket, I have seen either 0x89 or 0x8B). Is that reserved register giving me some useful status information? Can I use it to know that something has happened so I shouldn’t expect the SENDOK bit to get set? Can I recover from this state? In general, what would cause the SENDOK bit to never get set if the connection is still established? The client is not receiving any data, but reports no errors. Thank you. Update: I tried to see if the status goes through any temporary conditions during a normal write sequence, and found the following: After writing data to the TX FIFO: Sn_SSR (including the reserved byte) was one of 0x07A0, 0x07C0, 0x07E0, 0x0800 After writing Sn_TX_WRSR: Sn_SSR is back to normal 0x0117 After writing SEND command: Sn_SSR remains normal 0x0117 I have not been able to duplicate seeing 0x8917 or 0x8B17 in the reserved+SSR bytes in any normal operation; this only seems to occur when the freeze up happens. By the way, I have the keepalive set for 5 seconds, but I haven’t checked during the freeze up if the keepalive messages are going through. I’ll try to duplicate it and check with Wireshark. Update 2: I was able to duplicate the failure with 3 units running overnight. For the “frozen” units, I used Wireshark to analyze any traffic and found the following: The W5300 is trying to send a re-transmission over and over with a sequence number near the 32-bit rollover, and the ACK response has a non-matching sequence number. For example, the retransmission is 1460 bytes with sequence 0xFFFFFA70, and the ACK responsds with sequence 0x000000DC, which is the original sequence offset by 1644 instead of 1460. The other two units failed in similar fashion: one retransmitting 1460 bytes with sequence #0xFFFFFAD8, and the ACK coming back as 0x000001C4 (off by 1772 instead of 1460) one retransmitting 1460 bytes with sequence #0x0000088A, and the ACK coming back as 0x00000FF6 (off by 1900 instead of 1460) I looked at some of the normal data stream before the freeze up, and it looks like the packets were most often 1460 bytes, with some occasional smaller packets of sizes: 340, 416, 1028, 1112, 1368, 1452. It also seems common for the W5300 to send multiple packets at high speed and only get a single ACK. I was wondering if it’s possible that the W5300 is sending multiple packets around the time the sequence rolls…

abes · Answer

I don’t know how often re-transmits occur, but the data flows very well most of the time. The peer system is sending ACKs back. Are you saying that the ACK is not what the W5300 expects, so there is a failure to recognize the ACK? I have attached a Wireshark capture file that shows IP addresses 192.168.1.52 and 192.168.1.55 continually retransmitting. The peer is 192.168.1.79, and it keeps sending the same ACK. This is the state when I am waiting for the SEND_OK bit to be set. This is two separate units that froze up in this state. I already do check for the SEND_OK flag after every send command, and I also clear it every time it gets set. For sending a lot of data, I have large TX buffers of 27kBytes for each socket, and I fill them up whenever data is available. There may be less than 1460 bytes, and there may be more than 1460 bytes. If the W5300 already breaks up large amount of data into MTU sized packets, how will it help for me to do that manually? Even if this is rare, we will be sending large amounts of data in bursts, and this has happened many times. Every time I found it looks like the TCP sequence number is near rollover. If you can confirm that it has something to do with the TCP sequence number rolling over, maybe we can come up with a workaround. capture with 52_1 and 55_3 frozen.zip (44.9 KB)

abes · Answer

Yes, I noticed that the ACK numbers didn’t match up. There were many occasions when ACKs were sent out of order, or for multiple packets, so I assumed that is normal. But I have only seen the glitch when the sequence number is near 0xFFFFFFFF. Unfortunately, there is too much data to collect it all. Maybe I’ll find a way to have Wireshark filter or just save header information and then I can catch the packets right before the failure. I’ve made the recommended change to only send 1460 bytes at a time, and I’ll see how it goes. Thank you.

abes · Answer

I made the change to only send 1460 bytes at a time, and it did not fix the problem. I have 4 systems that ran overnight, and each one has 4 sockets open. So far, one of the sockets on one of the systems froze up. Again, the TCP sequence number is right near the 32-bit rollover: W5300 transmitted sequence number: FFFFFC6F peer ack number: 00000287 Also, I closed the peer program, and the frozen socket has status SSR of 1D, SOCK_LAST_ACK for many minutes. I have not changed the default values of RTR and RCR, so I would expect it to time out and close after some short time.

abes · Answer

I figured out how to capture just the sequence numbers near the rollover point with Wireshark, and I was able to capture one of the failure modes. The attached file shows the problem starting around packet number 5870. Event 5870: The PC sends an ACK with the ack sequence number in the middle of a packet. Event 5871: The W5300 sends another packet of data. Event 5872: The W5300 send a FIN flag. After that, the PC sends a FIN and tries to reconnect, but the W5300 is still trying to do something and it gets messy. The peer is a PC running Windows 7 and Labview. I don’t know how the buffers are configured, but I notice that sometimes the window size shrinks by more than the amount of data sent on a particular socket, so I think Labview or Windows is sharing some buffer space somehow, and the window size is not really available due to other traffic. We will need to see if we can allocate separate buffers for each port. Could this be the source of the problem? Should the W5300 be able to handle an ACK that only accepted part of a packet? Is that even acceptable for TCP? I have not seen another occurrence where the PC ack sequence is only accepting part of a message, but it may be happening. I have only seen the freezing behavior associated with the time the sequence number is rolling over, so I’m guessing there is some correlation. w5300_fin1.zip (903 KB)

abes · Answer

Regarding the 3-way handshake, I think it does that fine, but my Wireshark capture had a filter on it, so it didn’t show all of the packets. I don’t think it’s related to the selective ACK. When I capture everything without a filter, I see the correct 3-way handshake. I believe I may have found an issue that would cause the “broken ack number”. During the initial connection, the PC sends SYN with an option of window scaling = 2 (multiply by 4) The W5300 SYN only sends Maximum Segment Size, it does not send window scaling option. When the error occurred, the PC reported window size of 552, but it looks like the W5300 interpreted it as 552*4=2208, and sent a full packet of 1460 bytes. The PC then only acknowledged 552 bytes because that’s all it had room for. At this point, the W5300 seems to not be able to handle the incorrect ACK number and everything falls apart. According to someone who wrote this article: en.wikipedia.org/wiki/Transmiss … ow_scaling “Both sides must send the option in their SYN segments to enable window scaling in either direction” Since the W5300 did not send the window scaling option, I’m thinking it should ignore the window scale sent by the PC. Is it possible that the W5300 is using the scaling factor sent by the PC when it shouldn’t be? I will try to see if I can modify the client to use a scale factor of 0, and see if that might resolve the issue. Edit…I added an attachment of a second capture of a failure. You can see at event 69, the PC acks with window size of 2112, and then the W5300 sends 2 more packets of 1460 bytes each. Shortly after that, it all falls apart. I also added a capture of the normal startup showing the 3-way handshake where the PC sends the window scaling option of 2, and the W5300 doesn’t send the window scaling option. Also, I’m not ignoring your request to see the code, but it is written in Verilog and it not very easy to compare to a standard C program. So for now, I’m focusing on the strange behavior that has…

abes · Answer

Thank you for the detailed information about the Sn_SSR MSB. I already added code to try to disconnect when the Sn_SSR MSB came up with the 7th bit set, and that is why we see the FIN packet in the data captures, but I still don’t seem to get a clean disconnect. It seems to be in an unusual state, and unable to recover. I understand that the W5300 doesn’t handle the un-expected ACK number. However, the problem starts when the W500 sends more data than the window size, so I think there is an error that has something to do with the window size and the sequence number rollover that causes the W5300 to send too much data. This only happens right at the rollover and only if the receive window is very small due to a buffer getting close to full, so my best guess is some math operation that doesn’t handle the 32-bit overflow properly. It would be nice to see some more in depth research and information about this behavior if possible. In the meantime, I will try to find a workaround that ensures that there will always be buffer space so the window will never go down below 1460. I will probably have to throttle the data somehow, or ensure the PC/Labview program has high enough priority to read all the data without much buffering. Thanks for all your feedback on this so far.

abes · Answer

For the sockets that are failing: TMSR is set to 27kBytes per socket RMSR is set to 2kBytes per socket; I think this is what you are seeing in the capture file where Win=2046, but these sockets don’t receive data, they just send data Yes, I check TX_FSR before sending. But this only verifies that there is buffer space in the W5300 TX Buffer (27kBytes to start). It does not indicate the peer Window size. I think there is no way for me to know the peer window size. The W5300 uses it internally but doesn’t give me access to it. Your statement that “the next 1460 data packet can’t be sent without ack packet of 1st data packet” is not consistent with the answer in this forum topic: Is W5300 capable of back-to-back transmission? W5300 Hi, You misunderstand about ‘SENDOK’ When W5300 performed send command completely internaly, SENDOK is ‘1’. Not receiving ACK packet. Packet 1 SEND SENDOK == 1 Packet 2 SEND SENDOK == 1 ACK packet of packet 1 is received Sn_TX_FSR is increased about packet 1 size. ACK packet of packet 2 is received Sn_TX_FSR is increased about packet 2 size. Best regards. After the first transmission, I wait for the SENDOK flag to be set, then I clear it, then I check TX_FSR to verify that there is space in the TX buffer, and if there is space, then I send another packet. According to that post, when the ACK is received, the TX_FSR increases by the amount of data that was acknowledged by the peer, but I should not have to wait for an ACK after every packet. If I did that, I would wait for TX_FSR to show 27kBytes before I send each packet, and my data rate will go down. I’ve tried that.