--- title: "TCP stalls when socket is spammed" url: "https://maker.wiznet.io/forum/13117" markdown_url: "https://maker.wiznet.io/forum/13117/md" type: "Forum topic" category: "Ethernet Chips" author: "mzimmers" created: "2018-11-16T18:35:43+09:00" last_activity: "2018-12-10T19:38:10+09:00" language: "en" views: 1704 replies: 32 source: "WIZnet Makers forum (https://maker.wiznet.io/forum/)" --- # TCP stalls when socket is spammed ## Question Asked by mzimmers on 2018-11-16 in Ethernet Chips. Hi all - My company uses the W5200 in a product that has been deployed for years. A recent prospective customer, while evaluating our product, discovered that the TCP handler of the W5200 could be made to stall (without recovery) by inundating it with nonsensical messages. The test scenario was simply sending packets with a payload of a single character (the letter “A” in their case) as fast as possible. As these packets were being received, I could observe through Wireshark that the TCP window was shrinking. It soon became zero, and never recovered. Repeating this test with as little as 1ms between packets prevented this issue. Is this a known problem? Is there any available workaround? Would one of the newer WizNet devices resolve this problem? I can provide a Wireshark trace file if desired. Thank you. ## Replies ### Reply 1 by Eugeny, 2018-11-19 Please look at what is going on at the microcontroller interface. ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > As these packets were being received, I could observe through Wireshark that the TCP window was shrinking. It soon became zero, and never recovered. It can happen only if MCU does not “extract” data from the RX buffers of W5200. That’s why it is important to see what MCU is doing when receiving these ‘A’ messages. It may be failure of W5200 driver, and not W5200 TCP/IP stack. ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > the W5200 could be made to stall (without recovery) by inundating it with nonsensical messages. Information encapsulated into TCP packet will have a format anyway, understandable to the application running in the device. What device is expected to do when it receives ‘A’? What does it actually do? ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > Repeating this test with as little as 1ms between packets prevented this issue. If you increase the deay between packets, does the window size starts increasing? Make a test with variable delay. ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > I can provide a Wireshark trace file if desired. You explanation is clear - W5200 always reports window size 0, and this is what we will see in the log. So the log is not much of value until you figure out how device treats ‘A’ and how fast if flushes data from its RX buffers. ### Reply 2 by mzimmers, 2018-11-19 The MCU is a legacy 8-bit processor (AVR ATMega1284P). Its resources, both compute and memory, are quite limited. The original programmer, as an effort to save space, designed the application to leave data in the WizNet buffer until a complete message has been received, or a timer elapses. The incoming messages are encrypted, so this adds to the difficulty of knowing when a complete message has been received. The 1ms delay results in the TCP window remaining within ~20 bytes of it’s 2048 capacity. I don’t think there’s much point to testing with further delays. ### Reply 3 by Eugeny, 2018-11-19 (in reply to reply 2) So it is clearly the design flaw, right? It is just not designed for the conditions you test in. Application/MCU must free RX buffer in time for W5200 being able to receive data. ### Reply 4 by mzimmers, 2018-11-19 How do I free the buffer? It’s not clear from the datasheet. The current code does this: void HOSRxFlush(void) { HosRxPut = HosRxGet = 0; // Rx buffer pointers } Which seems to be updating pointers, but not actually freeing any memory. ### Reply 5 by Eugeny, 2018-11-19 (in reply to reply 4) ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > How do I free the buffer? W5100’s datasheet chapter 5 has full information on what should be done on the RX and TX buffers and their pointers. You copy data, and then update pointer(s). If you do not need data you can just update pointer to the proper value - this is the fastest way to “clear” the buffer, but data usually matters ![:slight_smile:](https://emoji.discourse-cdn.com/twitter/slight_smile.png?v=12) ### Reply 6 by mzimmers, 2018-11-19 Eugeny: > W5100’s datasheet chapter 5 Presumably about the same as the W5200’s datasheet? (I’m using the W5200.) So, it appears that my MCU isn’t fast enough to properly service the incoming data, and I should be prepared for the eventuality of a ZeroWindow problem. Can I diagnose this from the MCU and reset the socket? (This would be a bit help if possible.) ### Reply 7 by Eugeny, 2018-11-19 ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > Presumably about the same as the W5200’s datasheet? (I’m using the W5200.) I think so. Some datasheet misses valuable information (e.g. W5500 has a tiny piece of what is explained in W5100’s chapter 5). ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > I should be prepared for the eventuality of a ZeroWindow problem It depends on the application of your design. If incoming data is not expected to be like a DDoS you can live with it easily (given there’s none on the network starting behaving this way). ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > Can I diagnose this from the MCU and reset the socket? Not sure I understand. You want to discard data? Then you just update pointer to current position, this way “freeing” the buffer. No need to reset socket or even whole device. You can consider alternative solutions - depending on MCU models, there could be pin-compatible (or even code compatible) replacements with higher processing speed. I know Microchips are good at making such product lines of slower - faster -fastest devices which can be put into the same socket and require minor changes to the code when porting the applications. Also depends on how you service incoming data - by polling WIZnet chip registers or using interrupts. ### Reply 8 by mzimmers, 2018-11-19 Thanks for the good information, Eugeny. What I meant in my earlier post is, I think that – as our product is currently built – we need to address the issue of the TCP Window filling up, because the MCU can’t remove data from the W5200 as quickly as the W5200 can receive it from the network. When the buffer is full, the W5200 issues a ZeroWindow message. This will go on forever until the system is rebooted. My question is, is there an intermediate way to clear the buffer? The routine I posted above doesn’t do it. [![trace](https://maker.wiznet.io/forum_uploads/wiznet/original/2X/f/ff9b28f65ba8924bd49f84058e2df1856592147c.png) trace1172×396 15.9 KB](https://maker.wiznet.io/forum_uploads/wiznet/original/2X/f/ff9b28f65ba8924bd49f84058e2df1856592147c.png) We as a company are willing to look at replacing our MCU (and upgrading from the W5200 to the W5500 if need be), but I want to know what we can do with our current configuration. ### Reply 9 by Eugeny, 2018-11-19 (in reply to reply 8) ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > My question is, is there an intermediate way to clear the buffer? You must operate RX buffer pointers, and not forget about RECV command. Picture you posted shows that MCU does not “remove” any data, not touching RX pointers at all and/or not performing RECV command after it. It is not possible to advise as I do not know what your application is doing and how it can tolerate this “data loss” (when data is just skipped). There’re a number of ways to consider, e.g. waiting for specific data size in the buffer and only then reading it, bit their fit to your environment is very applications-dependent. ### Reply 10 by mzimmers, 2018-11-20 Hi Eugeny - Yes, you’re correct that the application doesn’t remove any data upon timeout. This may be a bug in our app, and would explain the current behavior. Right now, I’m prepared to live with losing data, as long as I can prevent this TCP stall. I’ve read section 5.2.1.1 in the data sheet, and I believe what I need to do is reset the pointers Sn_RX_RD and Sn_RX_WR for the socket that is giving me trouble. Can I clear these pointers directly, or do I need to execute a RECV command? Thanks for all the assistance. ### Reply 11 by Eugeny, 2018-11-20 (in reply to reply 10) You must issue RECV command after changing pointer. ### Reply 12 by mzimmers, 2018-11-20 So, write to the socket registers in this order: W5200RegWrite(0x4328, 0x0); W5200RegWrite(0x4329, 0x0); W5200RegWrite(0x432a, 0x0); W5200RegWrite(0x432b, 0x0); W5200RegWrite(0x4301, 0x40); // the RECV command Is that correct? Do I need to do something else to commit the command? Thanks. ### Reply 13 by Eugeny, 2018-11-21 ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > Is that correct? I do not think so. Why you write zeros? Which registers you write to? What must you write in these registers? ### Reply 14 by mzimmers, 2018-11-21 0x4328 is Sn_RX_RD0 0x4329 is Sn_RX_RD1 0x432a is S1_RX_WR0 0x432b is Sn_RX_WR1 0x4301 is Sn_CR; the 0x40 is the RECV command. If I don’t change the pointers to 0, what should I change them to, in order to flush the buffer? ### Reply 15 by Eugeny, 2018-11-21 Datasheet is pretty clear on it on page 50: > Sn_RX_RD += len; where > len = Sn_RX_RSR; What this does: you shift RD pointer past the received data size of RSR. Then perform RECV command which will mark skipped buffer area as free for new incoming data. If you assign 0 to pointers it will break up TCP sequence and most probably prevent further communication performing correctly due to improper window indication and calculation. ### Reply 16 by mzimmers, 2018-11-21 Oh…so: ``` u8 uc0; u8 uc1; W5200RegRead(Sn_RX_RSR0, &uc0); W5200RegRead(Sn_RX_RSR1, &uc1); W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); W5200RegWrite(Sn_CR, RECV); ``` Is that better? Thanks, Eugeny. EDIT: oops, no that’s not right…hang on a minute, please…I’ll correct it. ### Reply 17 by mzimmers, 2018-11-21 Here’s another go at it: ``` u8 uc0; u8 uc1; u16 rsr; u16 rd; W5200RegRead(Sn_RX_RSR0, &uc0); W5200RegRead(Sn_RX_RSR1, &uc1); rsr = (uc1 << 8) | uc0; W5200RegRead(Sn_RX_RD0, &uc0); W5200RegRead(Sn_RX_RD1, &uc1); rd = (uc1 << 8) | uc0; rd += rsr; uc0 = rd & 0xff; uc1 = rd >> 8; W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); W5200RegWrite(Sn_CR, RECV); rc = recv(SockTCPL, HosRxDat.bad, &len, &(HosRxIp.ul), &port); ``` Does this look correct now? ### Reply 18 by Eugeny, 2018-11-21 Read octet pairs into 16-bit unsigned variables (RSR and RD), then sum them up, and write resulting 16-bit value by its octets into RD. So second try seems to be the correct one ![:slight_smile:](https://emoji.discourse-cdn.com/twitter/slight_smile.png?v=12) I am sure you will see result immediately when running it. After you perform RECV wait until CR register clears to 0 before going further. > rc = recv(SockTCPL, HosRxDat.bad, &len, &(HosRxIp.ul), &port); What this one does? Is it that you instruct to read data from the socket? Then there’s no sense in previous code. Or I would say this is redundant call as you have just cleared the buffer. ### Reply 19 by mzimmers, 2018-11-21 Well, this looks hopeful. I may have solved the stall problem. I am now seeing, however, tons of TCP Dup ACK messages from my device: [![dupack](https://maker.wiznet.io/forum_uploads/wiznet/original/2X/4/4796683f860a29ec2dfc82e4ed5f8460dd21ee86.png) dupack1213×460 12.3 KB](https://maker.wiznet.io/forum_uploads/wiznet/original/2X/4/4796683f860a29ec2dfc82e4ed5f8460dd21ee86.png) Any idea what could be causing this? ### Reply 20 by Eugeny, 2018-11-21 Did you remove > rc = recv(SockTCPL, HosRxDat.bad, &len, &(HosRxIp.ul), &port); from the code? It may cause 0 size to be “received”, and thus dup ack packet sent. And by the way, what socket number command > W5200RegRead(Sn_RX_RSR0, &uc0); assumes? ### Reply 21 by mzimmers, 2018-11-21 No, I didn’t remove the recv() call…I thought it was needed to effect the change in the pointers. Sn_RX_RSR0 is defined with the socket number included in it (0x4326). The read just goes to an address; it doesn’t know about sockets. Socket 3 is the only one that needs this special attention. ### Reply 22 by Eugeny, 2018-11-21 Ok, remove `recv()` and try. Note that what we do now is a kind of “low level hacking”, ideally you must use `recv()` call which moves data (and which you want to avoid under your low performance circumstances). Do not forget to put comments into your code, as you definitely forget everything in a week or two. ### Reply 23 by mzimmers, 2018-11-23 Can’t get it working. Here’s the clear routine: ``` void W5200ClearRxBuffer() { __monitor char W5200RegRead(u16 adr); __monitor void W5200RegWrite(u16 adr, char data); const u16 Sn_MR = 0x4300; const u16 Sn_RX_RSR0 = 0x4326; const u16 Sn_RX_RSR1 = 0x4327; const u16 Sn_RX_RD0 = 0x4328; const u16 Sn_RX_RD1 = 0x4329; const u16 Sn_CR = 0x4301; const u8 RECV = 0x40; u16 len = APGmPktZ; u16 port = LANCLibPO; u8 uc0; u8 uc1; u16 rsr; u16 rd; // 2018 11 21 mzimmers: // temporarily read Sn_MR to see whether // no delayed ACK is set in bit 5. // note: it is. we might want to change this. uc0 = W5200RegRead(Sn_MR); uc0 = W5200RegRead(Sn_RX_RSR0); uc1 = W5200RegRead(Sn_RX_RSR1); rsr = (uc1 << 8) | uc0; uc0 = W5200RegRead(Sn_RX_RD0); uc1 = W5200RegRead(Sn_RX_RD1); rd = (uc1 << 8) | uc0; rd += rsr; uc0 = rd & 0xff; uc1 = rd >> 8; W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); // it isn't clear yet whether the lines below are necessary. // commenting them out for now. //W5200RegWrite(Sn_CR, RECV); //recv(SockTCPL, HosRxDat.bad, &len, &(HosRxIp.ul), &port); ``` } ### Reply 24 by Eugeny, 2018-11-23 (in reply to reply 23) ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > Can’t get it working. How it does not work? You must give details. ~~Code looks ok.~~ Sn_MR read is spare at the beginning (as your comment states), and I guess there’s no problem with `rd` overflow after addition (u16 type, MSB will be discarded). You are using socket #3, right? **Edit: no, code is not ok!** Manual says: > When reading this register, user should read upper byte(0x4026, 0x4126, 0x4226, 0x4326, 0x4426, 0x4526, 0x4626, 0x4726) first, and lower byte(0x4027, 0x4127, 0x4227, 0x4327, 0x4427, 0x4527, 0x4627, 0x4727) later to get the correct value. Note that upper byte of RSR will be in `uc0`, and lower byte will be in `uc1`. Thus you must have ``` rsr = (uc0 << 8) | uc1; ``` and same for RD. ### Reply 25 by mzimmers, 2018-11-27 Hi Eugeny - 1. By “not working” I meant that the stated problem persists: as the flood of incoming TCP messages rises, the TCP window steadily shrinks to 0, at which point the socket is unresponsive. 2. I made the changes you suggested, tried this with and without a final recv() call, and even eliminated the direct buffer manipulation in favor of a lone recv() call. Behavior is unchanged. I’m beginning to think the problem is simply that the W5200 is so fast, and the CPU so slow by comparison, that the W5200 is just “outrunning” the CPU. ``` __monitor void W5200ClearRxBuffer() { const u16 Sn_RX_RSR0 = 0x4326; const u16 Sn_RX_RSR1 = 0x4327; const u16 Sn_RX_RD0 = 0x4328; const u16 Sn_RX_RD1 = 0x4329; const u16 Sn_CR = 0x4301; const u8 RECV = 0x40; const u16 len = APGmPktZ; const u16 port = LANCLibPO; __monitor char W5200RegRead(u16 adr); __monitor void W5200RegWrite(u16 adr, char data); u8 uc0; u8 uc1; u16 rsr; u16 rd; uc0 = W5200RegRead(Sn_RX_RSR0); uc1 = W5200RegRead(Sn_RX_RSR1); rsr = (uc0 << 8) | uc1; uc0 = W5200RegRead(Sn_RX_RD0); uc1 = W5200RegRead(Sn_RX_RD1); rd = (uc0 << 8) | uc1; rd += rsr; uc0 = rd >> 8; uc1 = rd & 0xff; W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); W5200RegWrite(Sn_CR, RECV); recv(SockTCPL, HosRxDat.bad, (u16 *) &len, &(HosRxIp.ul), (u16 *) &port); ``` } ### Reply 26 by mzimmers, 2018-12-03 I’m still trying to get this solved. I’d like to begin by merely discarding any data in the buffer. To do this, is it sufficient to set Sn_RX_RD = Sn_RX_WR, and then write a RECV into Sn_CR? Or is something else necessary? ### Reply 27 by Eugeny, 2018-12-04 (in reply to reply 26) You do things properly except you must wait for Sn_CR to become zero after you write RECV command into it. You actually must wait Sn_CR to clear after any command write to it. As soon as you can control “spamming” process stop it at some time and see if buffer will get freed. ### Reply 28 by mzimmers, 2018-12-07 I still can’t update the pointers with the RECV command. Here’s my code: ``` uc0 = W5200RegRead(Sn_RX_RD0); uc1 = W5200RegRead(Sn_RX_RD1); rd = (uc0 << 8) | uc1; uc0 = W5200RegRead(Sn_RX_WR0); uc1 = W5200RegRead(Sn_RX_WR1); wr = (uc0 << 8) | uc1; rd += rsr; uc0 = rd >> 8; uc1 = rd & 0xff; W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); W5200RegWrite(Sn_CR, Sn_CMD_RECV); do { uc0 = W5200RegRead(Sn_CR); // read command register } while (uc0 != Sn_CR_DONE); // until command complete // for debugging only; just to look at the registers. uc0 = W5200RegRead(Sn_RX_RSR0); uc1 = W5200RegRead(Sn_RX_RSR1); rsr = (uc0 << 8) | uc1; uc0 = W5200RegRead(Sn_RX_RD0); uc1 = W5200RegRead(Sn_RX_RD1); rd = (uc0 << 8) | uc1; uc0 = W5200RegRead(Sn_RX_WR0); uc1 = W5200RegRead(Sn_RX_WR1); wr = (uc0 << 8) | uc1; ``` rd and wr do not change. By the same token, rsr always seems to be 0. What might I be doing wrong? Thanks. ### Reply 29 by Eugeny, 2018-12-10 I assume there’s RSR read before the whole code you posted? Of course **after** you perform RECV register RSR will contain 0, until socket receives some new data. Or you say that RSR is zero before you perform RECV (the RSR read is not in the code you posted)? ### Reply 30 by mzimmers, 2018-12-10 Hi Eugeny - 1. Yes, there’s an RSR read prior to the code I posted. I occasionally have issues inserting code into my posts here. 2. The RSR register is zero upon entry to this routine, but I think that’s a matter of how/where I call this routine, so it’s not a W5200 issue. 3. I’m not sure I fully understand the behavior of the W5200. After performing a RECV, should I expect the RD/WR pointers to change? Or is only the RSR register altered? ### Reply 31 by Eugeny, 2018-12-10 ![](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/48.png) mzimmers: > I’m not sure I fully understand the behavior of the W5200. After performing a RECV, should I expect the RD/WR pointers to change? Or is only the RSR register altered? There’re three locations within the chip: RD: location of the “user” pointer - the data which application must read; WR: system pointer, the location where the chip’s network stack writes when data arrives; RSR: it is simply WR-RD, and it can not be larger than RX buffer size. Note that all registers are 16-bit, and must be treated as 16-bit values even if buffer size is 8KB. Thus you do not touch WR under normal circumstances, it is managed by the chip. You only look at the RSR - amount of data received into the RX buffer, and at RD - the location to start reading data from. Thus if you read RSR, and it appears 0 - this means that buffer is empty, there’s no data to read, and chip must respond with full window size free in its network packets. Logically, if RSR is not 0, then RD is not equal to WR. Maximal value of RSR is RX buffer size. When you write to RD pointer new value, and then issue RECV command, chip assumes that data between old RD location and new RD location is flushed by the application, and considers this space as available for new incoming data. Therefore reading RSR, then assigning RD=RD+RSR and then RECV “frees” RSR bytes in the RX buffer, and decreases current value of RSR by RSR when it was previously read. Note that while you perform reading of RSR, then RD, then performing RD+RSR and assigning it back to RD time has passed, and actual RSR register value may increase. But it does not matter because RSR you originally read will always be <= current RSR, and there will be no buffer overrun, next time you read RSR you will get “remaining” value of RSR. Then, very important note: when reading any 16-bit register (RD or RSR or anything else) you **must** read byte at +0 first, and byte at +1 second. Not critical for RD because it will not change dynamically without your RECV instruction, but very critical for RSR: when reading +0 first and +1 second you will be guaranteed to get value equal or less than current, and will never overrun the buffer. If you read other way around, you may get value bigger than current RSR, shifting RD ahead of WR with RECV command, and it will mess whole communication. Example: RSR=0x01FF Right order: you read 0x01 first, then RSR changes to 0x0200 (one more byte was received), and you read 0x00. Thus you have RSR=0x100, which, while is not “immediately correct”, should give you amount of bytes less than in the buffer, and on next read RSR will read as 0x100 and finally you will read all the data from the RX buffer. Wrong order: you read 0xFF first, then RSR changes to 0x0200, and you read 0x02 for MSB. So you end up with RSR=0x02FF, and crash the socket communication. I hope it is clearer now, and you can find at least the area for further investigation yourself - as I do not fully understand what is going within your design. You clearly do something incorrectly, but hard to say what it is exactly. ### Reply 32 by mzimmers, 2018-12-10 Thanks for the explanation. I believe I have it working now. Here’s my modified code: ``` rsr = getW5200Reg16(Sn_RX_RSR0); rd = getW5200Reg16(Sn_RX_RD0); wr = getW5200Reg16(Sn_RX_WR0); while (rsr > 0) { rd += rsr; uc0 = rd >> 8; uc1 = rd & 0xff; W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); W5200RegWrite(Sn_CR, Sn_CMD_RECV); do { uc0 = W5200RegRead(Sn_CR); // read command register } while (uc0 != Sn_CR_DONE); // until command complete rsr = getW5200Reg16(Sn_RX_RSR0); } ``` This seems to be effective at minimizing the impact of TCP-level DoS attacks that consist of nonsensical messages. Thanks for all the help with this. --- Source: https://maker.wiznet.io/forum/13117