TCP stalls when socket is spammed

Question

Hi all - My company uses the W5200 in a product that has been deployed for years. A recent prospective customer, while evaluating our product, discovered that the TCP handler of the W5200 could be made to stall (without recovery) by inundating it with nonsensical messages. The test scenario was simply sending packets with a payload of a single character (the letter “A” in their case) as fast as possible. As these packets were being received, I could observe through Wireshark that the TCP window was shrinking. It soon became zero, and never recovered. Repeating this test with as little as 1ms between packets prevented this issue. Is this a known problem? Is there any available workaround? Would one of the newer WizNet devices resolve this problem? I can provide a Wireshark trace file if desired. Thank you.

Eugeny · Answer

Please look at what is going on at the microcontroller interface. mzimmers: As these packets were being received, I could observe through Wireshark that the TCP window was shrinking. It soon became zero, and never recovered. It can happen only if MCU does not “extract” data from the RX buffers of W5200. That’s why it is important to see what MCU is doing when receiving these ‘A’ messages. It may be failure of W5200 driver, and not W5200 TCP/IP stack. mzimmers: the W5200 could be made to stall (without recovery) by inundating it with nonsensical messages. Information encapsulated into TCP packet will have a format anyway, understandable to the application running in the device. What device is expected to do when it receives ‘A’? What does it actually do? mzimmers: Repeating this test with as little as 1ms between packets prevented this issue. If you increase the deay between packets, does the window size starts increasing? Make a test with variable delay. mzimmers: I can provide a Wireshark trace file if desired. You explanation is clear - W5200 always reports window size 0, and this is what we will see in the log. So the log is not much of value until you figure out how device treats ‘A’ and how fast if flushes data from its RX buffers.

mzimmers · Answer

The MCU is a legacy 8-bit processor (AVR ATMega1284P). Its resources, both compute and memory, are quite limited. The original programmer, as an effort to save space, designed the application to leave data in the WizNet buffer until a complete message has been received, or a timer elapses. The incoming messages are encrypted, so this adds to the difficulty of knowing when a complete message has been received. The 1ms delay results in the TCP window remaining within ~20 bytes of it’s 2048 capacity. I don’t think there’s much point to testing with further delays.

mzimmers · Answer

How do I free the buffer? It’s not clear from the datasheet. The current code does this: void HOSRxFlush(void) { HosRxPut = HosRxGet = 0; // Rx buffer pointers } Which seems to be updating pointers, but not actually freeing any memory.

mzimmers · Answer

Eugeny: W5100’s datasheet chapter 5 Presumably about the same as the W5200’s datasheet? (I’m using the W5200.) So, it appears that my MCU isn’t fast enough to properly service the incoming data, and I should be prepared for the eventuality of a ZeroWindow problem. Can I diagnose this from the MCU and reset the socket? (This would be a bit help if possible.)

Eugeny · Answer

mzimmers: Presumably about the same as the W5200’s datasheet? (I’m using the W5200.) I think so. Some datasheet misses valuable information (e.g. W5500 has a tiny piece of what is explained in W5100’s chapter 5). mzimmers: I should be prepared for the eventuality of a ZeroWindow problem It depends on the application of your design. If incoming data is not expected to be like a DDoS you can live with it easily (given there’s none on the network starting behaving this way). mzimmers: Can I diagnose this from the MCU and reset the socket? Not sure I understand. You want to discard data? Then you just update pointer to current position, this way “freeing” the buffer. No need to reset socket or even whole device. You can consider alternative solutions - depending on MCU models, there could be pin-compatible (or even code compatible) replacements with higher processing speed. I know Microchips are good at making such product lines of slower - faster -fastest devices which can be put into the same socket and require minor changes to the code when porting the applications. Also depends on how you service incoming data - by polling WIZnet chip registers or using interrupts.

mzimmers · Answer

Thanks for the good information, Eugeny. What I meant in my earlier post is, I think that – as our product is currently built – we need to address the issue of the TCP Window filling up, because the MCU can’t remove data from the W5200 as quickly as the W5200 can receive it from the network. When the buffer is full, the W5200 issues a ZeroWindow message. This will go on forever until the system is rebooted. My question is, is there an intermediate way to clear the buffer? The routine I posted above doesn’t do it. trace1172×396 15.9 KB We as a company are willing to look at replacing our MCU (and upgrading from the W5200 to the W5500 if need be), but I want to know what we can do with our current configuration.

mzimmers · Answer

Hi Eugeny - Yes, you’re correct that the application doesn’t remove any data upon timeout. This may be a bug in our app, and would explain the current behavior. Right now, I’m prepared to live with losing data, as long as I can prevent this TCP stall. I’ve read section 5.2.1.1 in the data sheet, and I believe what I need to do is reset the pointers Sn_RX_RD and Sn_RX_WR for the socket that is giving me trouble. Can I clear these pointers directly, or do I need to execute a RECV command? Thanks for all the assistance.

mzimmers · Answer

So, write to the socket registers in this order: W5200RegWrite(0x4328, 0x0); W5200RegWrite(0x4329, 0x0); W5200RegWrite(0x432a, 0x0); W5200RegWrite(0x432b, 0x0); W5200RegWrite(0x4301, 0x40); // the RECV command Is that correct? Do I need to do something else to commit the command? Thanks.

Eugeny · Answer

mzimmers: Is that correct? I do not think so. Why you write zeros? Which registers you write to? What must you write in these registers?

mzimmers · Answer

0x4328 is Sn_RX_RD0 0x4329 is Sn_RX_RD1 0x432a is S1_RX_WR0 0x432b is Sn_RX_WR1 0x4301 is Sn_CR; the 0x40 is the RECV command. If I don’t change the pointers to 0, what should I change them to, in order to flush the buffer?

Eugeny · Answer

Datasheet is pretty clear on it on page 50: Sn_RX_RD += len; where len = Sn_RX_RSR; What this does: you shift RD pointer past the received data size of RSR. Then perform RECV command which will mark skipped buffer area as free for new incoming data. If you assign 0 to pointers it will break up TCP sequence and most probably prevent further communication performing correctly due to improper window indication and calculation.

mzimmers · Answer

Oh…so: u8 uc0; u8 uc1; W5200RegRead(Sn_RX_RSR0, &uc0); W5200RegRead(Sn_RX_RSR1, &uc1); W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); W5200RegWrite(Sn_CR, RECV); Is that better? Thanks, Eugeny. EDIT: oops, no that’s not right…hang on a minute, please…I’ll correct it.

mzimmers · Answer

Here’s another go at it: u8 uc0; u8 uc1; u16 rsr; u16 rd; W5200RegRead(Sn_RX_RSR0, &uc0); W5200RegRead(Sn_RX_RSR1, &uc1); rsr = (uc1 << 8) | uc0; W5200RegRead(Sn_RX_RD0, &uc0); W5200RegRead(Sn_RX_RD1, &uc1); rd = (uc1 << 8) | uc0; rd += rsr; uc0 = rd & 0xff; uc1 = rd >> 8; W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); W5200RegWrite(Sn_CR, RECV); rc = recv(SockTCPL, HosRxDat.bad, &len, &(HosRxIp.ul), &port); Does this look correct now?

Eugeny · Answer

Read octet pairs into 16-bit unsigned variables (RSR and RD), then sum them up, and write resulting 16-bit value by its octets into RD. So second try seems to be the correct one I am sure you will see result immediately when running it. After you perform RECV wait until CR register clears to 0 before going further. rc = recv(SockTCPL, HosRxDat.bad, &len, &(HosRxIp.ul), &port); What this one does? Is it that you instruct to read data from the socket? Then there’s no sense in previous code. Or I would say this is redundant call as you have just cleared the buffer.

mzimmers · Answer

Well, this looks hopeful. I may have solved the stall problem. I am now seeing, however, tons of TCP Dup ACK messages from my device: dupack1213×460 12.3 KB Any idea what could be causing this?

Eugeny · Answer

Did you remove rc = recv(SockTCPL, HosRxDat.bad, &len, &(HosRxIp.ul), &port); from the code? It may cause 0 size to be “received”, and thus dup ack packet sent. And by the way, what socket number command W5200RegRead(Sn_RX_RSR0, &uc0); assumes?

mzimmers · Answer

No, I didn’t remove the recv() call…I thought it was needed to effect the change in the pointers. Sn_RX_RSR0 is defined with the socket number included in it (0x4326). The read just goes to an address; it doesn’t know about sockets. Socket 3 is the only one that needs this special attention.

Eugeny · Answer

Ok, remove recv() and try. Note that what we do now is a kind of “low level hacking”, ideally you must use recv() call which moves data (and which you want to avoid under your low performance circumstances). Do not forget to put comments into your code, as you definitely forget everything in a week or two.

mzimmers · Answer

Can’t get it working. Here’s the clear routine: void W5200ClearRxBuffer() { __monitor char W5200RegRead(u16 adr); __monitor void W5200RegWrite(u16 adr, char data); const u16 Sn_MR = 0x4300; const u16 Sn_RX_RSR0 = 0x4326; const u16 Sn_RX_RSR1 = 0x4327; const u16 Sn_RX_RD0 = 0x4328; const u16 Sn_RX_RD1 = 0x4329; const u16 Sn_CR = 0x4301; const u8 RECV = 0x40; u16 len = APGmPktZ; u16 port = LANCLibPO; u8 uc0; u8 uc1; u16 rsr; u16 rd; // 2018 11 21 mzimmers: // temporarily read Sn_MR to see whether // no delayed ACK is set in bit 5. // note: it is. we might want to change this. uc0 = W5200RegRead(Sn_MR); uc0 = W5200RegRead(Sn_RX_RSR0); uc1 = W5200RegRead(Sn_RX_RSR1); rsr = (uc1 << 8) | uc0; uc0 = W5200RegRead(Sn_RX_RD0); uc1 = W5200RegRead(Sn_RX_RD1); rd = (uc1 << 8) | uc0; rd += rsr; uc0 = rd & 0xff; uc1 = rd >> 8; W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); // it isn't clear yet whether the lines below are necessary. // commenting them out for now. //W5200RegWrite(Sn_CR, RECV); //recv(SockTCPL, HosRxDat.bad, &len, &(HosRxIp.ul), &port); }

mzimmers · Answer

Hi Eugeny - By “not working” I meant that the stated problem persists: as the flood of incoming TCP messages rises, the TCP window steadily shrinks to 0, at which point the socket is unresponsive. I made the changes you suggested, tried this with and without a final recv() call, and even eliminated the direct buffer manipulation in favor of a lone recv() call. Behavior is unchanged. I’m beginning to think the problem is simply that the W5200 is so fast, and the CPU so slow by comparison, that the W5200 is just “outrunning” the CPU. __monitor void W5200ClearRxBuffer() { const u16 Sn_RX_RSR0 = 0x4326; const u16 Sn_RX_RSR1 = 0x4327; const u16 Sn_RX_RD0 = 0x4328; const u16 Sn_RX_RD1 = 0x4329; const u16 Sn_CR = 0x4301; const u8 RECV = 0x40; const u16 len = APGmPktZ; const u16 port = LANCLibPO; __monitor char W5200RegRead(u16 adr); __monitor void W5200RegWrite(u16 adr, char data); u8 uc0; u8 uc1; u16 rsr; u16 rd; uc0 = W5200RegRead(Sn_RX_RSR0); uc1 = W5200RegRead(Sn_RX_RSR1); rsr = (uc0 << 8) | uc1; uc0 = W5200RegRead(Sn_RX_RD0); uc1 = W5200RegRead(Sn_RX_RD1); rd = (uc0 << 8) | uc1; rd += rsr; uc0 = rd >> 8; uc1 = rd & 0xff; W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); W5200RegWrite(Sn_CR, RECV); recv(SockTCPL, HosRxDat.bad, (u16 *) &len, &(HosRxIp.ul), (u16 *) &port); }

mzimmers · Answer

I’m still trying to get this solved. I’d like to begin by merely discarding any data in the buffer. To do this, is it sufficient to set Sn_RX_RD = Sn_RX_WR, and then write a RECV into Sn_CR? Or is something else necessary?

mzimmers · Answer

I still can’t update the pointers with the RECV command. Here’s my code: uc0 = W5200RegRead(Sn_RX_RD0); uc1 = W5200RegRead(Sn_RX_RD1); rd = (uc0 << 8) | uc1; uc0 = W5200RegRead(Sn_RX_WR0); uc1 = W5200RegRead(Sn_RX_WR1); wr = (uc0 << 8) | uc1; rd += rsr; uc0 = rd >> 8; uc1 = rd & 0xff; W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); W5200RegWrite(Sn_CR, Sn_CMD_RECV); do { uc0 = W5200RegRead(Sn_CR); // read command register } while (uc0 != Sn_CR_DONE); // until command complete // for debugging only; just to look at the registers. uc0 = W5200RegRead(Sn_RX_RSR0); uc1 = W5200RegRead(Sn_RX_RSR1); rsr = (uc0 << 8) | uc1; uc0 = W5200RegRead(Sn_RX_RD0); uc1 = W5200RegRead(Sn_RX_RD1); rd = (uc0 << 8) | uc1; uc0 = W5200RegRead(Sn_RX_WR0); uc1 = W5200RegRead(Sn_RX_WR1); wr = (uc0 << 8) | uc1; rd and wr do not change. By the same token, rsr always seems to be 0. What might I be doing wrong? Thanks.

Eugeny · Answer

I assume there’s RSR read before the whole code you posted? Of course after you perform RECV register RSR will contain 0, until socket receives some new data. Or you say that RSR is zero before you perform RECV (the RSR read is not in the code you posted)?

mzimmers · Answer

Hi Eugeny - Yes, there’s an RSR read prior to the code I posted. I occasionally have issues inserting code into my posts here. The RSR register is zero upon entry to this routine, but I think that’s a matter of how/where I call this routine, so it’s not a W5200 issue. I’m not sure I fully understand the behavior of the W5200. After performing a RECV, should I expect the RD/WR pointers to change? Or is only the RSR register altered?

Eugeny · Answer

mzimmers: I’m not sure I fully understand the behavior of the W5200. After performing a RECV, should I expect the RD/WR pointers to change? Or is only the RSR register altered? There’re three locations within the chip: RD: location of the “user” pointer - the data which application must read; WR: system pointer, the location where the chip’s network stack writes when data arrives; RSR: it is simply WR-RD, and it can not be larger than RX buffer size. Note that all registers are 16-bit, and must be treated as 16-bit values even if buffer size is 8KB. Thus you do not touch WR under normal circumstances, it is managed by the chip. You only look at the RSR - amount of data received into the RX buffer, and at RD - the location to start reading data from. Thus if you read RSR, and it appears 0 - this means that buffer is empty, there’s no data to read, and chip must respond with full window size free in its network packets. Logically, if RSR is not 0, then RD is not equal to WR. Maximal value of RSR is RX buffer size. When you write to RD pointer new value, and then issue RECV command, chip assumes that data between old RD location and new RD location is flushed by the application, and considers this space as available for new incoming data. Therefore reading RSR, then assigning RD=RD+RSR and then RECV “frees” RSR bytes in the RX buffer, and decreases current value of RSR by RSR when it was previously read. Note that while you perform reading of RSR, then RD, then performing RD+RSR and assigning it back to RD time has passed, and actual RSR register value may increase. But it does not matter because RSR you originally read will always be <= current RSR, and there will be no buffer overrun, next time you read RSR you will get “remaining” value of RSR. Then, very important note: when reading any 16-bit register (RD or RSR or anything else) you must read byte at +0 first, and byte at +1 second. Not critical for RD because it will not change dynamically without your RECV…

mzimmers · Answer

Thanks for the explanation. I believe I have it working now. Here’s my modified code: rsr = getW5200Reg16(Sn_RX_RSR0); rd = getW5200Reg16(Sn_RX_RD0); wr = getW5200Reg16(Sn_RX_WR0); while (rsr > 0) { rd += rsr; uc0 = rd >> 8; uc1 = rd & 0xff; W5200RegWrite(Sn_RX_RD0, uc0); W5200RegWrite(Sn_RX_RD1, uc1); W5200RegWrite(Sn_CR, Sn_CMD_RECV); do { uc0 = W5200RegRead(Sn_CR); // read command register } while (uc0 != Sn_CR_DONE); // until command complete rsr = getW5200Reg16(Sn_RX_RSR0); } This seems to be effective at minimizing the impact of TCP-level DoS attacks that consist of nonsensical messages. Thanks for all the help with this.