Module telnet behavior

Question

After weeks of scratching my head over inconsistent telnet behavior, I think I’ve finally isolated what causes my problem, though I have no idea why or how to fix it. WIZ145SR running 1.6.1 firmware Serial side: a small microprocessor console port, 9600-N-8-1. Network side: telnet from a Red Hat Enterprise Linux Server release 5.7 (Tikanga) box Scenario 1: When I establish the telnet session before allowing the microprocessor to boot (i.e. don’t push any characters out the console), everything works as it should as long as I make sure the telnet session is in character mode instead of the default line mode. Scenario 2: When I allow the microprocessor to boot and push a bunch of characters out the console before establishing the telnet session, all kinds of weird things happen. 1) I see a bunch of stuff come out of the terminal server that it must have received before the telnet session was established. 2) I see the terminal server throw some garbage character or characters toward the console. 3) all characters are now echoed. When I start in scenario 1 (and all is good), terminate the telnet session gracefully, and then re-establish the telnet session, I’m back to scenario 2. It looks like re-establishing the telnet session throws some junk at the console that gives my micro fits. Why would the terminal server appear to be buffering characters from before a telnet session was established? Wouldn’t initializing the session clear any characters in the buffer? I’m not talking about characters coming in during the initialization. The characters I see were sent out the console well before I attempt to establish the telnet session. Why would establishing a telnet session force any character or characters out the serial port?

gthompson · Answer

Anyone out there? This is a show stopper for me. I’ve got 100 modules here I can’t use if I don’t find a solution/work around for this. Gary

jameskim · Answer

Hi Gary, I’m sorry for our late response. I’m the developer of WIZ145SR. I’ll support you since now. To answer you correctly, I have to check it at my side. Please wait for several days and I’ll give you my feedback as soon as possible. Thank you. James.

jameskim · Answer

Hi Gary, I tested WIZ145SR with firmware ver 1.6.1 and figured out what you told. I looked through the source code of WIZ145SR carefully and found out what made this symptom. Yes, it has had some bug in handling serial buffer. There are several timer for handling serial data and several pointer to serial buffer. When WIZ145SR receives data after connection is established, initialization of several timers and pointer is done correctly. But the initialization is done by receiving data from serial port and timers and pointer is already being updated, even though the tcp connection is not established yet. This is the reason why WIZ145SR pushed wrong data. I fixed and tested it at my side. I hope you to test with my revised firmware at your side. WIZ140v1_6_2_140326.zip (24.5 KB) I put its version 1.6.2 temporarily and it is not the official version yet. In order to release its firmware officially, we need more time to test by our rule. Anyway, you can test with the attached file at your side.(I compressed it as a zip file. please decompress it first.^^) Thank you. BR, James.

gthompson · Answer

James, Thanks for looking at this. I think things are improving. I don’t see the buffered characters when I start a telnet session now, but I do still see some characters thrown out the serial port when a telnet session is started. I don’t think this is supposed to happen. Output from the terminal server serial port when I start a telnet session: 0xFF 0xFD 0x03 0xFF 0xFD 0x01 ~2.5ms gap 0xFF 0xFC 0x3c ~32ms gap Baud rate and signalling looks correct, but establishing a telnet session shouldn’t initiate anything out the serial port.

jameskim · Answer

Hi gthompson, I’m happy to hear that your test is going forward. Anyway I’m sorry you’re still in some trouble. When I was testing with the firmware ver 1.6.2 which I uploaded here, I couldn’t see the symptom you said above. I’ll test again. Could you tell me your test condition more details? How are all devices connected? When were above data, which were displayed on telnet program, sent from your terminal server? And are those real data from terminal server or garbage? If those are real data, I think timing mismatch between serial and ethernet make outputting data on telnet program. Because serial port is much slow compared to ethernet, your terminal server may be still having data to output into serial port in its tx buffer when telnet session is created. So, I recommend you check the timing of those signals, RXD of UARTs and STATUS pins. When every connection is established, its corresponding STATUS pin is changed to LOW. If RXD is not idle after STATUS pin is changed, above data is correct. If there is no transition of RXD after changing of STATUS pin, my firmware has still some fault. Please inform me your test result. Thank you. James.

gthompson · Answer

That was real output from the terminal server taken on an oscilloscope, so I know the timing and protocol look correct. This output occurs immediately after I establish a telnet session with the module, but before I start to send it any real data. It is consistent in that I get the same data every time, and every time I start a new telnet session, I get the same data. I see no difference between 1.6.1 and 1.6.2 in this regard. Here is my configuration: network side attached to a live network I telnet into the terminal server from a Linux box Linux version 2.6.18-274.18.1.el5 (mockbuild@x86-001.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-51)) #1 SMP Fri Jan 20 15:11:18 EST 2012 Serial port 0 is configured for 9600,N,8,1 port 5000. Serial port 0 is connected to a small microprocessor running Linux u-boot. Only RXD/TXD are connected. Handshake signals are not used. The microprocessor is booted to the u-boot prompt and sitting waiting for a command. It will not send any characters to the terminal server unless it receives something. So, sequence of events is as follows: power on terminal server and target system bring target system out of reset wait for target system to boot from linux box> telnet axx-g004-ts 5000 I see junk start to come out of the terminal server on the UART1_TX pin (module J1 pin 7). target system sees this junk, echoes it back to the terminal server and runs off into never-never land. If I move step 2 after step 4, everything works as it should (the target isn’t alive to see the junk at the time the telnet session is started, so it is happy). Unfortunately, this is not always a viable option since I need to be able to initiate and terminate telnet sessions without resetting the target. If I remove the Wiznet module from the system, and wire the target console connection to a Cisco 2800 or a Digi PortServer TS16 terminal server, everything works great. I can initiate and terminate telnet sessions at will without disturbing the…

gthompson · Answer

OK, a little more data. When I start the telnet session in your default line mode setting, I don’s see any characters out when the telnet session is initiated. When I escape back to the telnet prompt and change the mode to character, I see the above characters come out the serial port. Similarly, when I start a telnet session with a .telnetsrc file that automatically switches to character mode, I see the characters. So I guess I see why you don’t see the same thing I do. Try changing the telnet mode from line mode to character mode and see if you don’t see the same extraneous character pop out the serial port. BTW, I looked at CTS and RTS, and they both stay at 0 (on the module pins) through the whole process. I have an LED on the STATUS pin, and I do see it light (STATUS goes low) well before the characters are sent. Gary

jameskim · Answer

OK, I understood what the problem was. At first, WIZ145SR doesn’t have any special consideration for telnet modes. It just exchage data transparently between serial port and ethernet socket. I think normal telnet server do some different action according to telnet mode. This is why our WIZ145SR didn’t operate as you expected. I’ll check what I should do to meet your demand. Please wait for a few days. And as I told you before, You found that WIZ145SR get received data via UART port from your serial device after telnet session was established. The timing of signal transition is the evidence of what I told you. Yes, as CTS and RTS is signals for flow control, if you didn’t set RTX/CST mode for flow control, those wouldn’t change those level. When STATUS pin went to low, it means telnet session was established and WIZ145SR send all data received via UART from this point. WIZ145SR worked well even at your side as I expected. Anyway, the problem which you encountered came from the lack of special consideration for telnet mode. I’ll survey how I can fix it and inform you the result as soon as possible. Thank you. James.

tobias · Answer

Hello Gary, I did some tests to reproduce the behavior. My Testsetup was Putty as Telnet client and hterm as serial client programme. When I set putty to use Telnet mode I can see the extra characters. When I set putty to raw mode there are no extra characters. As James already mentioned this is because the current firmware has no support for “telnet” mode but only for “raw” mode. So my question is, do you really need the telnet mode? Or is “raw” mode sufficient? If you set your client to “raw” mode it will immediately work as you expect it. But if you really need telnet mode, then James has to enhance the firmware for this. Please tell if raw mode works for you. You can test this easily this with putty as client. Regards Tobias

gthompson · Answer

Tobias, Thanks for your help with this. Unfortunately, I really need telnet mode. I’ve got a bunch of software developers and hardware validation guys using these terminal servers to access test systems. They’ve got years worth of automated test scripts and processes that I can’t really ask them to change. My decision to transfer from rack mounted Cisco terminal servers to Wiznet embedded terminal servers needs to be completely transparent to them. If I had realized your terminal servers weren’t really telnet-capable, I would have never gone down this path.

jameskim · Answer

gthompson, OK, I’ll try to add minimal telnet capability onto WIZ145SR. Implementing full function for telnet mode may take long time. So, I try to change my code not to bypass “Telnet Negotiation Message” to Serial port. Can you test with it? Or do you need some more function? And regarding to implementing full function for telnet mode, I need to consider carefully and I’ll discuss my colleagues. Thank you. James.

gthompson · Answer

Ideally, we’d like the telnet negotiation to work, but we can live without it by forcing our end through .telnetrc. I think not passing the telnet negotiation string on to the serial port is all we really need for now. I’d be glad to test it once you are ready. Thanks to everyone for helping us get this going! Gary

jameskim · Answer

Hi Gary, I finished firmware and configuration tool customization to support telnet mode. Now my colleagues and I are doing internal test. After test, I will post here or email to you with the firmware and configuration tool. Please wait for a few days. Thank you. James.

gthompson · Answer

Thank you, James. I’ll look for your updates. Gary

jameskim · Answer

Hi Gary, I posted new firmware and configuration tool here, [Notice]Latest FW and Configuration Tool Please download them and test with them on you side. Thank you. James.

gthompson · Answer

James, I finally got some quality time with your new release, and I’m still seeing something strange. I made some comparisons between a Cisco terminal server that works for us and the Wiznet module. Using telnet netdata and termdata debug commands, I was able to comapre what was happening on both the netwkr side and ther terminal side as far as telnet was concerned. THEY BOTH LOOK IDENTICAL! Cisco network data () => > 0x0 0d00 < 0x0 0d0a3d3e20 => > 0x0 03 network data (CTRL-J) => > 0x0 0a < 0x0 0d0a3d3e20 terminal data () < 0x0 0d => > 0x0 0d0a3d3e20 terminal data (CTRL-J) < 0x0 0a => > 0x0 0d0a3d3e20 Wiznet network data () 0x0 0d00 < 0x0 0d < 0x0 0a < 0x0 3d =< 0x0 3e < 0x0 20 network data (CTRL-J) 0x0 0a < 0x0 0d < 0x0 0a < 0x0 3d =< 0x0 3e < 0x0 20 terminal data () < 0x0 0d 0x0 0d 0x0 0a => 0x0 3d 0x0 3e 0x0 20 terminal data (CTRL-J) < 0x0 0a 0x0 0d 0x0 0a => 0x0 3d 0x0 3e 0x0 20 This is definitely NOT what I expected. If they really are identical, both should work equally well, right? Unfortunately, while telnet reports a lone 0x0d () on the terminal side, what I actually get on the scope is 0x0d 0x00 ( NUL). Where did that null come from? It looks like a standard Linux u-boot will accept a CRLF, a CR, or a LF as suitable end of line characters. It doesn’t know what to do with the NUL, though, so it ends up stuck in the buffer until the next command comes along, causing the next command to parse incorrectly. It would be interesting to see if you see the same thing. When you send just a 0x0d from the network side, what comes out the serial side? Gary

jameskim · Answer

OK, I’ll try to test it soon. Thank you for your feedback. James.

jameskim · Answer

Hi Gary, I’m sorry I couldn’t produce its symptom on my side. It may be a reason that data from terminal was separate into several block. If you set your module up like below picture, you would get the whole data from terminal at once. Can you give me a feedback after you test again with above setting? Thank you. BR, James.

gthompson · Answer

What does changing the Time field to 100 do? I don’t really see any difference. Make sure you have your telnet session configured as I do in single character mode. I’ve got “mode char” set in my .telnetrc file to force character mode on all telnet sessions. If I stay in the line mode your telnet defaults to, I get , on the serial side, which u-boot interprets as a double end of line. Even with Time=100, I see a from the remote terminal pass a , to the serial port. telnet> status Connected to axx-g004-ts. Operating in single character mode Catching signals locally Remote character echo Escape character is ‘^]’. telnet> display will flush output when sending interrupt characters. won’t send interrupt characters in urgent mode. won’t skip reading of ~/.telnetrc file. won’t map carriage return on output. will recognize certain control characters. won’t turn on socket level debugging. won’t print hexadecimal representation of network traffic. won’t print user readable output for “netdata”. won’t show option processing. won’t print hexadecimal representation of terminal traffic. echo [^E] escape [^]] rlogin [off] tracefile “(standard output)” flushoutput [^O] interrupt [^C] quit [^] eof [^D] erase [^?] kill [^U] lnext [^V] susp [^Z] reprint [^R] worderase [^W] start [^Q] stop [^S] forw1 [off] forw2 [off] ayt [^T] resp DO_DONT ECHO: 1 want DO ECHO resp DO_DONT SUPPRESS GO AHEAD: 1 want DO SUPPRESS GO AHEAD => I still see this: changing to look at network side data telnet> set netdata Will print hexadecimal representation of network traffic. this is in response to a on the remote terminal 0x0 0d00 # this is the network data from remote terminal to Wiznet < 0x0 0d0a3d3e20 # this is the network data from wiznet to the remote terminal => changing to look at serial side data telnet> unset netdata Won’t print hexadecimal representation of network traffic. telnet> set termdata Will print hexadecimal representation of terminal traffic. this is in response to a on the remote…

gthompson · Answer

From RFC-854 (telnet protocol specification): “Note that “CR LF” or “CR NUL” is required in both directions (in the default ASCII mode), to preserve the symmetry of the NVT model. Even though it may be known in some situations (e.g., with remote echo and suppress go ahead options in effect) that characters are not being sent to an actual printer, nonetheless, for the sake of consistency, the protocol requires that a NUL be inserted following a CR not followed by a LF in the data stream. The converse of this is that a NUL received in the data stream after a CR (in the absence of options negotiations which explicitly specify otherwise)[color=#FF0000] should be stripped out prior to applying[/color] the NVT to local character set mapping.” Emphasis above is mine. The way I read this, when you see a from the network side, you should strip the and send only a to the serial side. When you see a , you should go ahead and pass this on unchanged. Similarly, when you see a only from the serial side, you should translate this to a or a on the network side depending on the configuration. Gary

jameskim · Answer

Hello Gary, Thank you for your detail information. I’ll update my firmware with our suggestion. Then, I have some questions about this. How often does come from ethernet side? And when I get from serial side, how can I consider it is whether sole or will be followed? For example, is coming from serial side, I change to when I received . Then What happen? Even though I received from serial side, I send [color=#FF0000][/color] to network side. Will not another side effect happen? What is your opinion? James.

gthompson · Answer

Taking a from the network side and stripping the to the serial side is the easy part. Going the other way is much harder. In my experience, if the serial device is going to generate a , it will typically be back-to-back characters with very little dead time between them. You might try something like this: CR_NUL.pdf (9.6 KB)

jameskim · Answer

Hello Gary, I fixed my firmware as you mentioned. However I need to explain what I fixed. Stripping which is very next to is adopted, but adding or to sole is too difficult without reorganizing my whole firmware. It is nearly impossible to do that. I thought how I could add the second function and I concluded it may take much more days to implement it without reconstruction of my firmware. For supporting standard serial-to-ethernet functions, my firmware does many thing with data from serial. But there is no action to check which byte was received and parse them according to application. And data receiving from serial port happens in UART ISR. All of your request should be done not affecting to WIZ145SR’s original functions. So, I hope you to test with the attached file which has only the function eliminating . And if you have some difficulty without the second function, then let me know it. I’ll try to find how to fix. Thank you. James. WIZ140v1_6_3_140527.zip (25.1 KB)

gthompson · Answer

Woo hoo! No more NULs! This looks like it will work for us. Thank you very much for your efforts on this. If this is the version you want to go with, I’ll start rolling it out to my installed base immediately. Gary

jameskim · Answer

It sounds good. Thank you for your quick test. I’ll release it soon officially after some more test. Thank you. James.