Corrupted response from PLC

[Joel Moore]

Lately we've been having a lot of problems with the communications between our HMI software and the T100MD888+ PLCs in our machine.  We started logging the comms using Hyperterminal and a split cable and noticed a strange pattern.  It seemed the response from the PLC was corrupted but it was always the same character of the response.  For example, here are some of the corrupted responses:

@01Wb?4*
@01RD00003?001F001F001F001F001F001F001F001F0000000002B82B*

The "?" is what Hyperterminal prints in place of an unprintable ASCII code I think.  These errors have been captured multiple times and the "?" always appears in the same location for those errors (there are others).  This tells me it's not random glitches or RS232 noise.

Has anyone seen this pattern before?  I'm at a loss as to what could be causing it.

[support]

1) Check if your PLC program could be sending out characters out of the same serial port.

2) Try pausing the PLC (pause during online monitoring, or turn on DIP switch #4 to pause the PLC.

3) If after pausing the PLC you still see the same error, it could be the PLC's serial port is electrically slightly damaged, or the PC's RS232 port is slightly damaged, or both. Try using an Auto485 to communicate via the RS485 port and see if the problem go away.

[Joel Moore]

This is still a frequent problem.  I initially dealt with it by adding some retry code to our software but we still have problems.

There's no other code on PLC that uses the COM port so it's not stray data.  It happens on a number of controllers and PCs so it's not port damage.  Besides, it's always the same character location in the response which implies it's not random.

It's weird because we went for years without having any trouble and then suddenly this problem appeared.  One thing that's been pointed out is that the firmware version on the PLCs is newer than what we started with (r50 vs. r49 I think).  So naturally people are wondering if there could be a connection there.  A few questions:

- When was the PLC firmware updated to r50?

- Was anything changed that could affect the serial port communications (are there release notes available anywhere)?

- Is there any way to revert to an older rev of the firmware?

[support]

I don't think r50 firmware changes any of the hostlink command handling routine. We can run our TRiLOGI software in online monitoring mode for days without a single break whether it is r49 or r50 firmware.

Are you using a built-in serial port or USB-to-RS232 converter? If it is the latter what brand of converter are you using?

Are you connected to the PLC via RS232 or RS485 port? Is it one PC to one PLC?

What programming language are you using to handle the communication?

Did you try to run TRiLOGI and TLServer to monitor the PLC and do you observe any communication error?

[hemiguy]

Hi.
I'm working on this project now and we are still having this problem.
To answer your questions:

Are you using a built-in serial port or USB-to-RS232 converter? If it is the latter what brand of converter are you using?
Two serial ports, one built in and an additional serial card installed in the pc.

Are you connected to the PLC via RS232 or RS485 port? Is it one PC to one PLC?
Rs232, one pc, two plcs.

What programming language are you using to handle the communication?
Visual basic 2003.

Did you try to run TRiLOGI and TLServer to monitor the PLC and do you observe any communication error?
Not sure what you mean.

I added some logging to capture the communication and it appears to be completely random. Both Plcs will simply send a bad character, or send nothing. We retry the command up to three times and usually it recovers, but sometimes not and we lose communication. I've tried different baud rates, electrical filtering, which helps a bit, but the problem is still there.
I delayed updating the watchdog (Sleep(200)) as its updated continually, which did not help.
Our customers are seeing failures often as they run 24x7.
I'm out of ideas!

Thanks,
John

Here are some examples:
4/6/09 13:52:41   PLC2 retry, response = @01Wbw4*
4/6/09 13:52:41   Command sent was: Wb041AFF
4/7/09 17:30:49   PLC2 retry, response =
4/7/09 17:30:49   Command sent was: WR0400

4/7/09 18:41:37   PLC1 retry, response = @01Wbw4*
4/7/09 18:41:37   Command sent was: Wb040CFF

This is a sequence where the system failed:
4/8/09 05:04:21   PLC1 retry, response =
4/8/09 05:04:21   Command sent was: WR0400
4/8/09 05:04:21   PLC2 retry, response =
4/8/09 05:04:21   Command sent was: Wb041AFF
4/8/09 05:04:22   PLC1 retry, response =
4/8/09 05:04:22   Command sent was: WR0400
4/8/09 05:04:22   PLC2 retry, response =
4/8/09 05:04:22   Command sent was: Wb041AFF
4/8/09 05:04:23   PLC1 retry, response =
4/8/09 05:04:23   Command sent was: WR0400
4/8/09 05:04:23   PLC2 retry, response =
4/8/09 05:04:23   Command sent was: Wb041AFF
4/8/09 05:04:24   PLC1 retry, response =
4/8/09 05:04:24   Command sent was: WR0400
4/8/09 05:04:24   Connection to PLC1 lost.
4/8/09 05:04:24   PLC2 retry, response =
4/8/09 05:04:24   Command sent was: Wb041AFF
4/8/09 05:04:24   Connection to PLC2 lost.

[support]

Do you have the PLC source program? Can you examine the program code to check the following:

1) Does your PLC send data out of its RS232 port? E.g. PRINT #, OUTCOMM, NETCMD$, READMODBUS, WRITEMODBUS, READMB2, WRITEMB2. Any of these command would send unsynchronized data out of the serial port and can disrupt your Visual Basic program attempts to talk to the PLC.

2) Are there are frequent reads/writes to the EEPROM via SAVE_EEP, LOAD_EEP, SAVE_EEP$, LOAD_EEP$, SETTIMERSV, SETCTRSV. When the T100M+ PLC accesses the EEPROM it shuts off the system interrupt to prevent corruption of data and this could affect communication if the EEPROM access is too frequent. You can check by pausing the PLC temporarily (turn on DIP switch #4 but do not reset the PLC) and check if the communication error would disappear.


3) What is the waiting time for the response string from the PLC? Did you write your Visual Basic program based on our sample program downloadable from:  

   http://www.tri-plc.com/applications/VBsample.htm

[hemiguy]

Hi, thanks for the quick reply.

Yes I have the PLC source, I will have to examine it, I do see a netcmd$, and we update EEprom, but I'm not sure how often.
I'm not very familiar with ladder logic yet.  


We use SendCommand from VB:
Private Sub UpdateWatchDog()
        SendCommand("Wb041AFF")

Looks like there is a 5 second timeout.
Dim timeLock As Haack.Threading.TimedLock = Haack.Threading.TimedLock.Lock(sendCommandLock, TimeSpan.FromSeconds(5))


This problem occurs also when the tool is idle. I can log all serial activity on the PC over time, so I know what's going on, but is there a tool to monitor and log what's going on in the PLC at the same time so I could correlate between the two, especially when idle?

Thanks,
John

[support]

As mentioned previously, temporarily pause the PLC by turning on DIP switch #4 (without reset) and then check if the comm problem goes away. If so, then the problem is with the PLC program interfering with COMM. You will need to examine the code and possibly modify it to minimize EEPROM access while in operation. Most of the data can be saved/read from DM area and only those the need to be kept before power turned off should be saved to the EEPROM.

[hemiguy]

Ok, I'll give it a try and let you know.

Thanks,
J