STEPMOVEABS not always setting Move Complete relay

[Joel Moore]

I think I've narrowed down the cause of a frustrating issue we've been having with our machine where a motorized axis would complete a move yet our PC-based control software would never detect that the move was complete.  It seems that on occasion STEPMOVEABS will successfully finish a move but the move complete relay never gets set (which is what our PC-based control software is monitoring).  By the way, this isn't related to the issue described here since we are definitely moving.

I narrowed it down to the PLC by capturing the serial communications between the PLC and the PC.  One of the commands in the PC software's update loop is to read a bank of relays (host command RR) which contains the move complete relay and the response clearly shows that the relay is not changing state.

My communications log captured 1255 sent commands that result in STEPMOVEABS, the last one of which failed to set the move complete relay.  That's many more moves/shift than a machine would typically make in a production environment but it's still concerning since I think it's reasonable to expect it to never happen.

Could the fact I'm reading the state of that relay many times a second via the RR command sent via RS232 be interfering with the PLC's ability to set it once a STEPMOVEABS is complete?  That sort of conflict would certainly explain why it seems so random.

BTW, we are using FMD88-10 PLCs.  If there's any other info that might be useful let me know.

[garysdickinson]

Joel,

In my client's application, the use of the stepmoveabs statements are at a very small rate.  In my case it is possible for the PLC to run for days or weeks without needing to move the stepper motors, so I have not run into the problem.

But, I did run into something, similar,  back in August 2016 with the high speed counters.  The problem with the highspeed counter mechanism was that it was supposed to invoke a CF when a count value was reached. But, periodically, the CF was not invoked. This caused all sorts of havoc with the behavior of the system that my client sold and delivered to his customers. I worked this problem offline with Leon at TRI.

Leon traced the problem to how the PLC firmware handled interrupts.  My understanding was that the low level firmware responded to an interrupt caused by the PLC hardware and in response to this event queued a request to call the CF.  The bottom line problem was that the queue mechanism would periodically, loose (ignore, discard, ...) the queued request and the CF never was invoked.

It is possible that the stepper notification (RELAY set) mechanism may be implemented in the low-level PLC firmware in a very similar way. If this is true, then the same issues with the low-level PLC firmware may be causing the problem that you are observing.

I do have a suggestion that may help both detect your problem and make your PLC way more reliable.  This is the sort of work-around that I am still doing for the HighSpeed counter:

• Use a "dead man timer" to detect the absence of a response (RELAY set from completion of the motion comand).
• When you issue the StepMoveAbs statement, set RELAY that is connected to the dead man TIMER.  The Set Value of the TIMER should be greater than the worst case time required for the stepper to move to the new position.
• If the RELAY associated with the StepMoveAbs is received, clear the RELAY controlling the dead man timer.
• If the dead man TIMER goes off, then you know that the StepMoveAbs command has failed to set the RELAY specified by the command and you can now clear the RELAY that controls the dead man TIMER.

Another approach is to set up a mechanism that periodically calls a CF while your StepMovAbs command is active.  The CF that is called periodically will use the StepCountAbs(1) function to read the current absolute position for the stepper system.  If the stepper has moved to the position that you have commanded and the RELAY has not been set, you can then set the RELAY in your CF and then the rest of your PLC code will operate as you have designed.  Problem is not solved, but you have a work around.

Best regards,

Gary D*ckinson

[Joel Moore]

Thanks for the thorough reply.  That's interesting about the counter issue.

I was actually considering using the absolute position as a backup means to check for completed moves since we're already pulling that information onto the PC as part of the regular update loop.  However this is all old code that I'd rather not have to poke too much at the moment lest everything falls to pieces so I was hoping to find out if there was a known PLC firmware issue that could be solved by upgrading.

This issues is actually something I only discovered on my own while trying to figure out a different problem we were having with one machine in particular.  I haven't heard any complaints from the field regarding this so maybe it's rare enough to be inconsequential.  

[support]

We are not aware of any issue with the relay not being set after a STEPMOVE or STEPMOVEABS command.

Does the PLC program use the completion relay to trigger another motion immediately? If so, then the completed flag relay that was set in the previous motion will be cleared once a new motion starts. If the PC software did not read the completion relay quickly enough it may just miss the change of state.

Also in your PC program do you use the WR command to write to a relay bank that also includes the completion flag?  What was the command that your PC send to the PLC to trigger the motion?

[Joel Moore]

No, it's not used to trigger another move.  It is only clearing a couple of relays.  It's also used as part of an axis homing sequence but that also requires a particular state of the homing sequence to be active which would also result in the axis position being zeroed which isn't happening.

I'm not writing to the bank of relays that includes the move complete, only reading (RR05).  I write to a different bank (WR04).  I do write to a variety of relays as single bits but never the "move complete" one used with STEPMOVEABS.

One of the relays I write to is the one used to trigger the move in the first place however according to the captured data that only gets written to once when the move is initiated.  Between the points when the move is initiated and when I manually intervene the following conversation repeats continuously:

[Send] @01WR040040*      Write output state (use relays as a buffer between PC and PLC outputs)
[Resp] @01WR44*
[Send] @01RR0544*      Read status relays (this includes the move done flag R47)
[Resp] @01RR0948*                
[Send] @01Wb041AFF00*   Reset watchdog timer
[Resp] @01Wb74*
[Send] @01RVIX54*         Read axis position
[Resp] @01RVIFFFFE4437A*  
[Send] @01RI005A*         Read inputs
[Resp] @01RI0C29*

I think I'm going to write a little program to parse the captured data to look for anything unexpected.  It's a bit much (8 million lines or so) to peruse manually.  I'll probably eventually wind up implementing the secondary check as suggested above.

[support]

Are there any other interrupt intensive activities going on at the same time. E.g. High speed counter or pulse measurement?

What is your PLC firmware version? If it is >= r83 you could try upgrade the firmware to the latest where interrupt service routine has been improved to avoid overrun of the interrupt service queue where there are lots of interrupt happening at high speed, as reported by Gary. The improved interrupt service has worked well for Gary and eliminated the issue he mentioned, although his issue is quite different from your report.