Internet PLC Forum

General => Technical support => Topic started by: JRobinson on November 13, 2016, 05:30:58 PM

Title: DATE update after midnight
Post by: JRobinson on November 13, 2016, 05:30:58 PM
I have a number of FMD88-10s with the FRAM-RTC that report at 12 hour interval to a proprietary server at a central site over the Ethernet connection. Part of the report string is a time-stamp string, generated as below (the format being for Excel)

   H$ = LOCSId$ + ";" + STR$(DATE[1]) + "/" + STR$(DATE[2]) + "/" + STR$(DATE[3]) + " "
   H$ = H$ + STR$(TIME[1]) + ":" + STR$(TIME[2]) + ":" + STR$(TIME[3]) + ";"

Occasionally, if the report happens between 00:00 and about 00:30, the day field (DATE[3]) remains the previous day. I haven't been able to characterise this. It certainly is not every report and only some sites.

The segment below is  from a log, the site, among 15 others reporting the date as the 13th, reports the 12th. Previous log at midday on the 12th for this site contained the correct date, as did the log before that at 00:00 on the 12th.

          ;2016/11/12 0:22:47;

Question is, does the day field of the DATE variable take some time to update after midnight? Can I spot this happening or do something to avoid it?

Title: Re:DATE update after midnight
Post by: support on November 13, 2016, 10:27:45 PM
1) Can you check the firmware version of the PLCs that you experienced the DATE[3] not updated correctly and those that do update correct.

2) Does the PLC connect to a time server (such as your central server or the NIST time server) to update its time periodically? If so when does this occur?

Title: Re:DATE update after midnight
Post by: garysdickinson on November 14, 2016, 12:45:47 PM
I think that you are running into a issue with how the PLC's firmware updates the TIME[] and DATE[] registers.

If you think just about the time registers what happens when the time just gets to 23:59:59 and one second later the PLC system firmware needs to update the time to 00:00:00.  I'll get you a hint the time will not go cleanly form 23:59:59 to 00:00:00.  It is possible to observe all of the possible values :

23:59:59
23:59:00
23:00:00
00:00:00

And only the first and the last values are "right".

And by the way the roll from 23:59:59 to 00:00:00 will cause the a ripple in the DATE[] registers as the day number will be changed and if this is the last day of the month, the month number will change and if this is the last day or the last month of the year, then the year will change.

Another way of thinking about this is to think about an old car that has a mechanical odometer and you it is reading 69999 km.  If you  watch it as you are driving you will see that the least significant digit will get to 0 and you might think that your car has 69990 km but just wait and eventually all of the the digits will roll and the correct reading of 70000 km will be displayed.  Each digit is mechanically coupled to the next one and there is a ripple from digit to digit.  The exact same thing happens with the PLC system firmware.

There is another possible issue with the PLC's RTC mechanism. If you have battery backed RTC (all Fx models and FMD and Nano-10 if you installed the RTC module) you may see a "wrinkle in time". The issue is that even with the hardware RTC installed, TBASIC is not reading from the RTC hardware but from registers maintained by the PLC system firmware.  Periodically the PLC registers are updated to match the values in the hardware RTC.  If this happens once a day, there could be a either a gap in time for a duplication in time based on how far the PLC's CPU clock has drifted between updates.  I have not been able to catch the this wrinkle with TBASIC code, but I been able to verify the issues with the TIME[] and DATE[] registers.

OK how do you get around the ripple issue? I would suggest that you take one of 2 approaches:
1. Avoid reading the time/date registers when the register values are changing.
2. Detect that the values are changing and wait until they quit rippling (changing).

The Avoidance approach can be achieved by polling until the seconds register reads 1.  Now you have about 58 seconds to read all of the other time/date registers before you get into problem with the values changing again.
This is an example of the avoidance approach. The following code is called every 0.5 seconds and sets a RELAY that is used to invoke a custom function to perform your data logging:

if (Status(18) = 080001)
   ' Time is 8:00:01 and this is a time to log data
   Setio Time2Log
elif (Status(18) = 200001)
   ' Time is 20:00:01 and this is a time to log data
   Setio Time2Log
else
   ClrIO Time2Log      ' this is not the time to log data
endif
[/color][/font]

The Detection approach has you read all of the registers twice in a row and if any of the registers changed from the first to the second read, then you detected a ripple. You need to do this until you can read the registers 2x with no change from the first to the second reading.

This is an code example of the detection approach:

'Read RTC - code to access time of day registers and return CORRECT values
' in the following variables:
' DM[301] TIME[1], RTC Hours
' DM[302] TIME[2], RTC Minutes
' DM[303] TIME[3], RTC Seconds
' DM[304] DATE[1], RTC Year
' DM[305] DATE[2], RTC Month
' DM[306] DATE[3], RTC Day

while 1
   ' read all TIME[] and DATE[] registers, one time
   '
   DM[301] = TIME[1]
   DM[302] = TIME[2]
   DM[303] = TIME[3]
   DM[304] = DATE[1]
   DM[305] = DATE[2]
   DM[306] = DATE[3]

   ' re-read the TIME[] register and verify that they did not change
   ' from the previous set.  The whole point of this exercise is to
   ' detect and discard erroneous readings that can occur because the PLC
   ' firmware is constantly updating these register.  It is possible to
   ' observe the following transitions:
   '
   '   23:59:59 -> 23:59:00 -> 23:00:00 -> 00:00:00
   '
   if ((DM[301]=TIME[1]) AND (DM[302]=TIME[2]) AND (DM[303]=TIME[3])
      AND (DM[304] = DATE[1]) AND (DM[305] = DATE[2]) AND (DM[306] = DATE[3]))
     ' time and date registers are all valid
     ' log data with the saved values
     '
      return   ' job is done
   endif

   ' If we are here then there was a roll over between the reads of the
   ' TIME[] registers...so try it again
   '
endwhile
[/color][/font]

Best regards,

Gary D*ckinson
Title: Re:DATE update after midnight
Post by: JRobinson on November 16, 2016, 03:17:33 AM
For Support:

Firmware of 4 persistent offenders is r82A. Found some r84G devices not doing this. Doesn't always happen on r82A devices.

The PLCs send a timestamp to our propriety server, every 12 hours. The server compares the timestamp to local time and updates the PLC time if more than a few minutes out. It is all done in text as we do not require accurate time. The intention is to manage daylight saving and time-zones.

The problem only happens for timestamps between 00:00 and about 00:25 (HH:MM) after midnight. Each time, the DAY register hasn't yet updated. It looks like the DAY (DATE[3]) register is not updating until about 25 past midnight.

 

Title: Re:DATE update after midnight
Post by: JRobinson on November 16, 2016, 03:38:52 AM
For Gary,

I am an 'old guy' too, keeping 200 of the PLCs working.

Thanks for the comprehensive response. I really appreciate the time you have given this.

I don't think the problem is the update ripple. The log below is the time-stamps from an hour of responses  from midnight. As the log is sequential in time, as received, you can see the problem individuals.
;2016/11/15 23:57:20;
;2016/11/15 0:1:46;
;2016/11/16 0:2:57;
;2016/11/16 0:6:10;
;2016/11/16 0:8:58;
;2016/11/15 0:11:39;
;2016/11/16 0:12:11;
;2016/11/16 0:15:29;
;2016/11/16 0:16:16;
;2016/11/16 0:21:27;
;2016/11/16 0:22:7;
;2016/11/15 0:22:51;
;2016/11/16 0:22:33;
;2016/11/16 0:26:26;
;2016/11/16 0:28:33;
;2016/11/16 0:43:23;
;2016/11/16 0:52:10;
;2016/11/16 0:54:13;
;2016/11/16 0:54:1;
;2016/11/16 0:55:44;
;2016/11/16 0:57:29;
;2016/11/16 1:8:14;
That night there were 3

I will try some comparisons, but I suspect the problem isn't transitory but some mechanism in the firmware is delaying the date update till up to 25 minutes after midnight. I never see the problem after about 25 minutes after midnight.

The easy work-around is in the server, simply disbelieving timestamps that are close to exactly a day out for 30 minutes after midnight. I get another report at midday where this problem doesn't happen so can do real updates then.  It is, however, an effect that would be good to understand better.









Title: Re:DATE update after midnight
Post by: garysdickinson on November 16, 2016, 08:30:30 AM
Yes, younare not seeing an update ripple.  It would not occur in mid hour.

this makes me suspect that you are seeing the "wrinkle in time" problem. Periodically the PLC's system firmware reads the registers from the hardware realtime clock that is mounted on the little board that you installed on the FMD8888.  It is possible that you are catching some of time and date registers at one of these update points.

You will have to work with TRI on this issue.  However, my suggestion on reading the entire time and date registers twice in a row and observing that the values have not changed would detect and allow your PLC to not report corrupted time stamps.

I know that TRI has changed how/when the hardware RTC updates occur, but this is mostly undocumented.

Good luck,

Gary D*ckinson
Title: Re:DATE update after midnight
Post by: support on November 17, 2016, 12:56:17 PM
We have checked with engineering and what Gary has suggested is correct. The CPU periodically tried to update its RTC (both calendar and time registers) by loading the data from battery-backed RTC on the FRAM-RTC board as it has a more accurate crystal. In the older firmware the update occur during the hourly crossover from xx: 59:59 to (XX+1):00:00.

Unfortunately if the updates occur during the midnight crossover (from 23:59:59 to 0:00:00) when the calendar date also change, then a race condition could occur such that the FRAM-RTC may still have the previous day date during the RTC update but after the RTC update the clock already passed 0:00:00 and the calendar date was not incremented. The date will be corrected after another RTC update.

Engineering has modified the firmware since r82B to avoid such a scenario so PLCs with firmware >= r82B should not experience this issue. Your PLCs vith firmware r82A unfortunately are still running the old firmware. Also for firmware before r83 they can't be field upgraded by users so if you wish to upgrade please contact support@triplc.com to arrange for RMA.


There are however some workaround we can suggest as follow:

If your PLCs  do periodically update their RTC from internet time server, then you may not need the RTC to be updated from the backup RTC every hour. In that case you can simply disable the hourly update of the CPU's RTC from the backup RTC. This can be achieved using the following statement once:

   SETSYSTEM 253,2  ' disable hourly update from backup RTC

The backup RTC will still function to load the CPU's RTC when the PLC is power ON. The backup RTC values is also updated simultaneously when you synchronize the CPU's RTC to the Internet Time Server so it will maintain the same accuracy.

Note that once the SETSYSTEM 253,2 is run it will stop re-enabling the hourly update from the backup RTC until any of the TIME[n] or DATE[n] variable is written to. Writing to any of these variables automatically re-enable the periodic update from backup RTC. (So if you are updating the PLC's RTC from Internet Time Server you will need to re-run the SETSYSTEM 253,2 statement to disable the updates again.)

So your program could disable the update from FRAM-RTC before midnight crossover and re-enable after midnight crossover.  Usually you can monitor the RTC in your program and do the necessary, or you can also setup an Alarm interrupt to handle it automatically. I have attached a sample program demonstrating how to use the alarm interrupt to work around this issue.


Title: Re:DATE update after midnight
Post by: JRobinson on November 18, 2016, 12:53:23 AM
Thanks for the help.

I will implement the update disable method and see if that resolves the problem. We use the RTC module to provide non volatile clock ( and more FRAM) so only the power-up loading is important in our application.