Author Topic: Are there limitations to DM significant digits?  (Read 11502 times)

TravisB

  • Newbie
  • Posts: 33
  • I am NOT a llama!
    • View Profile
Are there limitations to DM significant digits?
« on: April 03, 2012, 01:51:14 PM »
I am using an FMD series PLC to read a pressure sensor. In order to get the number of significant digits I need for accuracy, I have to use very large numbers (1,000,000.) My proceedure works when using "letter variables" but not when using DM[] variables. I thought there might be a limit to the number of significant digits for a DM[] variable, but I can't find any mention of this, so now I'm confused.

Sample:
'---Read Main Tank water level
A = ADC(1)       'Tank sensor raw output
DM[11] = ADC(1)
'Calibrate the reading
'These sensors output values on the order of 1000 / foot,
' so to get acuracy of 1/10th of a foot we need to convert to
' 1/1,000,000 to calibrate, then convert back to 1/100th of a
' foot before moving on in the program
B = 2658             'Slope (1/1,000,000 feet)
DM[12] = 2658
C = -2120000             'Intercept (1/1,000,000 feet)
DM[13] = -2120000
D = B * A + C     'Calibrated water level (1/1,000,000 feet)
DM[14] = DM[12] * DM[11] + DM[13]
'Convert back to 1/100th using MOD to trim unwanted sig.dig.
DM[100] = (D-(D MOD 10000))/10000
DM[101] = (DM[14]-(DM[14] MOD 10000))/10000
DM[102] = ((DM[12] * DM[11] + DM[13])-((DM[12] * DM[11] + DM[13]) MOD 10000))/10000
D = DM[100]       'Calibrated water level (1/100 feet)

"D" gives me the correct value, DM[101] doesn't get calculated, and DM[102] returns an incorrect value. (They should all be equal.)
Do DM[] variables keep fewer significant digits that "A, B, C" type variables?

TravisB

  • Newbie
  • Posts: 33
  • I am NOT a llama!
    • View Profile
Re:Are there limitations to DM significant digits?
« Reply #1 on: April 03, 2012, 02:52:47 PM »
To answer my own question:
Variables A to Z are 32 bits in length, allowing -2^31 to 2^31 while DM[] variables allow half the size, numbers -32768 to 32768.

Thanks TravisB

support

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3174
    • View Profile
    • Internet Programmable PLCs
Re:Are there limitations to DM significant digits?
« Reply #2 on: April 04, 2012, 03:32:54 PM »
Besides A to Z there is an array of 32-bit variables EMLINT[1] to EMLINT[16].  These 16 variables can store the same range of results as A to Z.

Note that there are commands such as SETHIGH16 and GETHIGH16 for you to transfer the data from 16-bit variables to 32-bit variables.
Email: support@triplc.com
Tel: 1-877-TRI-PLCS