I am using an FMD series PLC to read a pressure sensor. In order to get the number of significant digits I need for accuracy, I have to use very large numbers (1,000,000.) My proceedure works when using "letter variables" but not when using DM[] variables. I thought there might be a limit to the number of significant digits for a DM[] variable, but I can't find any mention of this, so now I'm confused.
Sample:
'---Read Main Tank water level
A = ADC(1) 'Tank sensor raw output
DM[11] = ADC(1)
'Calibrate the reading
'These sensors output values on the order of 1000 / foot,
' so to get acuracy of 1/10th of a foot we need to convert to
' 1/1,000,000 to calibrate, then convert back to 1/100th of a
' foot before moving on in the program
B = 2658 'Slope (1/1,000,000 feet)
DM[12] = 2658
C = -2120000 'Intercept (1/1,000,000 feet)
DM[13] = -2120000
D = B * A + C 'Calibrated water level (1/1,000,000 feet)
DM[14] = DM[12] * DM[11] + DM[13]
'Convert back to 1/100th using MOD to trim unwanted sig.dig.
DM[100] = (D-(D MOD 10000))/10000
DM[101] = (DM[14]-(DM[14] MOD 10000))/10000
DM[102] = ((DM[12] * DM[11] + DM[13])-((DM[12] * DM[11] + DM[13]) MOD 10000))/10000
D = DM[100] 'Calibrated water level (1/100 feet)
"D" gives me the correct value, DM[101] doesn't get calculated, and DM[102] returns an incorrect value. (They should all be equal.)
Do DM[] variables keep fewer significant digits that "A, B, C" type variables?