Truncating or rounding with least significant integer digit.

[K.Newton]

Is there a supported TBasic command for this? I don't really care about the error due to truncating but rounding would be better if it's efficient. Either way would be better than what I have now....

Anybody have advise on this for a simp faking it as a programmer?

[support]

Do you mean you wish to round off the least signifcant digit. Meaning if a number is 12344 you want it to round to 12340 and if it is 12345 you want it to round to 12350?

If that is the case, assuming the number is in stored A

A  = (A + 5)/10 * 10

The number in A will be rounded to the nearest 10.

[K.Newton]

Ah that would be a good way to round. Simple yet effective. Clever, thank you. After I posted I found a similar method to truncate, ie: A =(A/10)*10.

Unfortunately, I think my number must be too big or something as it is exploding when I try these types of things. Even when splitting up into seperate lines and/or variables, it still explodes. I should have mentioned in my first post that this number is an 9 digit 32 bit number.

Even this wont do it (500002816 is just a random example I made up), it still explodes:

A = 500002816
B = A/10
C = A * 10

I did this which is working but sure feels like a hack.

A = 500002816
B$ = STR$(A,9)
C$ = MID$(B$,1,8)
D = VAL(C$)

It's working but there must be a better way??? At this point rounding would be stupendous but a clean method of truncating would be a start.

[support]

You made a mistake. It should be:

A = 500002816
B = A/10
C = B * 10

I tested it and it worked.

A simpler statement with rounding is:

A = (A+5)/10*10

I tested it using simulator and transferred to a F-series PLC and works OK on both.

[K.Newton]

Good catch on my error. I just rewrote it wrog the original had it right. I am stumped.

Here is my actual code which isn't working. Elsewhere DM[93] is set to 2500 and DM94 is 21, ie just scaler values. Note the comments in the code below. This version wont work but removing the comments and adding one to the line L=K/10 fixes it and it works fine.

K = DM[992] * 122071    'convert ADC counts to Volts
'L$ = STR$(K,9)
'G$ = MID$(L$,1,5)
'L = VAL(G$)
L = K/10
M = L - DM[93]
N = M * DM[94]/100
DM[131] = N

[support]

The maximum a 32-bit integer can handle is 2^31-1. If any of your computation exceeds 32-bit integer range then it results in integer overflow and computational result will be wrong.

DM[93] = 2500
DM[94] = 21
DM[992] = 4096

K = DM[992] * 122071    'convert ADC counts to Volts
L = K/10
M = L - DM[93]
N = M * DM[94]/100
DM[131] = N


I got these values, seems OK to me except for DM[131], which is a 16-bit variable and is unable to hold the value of N.

[K.Newton]

Indeed. I see it clearly now. Thanks for the insight.

Perhaps, though I didn't know it at the time I first made this thread, my question should have been: What is the proper technique for stripping 32 bit values for storage in a 16 bit location once the 32 bit math is done?

Edit: Wait a minute! Where is my calculation exceeding 2^31-1? My biggest value is 122071 *4096 = 500002816 which is within the limits. Every step from there it just gets smaller with divides or subtractions. Is the divide causing it to go over 2^31-1?

[support]

when you assign the value of a 32-bit variable to 16-bit, only the lower 16-bit of the 32-bit variable are transferred.

If you want to store the upper 16-bit of a 32-bit variable to 16-bit, you can use the GETHIGH16 function.

Regarding integer overflow, I am referring to your original source:

A = 500002816
B = A/10
C = A * 10
==> C = 5,000,028,160 > 2^31-1.

And that's why the resulting value in C was wrong.

[K.Newton]

Man I really have seen the light this time and am feeling sufficiently lame for missing it all this time. Thank you for the help Mr. Mod.