Be careful when using the ** operator

[Barbara Morris]

The following statement has a potential problem.

numVeld = intVeld / (10 ** dsFFD(ix).decPos);

It uses the ** operator, which returns a float value. Having one operand with a float value means that the entire right-hand-side expression returns a float value. When a float value is assigned to a decimal value (a non-float value), the result is truncated rather than rounded. RPG uses binary floating point, which can't hold every decimal value exactly. So a value that should be say 1.234 might have a floating pointvalue of 1.23999999, which would give 1.233 instead of 1.234 in the result.

One way to solve the rounding problem is to use EVAL(H):

eval(h) numVeld = intVeld / (10 ** dsFFD(ix).decPos);

Or, what I would do, calculate the power-of-10 separately. Define "precision" big enough to hold the maximum power of 10 you might need.

EVAL(H) precision = 10 ** dsFFD(ix).decPos;
numVeld = intVeld / precision;

[mlopez01]

Hi, Barbara.

I know this is old but couldn't you also use the %int() BIF?

numVeld = intVeld / %int(10 ** dsFFD(ix).decPos);

[Vectorspace]

I missed this the first time round - thanks Barbara! I have used this method before, I never knew about the pitfall.

[Barbara Morris]

@mlopez, use %INTH, not %INT. %INT doesn't round a float value, so %INT(4.999999999) would give 4. Maybe there is no issue with 10**x, but it's best to always use half-adjust when converting floating point to decimal.

Also, if decpos is bigger than say 16, you should use %DECH instead of %INTH, since %INTH(10**large_number) would overflow. I'm not sure whether the limit is 16 or 17.