Tweet
I'm doing the advent of code on the iSeries; which is fun and brings some unique challenges.
I recently came across an issue with my program that I couldn't explain - I checked the question and the code literally 20 times and it was doing what it was supposed to.
After many expletives, pulling out my hair and checking individual lines; I finally tracked it down to the operator: **=
I used this operator to square a number as per; n **= 2. In some instances, i.e. for the below example number, the result of this is not the same as n = n * n.
Was just wondering if anyone had thoughts on why? My guess would be something to do with how it's being stored whilst being calculated, maybe decimal positions are being used and as a result things are being dropped? Has been a right PITA!
Example code to replicate;
I recently came across an issue with my program that I couldn't explain - I checked the question and the code literally 20 times and it was doing what it was supposed to.
After many expletives, pulling out my hair and checking individual lines; I finally tracked it down to the operator: **=
I used this operator to square a number as per; n **= 2. In some instances, i.e. for the below example number, the result of this is not the same as n = n * n.
Was just wondering if anyone had thoughts on why? My guess would be something to do with how it's being stored whilst being calculated, maybe decimal positions are being used and as a result things are being dropped? Has been a right PITA!
Example code to replicate;
Code:
d n s 20p 0
d x s 20p 0
d y s 20p 0
d z s 20p 0
/free
n = 3740147530;
x = n;
x **= 2;
y = n ** 2;
z = n * n;
dsply x; // 13988703546165100544 - wrong
dsply y; // 13988703546165100544 - wrong
dsply z; // 13988703546165100900 - right
return;
Comment