Unexpected results from Qc3CalculateHash

[newwardDev]

We have an interface with a company that is request we send them a SHA-1 hash to validate the request being sent to them. I have found that in certain cases the hash being returned is not the expected result.

In the test case of hashing the value Y8xBv4vS3330034646474832741000000008995840 the program below that we have using Qc3CalculateHash returns the value as WC2JQXZbDvrs/skvss+/2+LtOA==.

The vendor we are working with stated that this value is incorrect and they were expecting WC2JQXZbDvrs/skvss+/2+LtOEA=. So we created a php program to return the SHA-1 hash of this program and we did receive the same results as what the vendor expected.

Now most of the time with other values to hash Qc3CalculateHash will return the expected results but occasionally it does not. Below is the code that we are using to produce the SHA-1 hash. This is running on a machine with V7r1 installed.

h DFTACTGRP(*NO) ACTGRP('AS') BNDDIR('COMBASE64') debug                 
                                                                        
                                                                        
 /copy *libl/qprotosrc,BASE64_H                                         
d Qc3CalculateHash...                                                   
d                 PR                  ExtProc('Qc3CalculateHash')       
d   InData                        *   value                             
d   IndataL                     10i 0 const                             
d   InDataF                      8a   const                             
d   AlgoDes                     16a   const                             
d   AlgoFmt                      8a   const                             
d   CryptoSP                     1a   const                             
d   CryptoDev                    1a   const options(*omit)   
d   Hash                        64a   options(*varsize:*omit)
d   ErrorCode                32767a   options(*varsize)      
d ALGD0500_t      ds                  qualified              
d                                     based(Template)        
d   HashAlg                     10i 0                        
                                                             
d QDCXLATE        PR                  ExtPgm('QDCXLATE')     
d   len                          5p 0 const                  
d   data                     32702a   options(*varsize)      
d   table                       10a   const                  
                                                             
d cvthc           PR                  ExtProc('cvthc')       
d   target                   65534A   options(*varsize)      
d   src_bits                 32767A   options(*varsize) const
d   tgt_length                  10I 0 value                  
                                                             
d ErrorNull       ds                  qualified              
d    BytesPro                   10i 0 inz(0)                 
d    BytesAvai                  10i 0 inz(0)                    
                                                                
d HASH_MD5        c                   1                         
d HASH_SHA1       c                   2                         
d HASH_SHA256     c                   3                         
d HASH_SHA384     c                   4                         
d HASH_SHA512     c                   5                         
                                                                
d data            s           2000A                             
d len             s             10i 0                           
d alg             ds                  likeds(ALGD0500_t)        
d bin             s             20a                             
d $hex            s             40a                             
 *****************************                                  
C     *entry        plist                                       
C                   parm                    InValue          47 
C                   parm                    OutValue         28 
                                                                
 /free                                                          
      data = InValue;                            
      len = %len(%trimr(data));                     
       alg.HashAlg = HASH_SHA1;                                               
    //Set the HASH Algorithm you want to use !                                
                                                                              
    //Convert from EBCDIC to ASCII (skip this step if you want Hash in EBCDIC)
     QDCXLATE(len: data: 'QTCPASC');                                          
    //API to calculate the SHA1 hash                                          
          Qc3CalculateHash( %addr(data)                                       
                          : len                                               
                          : 'DATA0100'                                        
                          : alg                                               
                          : 'ALGD0500'                                        
                          : '0'                                               
                          : *OMIT                                             
                          : bin                                               
                          : ErrorNull );                                      
                                                                              
  base64_encode( %addr(bin)   
  : %len(%trimr(bin))         
  : %addr(OutValue)           
  : %size(OutValue));         
dsply OutValue;               
     *inlr = *on;             
                              
/end-free

And here is the php program to hash the value

<?php
$stringtohash ='Y8xBv4vS3330034646474832741000000008995840';
$hash = sha1($stringtohash, true);
echo $signature = base64_encode($hash);
?>

If anyone has any suggestions as to what could be causing this issue it would be greatly appreciated.

Thank you
Bob

[Scott Klement]

Re: Unexpected results from Qc3CalculateHash

Most of the time when people complain about this it's because their data in RPG is EBCDIC in PHP it's ASCII. So they have different hashes because even though they look alike to human eyes, they're really very different.

Make sure the hex value of the data is the same in both cases.

[newwardDev]

Re: Unexpected results from Qc3CalculateHash

Thank you for your reply. I did look into the Hex value of the data within both programs and they are the same.

Results from running RPGLE program in debug and viewing hex value of data right after this statement QDCXLATE(len: data: 'QTCPASC');
EVAL data:x
00000 59387842 76347653 33333330 30333436 - ß.ÌâÎ.Îë........
00010 34363437 34383332 37343130 30303030 - ................
00020 30303038 39393538 34304040 40404040 - ..........
00030 40404040 40404040 40404040 40404040 -
00040 40404040 40404040 40404040 40404040 -
00050 40404040 40404040 40404040 40404040 -
00060 40404040 40404040 40404040 40404040 -
00070 40404040 40404040 40404040 40404040 -
00080 40404040 40404040 40404040 40404040 -
00090 40404040 40404040 40404040 40404040 -
000A0 40404040 40404040 40404040 40404040 -
000B0 40404040 40404040 40404040 40404040 -
000C0 40404040 40404040 40404040 40404040 -

Hex value of php value right after this statement $stringtohash ='Y8xBv4vS3330034646474832741000000008995840';
59387842763476533333333030333436343634373438333237 3431303030303030303038393935383430

So comparing them they both are the same
RPG
59 38 78 42 76 34 76 53 33 33 33 30 30 33 34 36 34 36 34 37 34 38 33 32 37 34 31 30 30 30 30 30 30 30 30 38 39 39 35 38 34 30
PHP
59 38 78 42 76 34 76 53 33 33 33 30 30 33 34 36 34 36 34 37 34 38 33 32 37 34 31 30 30 30 30 30 30 30 30 38 39 39 35 38 34 30

Does this seem to be a good test or do I need to do something else to compare the Hex values?

Thank you for your help

[newwardDev]

Re: Unexpected results from Qc3CalculateHash

Anyone have any other thoughts on this?

Any suggestions would be greatly appreciated.

[c0nfitty]

I'm having the exact same issue. I've verified the hex as well.
Did you ever figure this out?

[subha123]

Hello,

Qc3CalculateHash API currently supports below variations of hashing. It supports till SHA2 hashing algorithms. I was using HASH_SHA512 (SHA2-512) for calculating hash value till now. As per my new requirement I need now to have SHA3-512 hashing to be done. Can anyone help to know if i can achieve SHA3-512 hashing with this API and how can i do that?
d HASH_MD5 c 1 d HASH_SHA1 c 2 d HASH_SHA256 c 3 d HASH_SHA384 c 4 d HASH_SHA512 c 5 If not possible with this API, is there any other way to achieve this.

Thanks in advance.

[c0nfitty]

bumping so that this will show on "latest topics"

[Scott Klement]

c0nfitty, it'd be much easier to help you if we could reproduce the problem. Can you post a simple PHP and simple RPG program that calculate the hash so we can run them, see they aren't the same, and debug them?

[Scott Klement]

FWIW, the problem with the original code posted by newwardDev above is that he's running %TRIMR() on a binary field. That means if the data happens to end in x'40' (which it does it this case) it will remove the character, changing the result. %TRIM should never be used on a non-text field.

To fix it, change this:

  base64_encode( %addr(bin)   
               : %len(%trimr(bin)) // <-- Bug: Using %TRIM on binary field
               : %addr(OutValue)           
               : %size(OutValue));

To this:

  base64_encode( %addr(bin)   
               : %len(bin)        // <-- Will always be 20, which is correct.
               : %addr(OutValue)           
               : %size(OutValue));

[c0nfitty]

I'm actually using hash_sha256 but the issue is quite similar. I made it work but I'm not sure why it worked. The goal was to hash this canonical request for AWS SellingPartner API (pseudo code included)...

 
 CanonicalRequest =   HTTPRequestMethod + '\n' +   CanonicalURI + '\n' +   CanonicalQueryString + '\n' +   CanonicalHeaders + '\n' +   SignedHeaders + '\n' +   HexEncode(Hash(RequestPayload))     
 GET / Action=ListUsers&Version=2010-05-08 content-type:application/x-www-form-urlencoded; charset=utf-8 host:iam.amazonaws.com x-amz-date:20150830T123600Z  content-type;host;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  (HEX) 4745540D0A2F0D0A416374696F6E3D4C69737455736572732656657273696F6E3D323031302D30352D30380D0A 636F6E74656E742D747970653A6170706C69636174696F6E2F782D7777772D666F726D2D75726C656E636F6465 643B20636861727365743D7574662D380D0A686F73743A69616D2E616D617A6F6E6177732E636F6D0D0A782D61 6D7A2D646174653A3230313530383330543132333630305A0D0A0D0A636F6E74656E742D747970653B686F7374 3B782D616D7A2D646174650D0A6533623063343432393866633163313439616662663463383939366662393234 3237616534316534363439623933346361343935393931623738353262383535

I pasted the above request into notepad++ and analyzed the hex data. My issue was with the newline hex values. Ascii translates it to x'0D0A' so I defined that hex value as a constant in my code. I still don't understand why but the procedures ( I tried http_xlate and QCDXLATE) i used to convert to Ascii changed the hex values to x'0D1A'. I spent a lot of time playing with this.... trying to %scanrpl and force the hex value to match exactly what I saw in notepad++. In this case I was following a guide so I knew exactly what the hashed value should be. I finally found that if I define the newline like this x'0A'it worked perfectly. I did have to %scanrpl on it after the EBCDIC->Ascii conversion though. I still have no clue why.... The hex is different... See code below. Thanks for the help!

    h DFTACTGRP(*NO) ACTGRP('AS') BNDDIR('QC2LE':'HTTPAPI') debug

     d Qc3CalculateHash...
     d                 PR                  ExtProc('Qc3CalculateHash')
     d   InData                        *   value
     d   IndataL                     10i 0 const
     d   InDataF                      8a   const
     d   AlgoDes                     16a   const
     d   AlgoFmt                      8a   const
     d   CryptoSP                     1a   const
     d   CryptoDev                    1a   const options(*omit)
     d   Hash                        64a   options(*varsize:*omit)
     d   ErrorCode                32767a   options(*varsize)

     d ALGD0500_t      ds                  qualified
     d                                     based(Template)
     d   HashAlg                     10i 0

     d QDCXLATE        PR                  ExtPgm('QDCXLATE')
     d   len                          5p 0 const
     d   data                     32702a   options(*varsize)
     d   table                       10a   const

     d cvthc           PR                  ExtProc('cvthc')
     d   target                   65534A   options(*varsize)
     d   src_bits                 32767A   options(*varsize) const
     d   tgt_length                  10I 0 value

     d ErrorNull       ds                  qualified
     d    BytesPro                   10i 0 inz(0)
     d    BytesAvai                  10i 0 inz(0)

     d HASH_SHA256     c                   3

     d data            s           2000A
     d data2           s           2000A
     d len             s             10i 0
     d alg             ds                  likeds(ALGD0500_t)
     d bin             s             32a
     d $hex            s             64a
      /copy libhttp/qrpglesrc,httpapi_h
          dcl-pr HtC extproc('cvtch');
            *n pointer value;
            *n pointer value;
            *n int(10) value;
          End-Pr;

        dcl-s @hex char(1000);
        dcl-s new char(1) inz(x'0A');
        dcl-s i int(10);
      /free
       data = 'GET' + new +
              '/' +  new +
              'Action=ListUsers&Version=2010-05-08' + new +
              'content-type:application/x-www-form-urlencoded; charset=utf-8' +
                new +
              'host:iam.amazonaws.com' +  new +
              'x-amz-date:20150830T123600Z' + new + new +
              'content-type;host;x-amz-date' +  new +
         'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855';
         len = %len(%trimr(data));
         alg.HashAlg = HASH_SHA256;

             QDCXLATE(len:data: 'QTCPASC');
          len = %len(%trim(data));
            data = %Scanrpl(x'1A':x'0A':data);

         Qc3CalculateHash( %addr(data)
                      : len
                      : 'DATA0100'
                      : alg
                      : 'ALGD0500'
                      : '0'
                      : *OMIT
                      : bin
                      : ErrorNull );

      //Convert to HEX
         cvthc( $hex: bin: %len(bin)*2);
          *inlr = *on;
      /end-free

Here is the hex value after the conversion.

> EVAL data:x 500                                                         
     00000     4745540A 2F0A4163 74696F6E 3D4C6973   - åáè... ÄÈÑ?>.<ÑË   
     00010     74557365 72732656 65727369 6F6E3D32   - ÈíËÁÊË.îÁÊËÑ?>..   
     00020     3031302D 30352D30 380A636F 6E74656E   - ..........Ä?>ÈÁ>   
     00030     742D7479 70653A61 70706C69 63617469   - È.È`øÁ./øø%ÑÄ/ÈÑ   
     00040     6F6E2F78 2D777777 2D666F72 6D2D7572   - ?>.Ì.ÏÏÏ.Ã?Ê_.ÍÊ   
     00050     6C656E63 6F646564 3B206368 61727365   - %Á>Ä?ÀÁÀ..ÄÇ/ÊËÁ   
     00060     743D7574 662D380A 686F7374 3A69616D   - È.ÍÈÃ...Ç?ËÈ.Ñ/_   
     00070     2E616D61 7A6F6E61 77732E63 6F6D0A78   - ./_/:?>/ÏË.Ä?_.Ì   
     00080     2D616D7A 2D646174 653A3230 31353038   - ./_:.À/ÈÁ.......   
     00090     33305431 32333630 305A0A0A 636F6E74   - ..è......!..Ä?>È   
     000A0     656E742D 74797065 3B686F73 743B782D   - Á>È.È`øÁ.Ç?ËÈ.Ì.   
     000B0     616D7A2D 64617465 0A653362 30633434   - /_:.À/ÈÁ.Á.Â.Ä..   
     000C0     32393866 63316331 34396166 62663463   - ...ÃÄ.Ä.../ÃÂÃ.Ä  
     000D0     38393936 66623932 34323761 65343165   - ....ÃÂ...../Á..Á
     000E0     34363439 62393334 63613439 35393931   - ....Â...Ä/......
     000F0     62373835 32623835 35404040 40404040   - Â....Â...

[c0nfitty]

by the way.... do you know of a way to define the QC3CalculateHash in free form?
What does the '...' represent and how in the heck can the object name be over 10 characters?
Thanks again

[Scott Klement]

I'll see if I can answer your questions. Let me know if I miss anything

Q1) Why does QDCXLATE convert x'0D0A' to x'0D1A'

A1) x'0D' is the ASCII code for carriage return (CR), and x'0A' is the ASCII code for linefeed (LF). The problem is that the QDCXLATE API is expecting EBCDIC input, not ASCII. The CR character is not a problem because the EBCDIC code for CR happens to be the same as the ASCII one, they are both x'0D'. However, LF ix x'25' in EBCDIC and x'0A' in ASCII. Since its expecting EBCDIC input, you'd need to use x'0D25' to get x'0D0A' as the ASCII output. You may know that the data was already ASCII, but the computer doesn't... and it'll try to translate it.


Q2) I finally found that if I define the newline like this x'0A'it worked perfectly. I did have to %scanrpl on it after the EBCDIC->Ascii conversion though. I still have no clue why...

A2) Same as A1, above. All of your text is in EBCDIC, except this one character x'0A'. But, you're telling it to translate the whole thing from EBCDIC to ASCII. So even though you and I may know that the x'0A' was already ASCII, the computer doesn't. It's looking for the EBCDIC character x'0A' (which isn't the linefeed, aka newline, character!) and converting it to the ASCII equivalent. Specify x'25' (the EBCDIC linefeed, aka newline) instead.

The reason that the one character x'25' (or x'0A') may be preferred vs. x'0D25' (or x'0D0A') is because some environments (notably, Windows) expects CRLF for newline, whereas others (Unix, MacOS, Android, iOS... almost everything aside from Windows) expects just LF for newline. Since AWS is typically based on Linux, I would expect LF by itself to be correct. For the hash to match, it's critical that the underlying hex value matches, so you need to match what the other computer is expecting exactly.


Q3) by the way.... do you know of a way to define the QC3CalculateHash in free form?

A3) Yes, its no different than converting any other prototype, and is explained in the ILE RPG reference manual. Unfortunately, I'm short on time right now but give it a try and see if you can figure it out... if not, let me know, and I'll post an example when I have more time.


Q4) What does the '...' represent and how in the heck can the object name be over 10 characters?

A4) Are you thinking of the 10 character limitation on system object names? Qc3CalculateHash is an ILE procedure name, not an object name. Procedure names max out at 4096 characters. The '...' in fixed format is used to let you write longer names, and to continue them on the next line.