ibmi-brunch-learn

Announcement

Collapse
No announcement yet.

Hashing the input string with SHA-1 and a RSA private Key.

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Hashing the input string with SHA-1 and a RSA private Key.

    Hi,

    I have a program to digitally sign records for tax porposes.

    The program uses Qshell's Openssl and although it works it is quite slow with many records to sign. On average I have about 70,000 records with peaks of 90,000.
    My CL:

    QSH CMD('openssl dgst -sha1 -sign "$_KeyPrivFile" +
    -out "$_Hash1" "$_Record" ; openssl enc +
    -base64 -in "$_Hash1" -out "$_Hashb64" -A')

    $_KeyPrivFile is my RSA Private File on IFS
    $_Hash1 is the Hash in Binary
    $_Record is my string that i want to Sign.
    $_Hashb64 is the Hash in base64 encoding

    I'm looking for some API or RPG/RPGLE program to replace OpenSSL.

    Can anyone indicate the right API or post an example (OS V7R3)?

  • #2
    If I understand correctly (and I'm only just getting into this myself) I think what you may be looking for is the Qc3CalculateHMAC API. If so IBM have a full RPG example here: https://www.ibm.com/support/pages/qc...ac-api-example

    Comment


    • #3
      If you're using a SHA-1 shared key (which your openssl command suggests you are) then Qc3CalculateHMAC sounds right. If you want to sign with an actual RSA private key, then you need Qc3CalculateSignature.

      You'll also need a solution to convert the binary output to base64. One option is the BASE64ENCODE function in SYSTOOLS. If that doesn't work for you, Scott Klement also wrote a library.

      Comment


      • #4
        First of all thank you very much for the answers.
        Looking at the IBM documentation (which is never very clear...) of the Qc3CalculateSignature api "API produces a digital signature by hashing the input data and encrypting the hash value using a public key algorithm (PKA)." I think this is not what I want since I have a string and the RSA private key and I want to get a hash (with SHA-1 algorithm) using both (the string and the RSA private key).

        This hash will be sent along with other invoice information to the tax people. Then they with the RSA public key (which I made available to them) and with the same invoice data (the string is made with some invoice data, so is the same string I use) will check if the Hash I produced for that invoice is correct or not.

        So, looking at the Calculate HMAC example (thanks JonBoy) I need to replace:

        d*SHA_256 c const (3)
        d SHA_1 c const (2)

        and all references to SHA_256 by SHA_1.

        So I think I need to do this Steps:

        - Convert the string to CCSID 819 (in my CL I add/replace the QIBM_CCSID environment variable to 819 before the QSH command).

        - Read my RSA private key (IFS file) into a String. <<< The CalculateHMAC API needs a String

        - Convert the string (with the private key) to CCSID 819. <<< not sure I need this

        - call the Calculate HMAC API.

        - Convert the binaryHMAC field to base64 (Thanks Martin for providing 2 solutions).

        - Convert Base64HMAC to EBCDIC (ccsid 37) <<< not sure I need this but once the Hash was generated with the string with ccsid 819 and I need to save the Hash at 37

        Please correct me in the necessary steps if something is missing or not correct ...

        Thanks in advance,
        Rui

        Comment


        • #5
          Thanks for adding more detail. I'm convinced now that you actually need Qc3CalculateSignature.

          Calculating an HMAC uses a hashing algorithm plus a symmetric key. If you want to use RSA for signing, then the signature is produced by combining the RSA algorithm with a separate hashing algorithm. I don't understand the distinction between all the different schemes for RSA signing (e.g. RSS-PSS) but this API was enough for our use case of signing JWTs with an RSA key plus SHA-256.

          If you take a look through the parameters for the API, you'll see that the Algorithm Description format ALGD0400 allows you to specify both the public key algorithm (50 = RSA) and the hash algorithm (2 = SHA-1).

          You would still use Key Description format KEYD0200, like in the example Jon linked, if you want to specify the RSA private key as a literal value. The biggest differences will be that myKey.Fmt would be '1' instead of '0', and myKey.Value would be the BER format (a.k.a. DER format) representation of your RSA private key. If your key file starts with the text "-----BEGIN RSA PRIVATE KEY-----") then it is in PEM format and you will need to convert it - the openssl command should be capable of doing that, though I don't have the exact options to hand.

          Here's roughly the code we wrote for our JWT implementation. Note that we put the RSA key pair into an IBM i Cryptographic Keystore File to protect it, so we use key description KEYD0400 to make a reference to the key. Still, this might be a useful starting point for you.

          Code:
          // Stuff /included from a header member
          dcl-c HASH_ALGORITHM_SHA1 2;  
          dcl-c HASH_ALGORITHM_SHA256 3;
          
          dcl-c CRYPTO_ALGORITHM_RSA 50;
          
          dcl-ds t_ALGD0400 len(12) qualified template;
            publicKeyCipherAlgorithm int(10) pos(1);
            pkaBlockFormat char(1) pos(5);
            signingHashAlgorithm int(10) pos(9);
          end-ds;
          
          dcl-ds t_KEYD0400 len(56) qualified template;
            keystoreFile char(10);
            keystoreLibrary char(10);
            recordLabel char(32);
          end-ds;
          
          dcl-pr Qc3CalculateSignature extproc(*dclcase);
            inputData char(30000) options(*varsize) const;
            inputDataLength int(10) const;
            inputDataFormat char(8) const;
            algorithmDescription char(32000) options(*varsize) const;
            algorithmDescriptionFormat char(8) const;
            keyDescription char(30000) options(*varsize) const;
            keyDescriptionFormat char(8) const;
            cryptoServiceProvider char(1) const;
            cryptoDeviceName char(10) const;
            signature char(512) ccsid(*hex) options(*varsize);
            signatureLengthProvided int(10) const;
            signatureLengthReturned int(10);
            errorCode like(t_qusec);
          end-pr;
          
          
          // Adapted from a service program procedure
          
          dcl-s payload varchar(10000) ccsid(*hex)
          dcl-ds keyDescription likeds(t_KEYD0400);
          dcl-ds algorithmDescription likeds(t_ALGD0400);
          dcl-ds qusec likeds(t_qusec) inz;
          dcl-s signature char(512) ccsid(*hex);
          dcl-s signatureLength int(10);
          
          keyDescription = *allx'00';
          keyDescription.keystoreFile = 'KEYSTORE'
          keyDescription.keystoreLibrary = '*LIBL';
          keyDescription.recordLabel = 'RSA_KEY_LABEL';
          
          algorithmDescription = *allx'00';
          algorithmDescription.publicKeyCipherAlgorithm = CRYPTO_ALGORITHM_RSA;
          algorithmDescription.pkaBlockFormat = '1';
          algorithmDescription.signingHashAlgorithm = HASH_ALGORITHM_SHA256;
          
          Qc3CalculateSignature(
            payload :
            %len(payload) :
            'DATA0100' :
            algorithmDescription :
            'ALGD0400' :
            keyDescription :
            'KEYD0400' :
            ANY_CSP :
            '' :
            signature :
            %size(signature) :
            signatureLength :
            qusec
          );

          Comment


          • rmmcf@clix.pt
            rmmcf@clix.pt commented
            Editing a comment
            Thanks for the answer.
            Yes my key starts with "----- BEGIN RSA PRIVATE KEY -----" and ends with "-----END RSA PRIVATE KEY-----" and is in .PEM format

            However, I can transform the key into .DER with openssl:
            'openssl rsa -in "$ _KeyPrivFile" -outform DER -out "$ _KeyPrivDERFile"'

            I think .DER will be the same as .BER and is the key in Binary right?

            So, if I use Qc3CalculateSignature:
            KEYD0200 with:
            Key Type = 51 (RSA Private Key)
            Key Format = 1 (BER String)
            Key String = My .DER RSA Private Key

            and ALGD0400 with:
            Public key cipher algorithm = 50 (RSA)
            PKA block format <<< What value to put here?
            Signing hash algorithm = 2 (SHA-1)

            However, I'm confused by the API to use...
            Isn't Qc3EncryptData a better and simpler solution?

            In the meantime, I'm on vacation for 2 weeks, so I only answer at that time.
            Thank you

        • #6
          PKA block format <<< What value to put here?
          This influences the algorithm used for combining your hashing algorithm with your RSA key. There's a little more info here:



          In our own code we use block type 1, which corresponds to the RSASSA-PKCS1-v1_5 scheme. Block type 0 seems to refer to an older scheme that has since been dropped from the PKCS#1 standard.

          Isn't Qc3EncryptData a better and simpler solution?
          I am reading this as you suggesting that you encrypt the HMAC produced through Qc3CalculateHMAC? Obviously you could encrypt the tax data itself with your private key and then you wouldn't have any reason to create a signature.

          I think combining two API calls in this way is more complicated due to the additional code, but worse, I don't think it's equivalent to what your openssl command does. This page has a similar command string to yours, and it shows how it uses RSASSA-PKCS1-v1_5 including some padding operations. My opinion is that Qc3CalculateSignature seems like your best bet to recreate that functionality.

          Comment


          • rmmcf@clix.pt
            rmmcf@clix.pt commented
            Editing a comment
            Hi,
            I had already done openssl to convert ".pem" to ".der" but using the command:
            openssl rsa -in "$ _KeyPrivFile" -outform DER -out "$ _KeyPrivDERFile"

            So, I made the command that you say and tried ADDCKMKSFE again, but it still gives the same error.

            On the other hand, the ADDCKMKSFE command with KEYTYPE = *RSAPRV only allows FORMAT = *BER

            I don't know how to solve this problem ...

          • MartinTosney
            MartinTosney commented
            Editing a comment
            Okay, I read the docs for ADDCKMKSFE and they say it needs to be in PKCS #8 format. So I had a play around, and this conversion works for me:

            openssl pkcs8 -inform PEM -outform DER -in private.pem -out private.der -topk8 -nocrypt

            Hopefully that works for you too!

          • rmmcf@clix.pt
            rmmcf@clix.pt commented
            Editing a comment
            Hi,
            Yes, this works for me too... Now i have my key on keystore file.

            I'm back to your code and adapting it to my needs. I have some doubts:
            1) I have an input string (your 'payload' field) (EBCDIC). Will I have to convert to ASCII before using Qc3CalculateSignature?
            2) Your payload is defined in *hex. It is mandatory that the string be in *hex before using Qc3CalculateSignature?
            3) After the Qc3CalculateSignature call, does the 'signature' field come in *Hex?
            4) In my case I need to convert the 'signature' to Base64. Have you an example of how to use the Base64 Scott Klement tool? (i have download the sources already...)

            Many thanks,
            Rui

        • #7
          If you are on Release 7.3 or 7.4 on the latest TR you may have also a look whether the HASH_ROW SQL Scalar function will deliver what you need.

          Birgitta

          Comment


          • rmmcf@clix.pt
            rmmcf@clix.pt commented
            Editing a comment
            Hi Birgitta,
            Thanks for your reply.

            I'm on V7R3 but I can't find that SQL Scalar function.
            Then, how do I call this function from an RPGLE?
            Regards

        • #8
          Hi Rui, have you made progress on this? I've gotten to the point of creating a signature, but the signature does not match the signature that jwt.io creates for the same payload. A request to the web service using jwt.io's signature works, but a request using my signature is rejected by the web service. So I know my signature is not right somehow. At first I thought there might be a problem with my base64URL-encoding of the signature, but I decoded jwt.io's signature and compared its hex codes to the hex codes of my signature, and they are completely different. So I'm pretty sure my signature is wrong.
          Like questions 2 and 3 in your December comments, I noticed the ccsid(*hex) in Martin's code, and I did a bunch of experimentation with that versus leaving it as EBCDIC, etc., but nothing seemed to help.
          Anything you can relate from your experience with this would be great. Thanks!

          Comment


          • rmmcf@clix.pt
            rmmcf@clix.pt commented
            Editing a comment
            Hi,
            Yes, I used the "Qc3CalculateSignature" API and it worked for me.

            Basically I used the code that MartinTosney posted (Many thank's Martin).

            DATA0100 Format (Input String must be converted to Hex)
            ALGD0400 Format (I use HASH_SHA1 instead of HASH_SHA256)
            KEYD0400 Format (with a KeyStore)

            At the End i used base64_encode to Convert signature from * hex to Base64

            A note: to use KeyStores, your key must be in *BER Format (.ber is like .der).
            I used openssl to convert my key because it was .pem

            Rui

          • BAndrewsRTC
            BAndrewsRTC commented
            Editing a comment
            Finally got it - it was GIGO through Qc3CalculateSignature. As you suggested in question #1 of your December comment, I needed to convert the payload from EBCDIC to ASCII, since JWT and web services live in an ASCII world. As soon as I did that, the signature came out correct. Still having trouble getting the resulting HTTP request accepted by the web service, apparently something to do with my UTF-8 conversion, but the signature is good.

        • #9
          Hello Rui,
          i'm also around this subject, creating an HASH for Tax purposes (Portuguese TAX) , i've the private key, i've been reading this post from top to bottom but i can't really understand how to do this,
          in RPG, as i dont really know about encription, i dont know if you can help me on this.
          Thank you in advance.

          Best regards
          Helder

          Comment


          • rmmcf@clix.pt
            rmmcf@clix.pt commented
            Editing a comment
            Hello,
            Without knowing specifically what your doubts are, it's difficult...
            I will try to briefly explain the process:
            1) You have an input String that you want to sign. As it is intended to be Ascii the 1st thing to do is convert this input string from ebcdic to ascii.

            2) The Qc3CalculateSignature API uses a Hex String as Input, so you have to convert the ascii String resulting from the previous step to Hex. You also need a private key stored in a KeyStore that will have to be passed to the Qc3CalculateSignature API (KEYD0400 format).

            3) Call the Qc3CalculateSignature API:
            Qc3CalculateSignature(
            Hex_string :
            %len(Hex_String) :
            'DATA0100' :
            algorithmDescription :
            'ALGD0400' :
            keyDescription :
            'KEYD0400' :
            cryptoSvcProv :
            '' :
            signature :
            %size(signature) :
            signatureLength :
            t_qusec );

            4) Finally, convert the resulting signature from Hex to Base64 using "base64_encode" referred to in the previous comments.

            Hope this helps...

        • #10
          x

          Comment


          • HMAURICIO
            HMAURICIO commented
            Editing a comment
            Hello Rui,
            first of all thank yo for your help, when the Qc3CalculateSignature API runs it returns an error (CPF9DE7 Key type not valid), i wondering
            if you've the some error at any time?
            Could this error appear because the key was not correctly created in the keystore?
            My key in the key store as the type SHA-1 HMAC , i've tried to create it as RSA, but when i save it it shows me the error message
            'key not created. The kwy sequence or the Diffie-Hellman parameter not válid.' (Falha na criação da chave. A falha deve-se ao seguinte. A sequência de chave ou a sequência do parâmetro Diffie-Hellman não é válido.)
            Can you give me somo help here ?
            Thank you !
            Helder

        • #11
          Thank you so much for your help Rui, i'll try to put it to work !
          Best regards

          Comment


          • #12
            Hello Rui,
            first of all thank yo for your help, when the Qc3CalculateSignature API runs it returns an error (CPF9DE7 Key type not valid), i wondering
            if you've the some error at any time?
            Could this error appear because the key was not correctly created in the keystore?
            My key in the key store as the type SHA-1 HMAC , i've tried to create it as RSA, but when i save it it shows me the error message
            'key not created. The kwy sequence or the Diffie-Hellman parameter not válid.' (Falha na criação da chave. A falha deve-se ao seguinte. A sequência de chave ou a sequência do parâmetro Diffie-Hellman não é válido.)
            Can you give me somo help here ?
            Thank you !
            Helder

            Comment


            • #13
              Originally posted by HMAURICIO View Post
              Hello Rui,
              first of all thank yo for your help, when the Qc3CalculateSignature API runs it returns an error (CPF9DE7 Key type not valid), i wondering
              if you've the some error at any time?
              Could this error appear because the key was not correctly created in the keystore?
              My key in the key store as the type SHA-1 HMAC , i've tried to create it as RSA, but when i save it it shows me the error message
              'key not created. The kwy sequence or the Diffie-Hellman parameter not válid.' (Falha na criação da chave. A falha deve-se ao seguinte. A sequência de chave ou a sequência do parâmetro Diffie-Hellman não é válido.)
              Can you give me somo help here ?
              Thank you !
              Helder
              Hi,
              Yes, I had some problems using the KeyStore but I don't remember if it was this message. See post #6.1 to #6.5.
              Surely this message has to do with the type of key...

              Regarding the keys, what I did was use openssl to generate the public and private keys (below you have some parts of the code):
              /* PRIVATE FILE: */
              /* Encrypt Type: RSA ; Key bit length = 1024 bits */
              ADDENVVAR ENVVAR('_KeyPrivFile') VALUE(&KEY_PRIV_F) +
              REPLACE(*YES)

              STRQSH CMD('openssl genrsa -out "$_KeyPrivFile" 1024')

              /* Convert Private Key from .pem to .der (.ber) */
              /* To use the KeyStore it must be in PKCS #8 format */
              STRQSH CMD('openssl pkcs8 -topk8 -inform PEM +
              -outform DER -in "$_KeyPrivFile" +
              -out "$_KeyPrivDERFile" -nocrypt')

              /* PUBLIC FILE: */
              /* Encrypt Type: RSA ; Create Public Key based on Private */
              ADDENVVAR ENVVAR('_KeyPubFile') VALUE(&KEY_PUB_F) +
              REPLACE(*YES)

              STRQSH CMD('openssl rsa -in "$_KeyPrivFile" -out +
              "$_KeyPubFile" -outform PEM -pubout')

              Don't forget to authorize the resulting files.
              $_KeyPrivFile and $_KeyPubFile are IFS directories

              Regards,
              Rui

              Comment


              • #14
                Originally posted by rmmcf@clix.pt View Post

                Hi,
                Yes, I had some problems using the KeyStore but I don't remember if it was this message. See post #6.1 to #6.5.
                Surely this message has to do with the type of key...

                Regarding the keys, what I did was use openssl to generate the public and private keys (below you have some parts of the code):
                /* PRIVATE FILE: */
                /* Encrypt Type: RSA ; Key bit length = 1024 bits */
                ADDENVVAR ENVVAR('_KeyPrivFile') VALUE(&KEY_PRIV_F) +
                REPLACE(*YES)

                STRQSH CMD('openssl genrsa -out "$_KeyPrivFile" 1024')

                /* Convert Private Key from .pem to .der (.ber) */
                /* To use the KeyStore it must be in PKCS #8 format */
                STRQSH CMD('openssl pkcs8 -topk8 -inform PEM +
                -outform DER -in "$_KeyPrivFile" +
                -out "$_KeyPrivDERFile" -nocrypt')

                /* PUBLIC FILE: */
                /* Encrypt Type: RSA ; Create Public Key based on Private */
                ADDENVVAR ENVVAR('_KeyPubFile') VALUE(&KEY_PUB_F) +
                REPLACE(*YES)

                STRQSH CMD('openssl rsa -in "$_KeyPrivFile" -out +
                "$_KeyPubFile" -outform PEM -pubout')

                Don't forget to authorize the resulting files.
                $_KeyPrivFile and $_KeyPubFile are IFS directories

                Regards,
                Rui
                Hello Rui,
                once again thankyou for your help.

                Now i have another situation, the signatures is returned by the Qc3CalculateSignature correctly, but when
                the conversion to base64 occurs, some of the hashcodes are wrong, i've tried to convert to Base64 with
                EXEC SQL SET : hashg = SYSTOOLS.BASE64ENCODE(Trim(:signature));
                and with the Scott Klement code, but the result is the same, some of the Hashs are wrong after the conversion !

                i'm converting the field signature returned by the API in *Hex, some of the Bad Hashs just have the last 3 characters wrong, the rest
                of it is Ok, i dont know if you can help me on this ?

                Thankyou so much

                Best Regards
                Helder

                Comment


                • #15
                  Originally posted by HMAURICIO View Post

                  Hello Rui,
                  once again thankyou for your help.

                  Now i have another situation, the signatures is returned by the Qc3CalculateSignature correctly, but when
                  the conversion to base64 occurs, some of the hashcodes are wrong, i've tried to convert to Base64 with
                  EXEC SQL SET : hashg = SYSTOOLS.BASE64ENCODE(Trim(:signature));
                  and with the Scott Klement code, but the result is the same, some of the Hashs are wrong after the conversion !

                  i'm converting the field signature returned by the API in *Hex, some of the Bad Hashs just have the last 3 characters wrong, the rest
                  of it is Ok, i dont know if you can help me on this ?

                  Thankyou so much

                  Best Regards
                  Helder

                  Hi,
                  Yes, I had the same problem with the last 2 characters.

                  This is because you have to pass to the Base64 function not the string size (fixed size) but the size returned in the Qc3CalculateSignature function witch returns the actual length of the produced signature in the 12th parameter – the "signatureLength" variable in my case.

                  Qc3CalculateSignature(
                  Hex_string :
                  %len(Hex_String) :
                  'DATA0100' :
                  algorithmDescription :
                  'ALGD0400' :
                  keyDescription :
                  'KEYD0400' :
                  cryptoSvcProv :
                  ' ' :
                  signature :
                  %size(signature) :
                  signatureLength :
                  t_qusec );

                  The Call to the Base64 procedure must be like this:

                  WEncLen = base64_encode(%addr(signature)
                  : signatureLength
                  : %addr(DataB64)
                  : %size(DataB64) );

                  Regards,
                  Rui



                  Comment

                  Working...
                  X