No announcement yet.

Serialize data structure to JSON

  • Filter
  • Time
  • Show
Clear All
new posts

  • Serialize data structure to JSON


    recently my company has been using web services to decouple our very very old legacy system into more modern micro-services linked by API's. We have created a our own JSON parser to to tokenise the input JSON message string and map it to a corresponding data structure using the new DATA-INTO opcode. However we have been wondering about how to go the other way, serialize a data structure to a JSON string.

    We looked at using YAJL initially but it was a manual process and each data structure had to be serialized manually - unless I have missed something obvious. With C# and JAVA there are built in reflection libraries that allow the programmer to determine the properties of a class/type and recursively check for any additional sub-classes/types all the way down to the primitive types. I might be barking mad but would there be anything in RPG that allows us to do the same? That way we could determine the type of data structure subfields and serialize them to the correct JSON type.

  • #2
    Not a direct method that I can think of that would work in all scenarios. Since RPG is a compiled language the information about data types and structures is not normally part of the compiled program and therefore not available to query.

    One possible solution that _might_ fit your needs - but it will depend on the structures you need to handle. If a program receives a DS as a parameter and/or a subprocedure does the same or returns a DS, then the compiler can be told to generate PCML (XML format) which could be embedded in the program. That can then be used to dynamically interrogate the data passed. There are limits on the depth of nested DS etc. I think but I don't recall what the current limits are. The latest version of PCML removes many restrictions.

    I suspect though that the easiest approach may be to write a code generator that will example the DS and use YAJL calls to generate the required JSON. This would be simplified by having the required DS be in separate /COPY sources that way the generator doesn't have to "hunt" for the required DS and can also include a /COPY of the same source for the internal definitions in the generated code. The genrated code could be a subproc or program that is called from the user program and passed the DS and return the JSON.

    P.S. You said you wrote your own JSON parser - I highly recommend that you consider switching to YAJLINTO (was JSONINTO which is wrong) - less maintenance issues and I suspect that it would also be faster.
    Last edited by JonBoy; April 11, 2019, 08:59 AM.


    • #3
      It'd be nice if IBM would provide a way for us to do what you describe.

      As you noted, there's no reflection available in RPG, so no way for a 3rd party program to know the format of your data, and therefore no way to serialize it. As for PCML, unless something has changed since I looked at it last, PCML is only for the procedure interface, not for random data structures within the program. Though, I suppose that could be workable if you didn't mind defining it in the PI. Maybe that's something I should play with some time.

      If IBM were to provide something like DATA-INTO but in the opposite direction (DATA-OUTOF?) a third party tool could be written to do what you need... but they don't provide anything like that.


      • #4
        There's an RFE for that:

        The title of the RFE is "Create XML in RPGLE, BIF like %XML-CREATE()", but the official IBM response says
        If RPG implements this, it would be similar to DATA-INTO,where the the details of the XML (or other output format) would be left to a data-generator. The generator would get the subfield names (mixed-case similar to the *DCLCASE mechanism for PCML), plus human-readable values, and be free to format the data in any way.


        • #5
          Hi all, thanks very much for your replies.

          JonBoy, C# and Java are compiled languages and they still have that ability to query information about specific classes and properties. Thats a very interesting idea using the PCML. I've had to do a bit of reading about that as no-one where I work (including myself) seems to know anything about it! Also, what is this JSONINTO? I haven't heard of this!

          JonBoy / Scott Klement
          So if I was to write a service procedure that took any data structure as a parameter would the PCML definition change on each program call if I was passing in a different data structure? Or is it the PCML generated at compile time for a specific data structure? You'll have to excuse my lack of understanding around this area of the language/compiler, guess it's time for me to have a play around with it and use the QBNRPII API and see what I can get working.

          Scott Klement your YAJL tooling kit has been massively helpful!! I tend to agree with you that something such as DATA-OUT would be the best thing.

          Barbara Morris what is taking you IBM boffins so long! Only kidding Thanks very much for the link, I have up voted the issue. Would be very useful to have some sort of DATA-OUT or even better a, reflection API.


          • #6
            1) No quite true. They "compile" down to an intermediate code that runs on a virtual machine. Since that virtual machine has to be platform independent it has to have a lot more information about the data types that it is dealing with than a conventionally compiled program that is compiled down to execute on a specific machine.

            YAJLINTO (edited from incorrect name of JSONINTO) is Scott's extension to the YAJL parser to allow it to act as a DATA-INTO parser. It is part of Scott's YAJL package.

            2) PCML is generated at compile time. So you'd basically need one procedure per DS. That is really not a limit since (other than via pointers) you can't pass a generic value anyway. I would suggest doing your original investigations by having the PCML generated into the IFS rather than mess with QBNRPII. Worry about that once you know if the PCML will work for you.
            Last edited by JonBoy; April 11, 2019, 08:57 AM.


            • #7
              I agree that dealing with the PCML in the stream file would probably be easier, but here's the source for a program that gets the PCML out of a program or service program using QBNRPII.

              If you do go the PCML route, take advantage of the ability to code PGMINFO(*YES) on the procedures you want PCML generated for (or PGMINFO(*NO) on the ones you don't). Otherwise, you might run into PCML restrictions related to your other procedures that you don't need PCML for.


              • #8
                Originally posted by JonBoy View Post
                JSONINTO is Scott's extension to the YAJL parser to allow it to act as a DATA-INTO parser. It is part of Scott's YAJL package.
                You're thinking of YAJLINTO.


                • JonBoy
                  JonBoy commented
                  Editing a comment
                  Doh! Thanks Scott - I've corrected the original posts.

              • #9
                I had it in my head that there were restrictions with PCML concerning nesting etc. but couldn't find any docs so I knocked up a simple(ish) eaxmple and all the stuff I would need appears to be working. Guess I'm thinking back to V1 or 2 of PCML.

                Here's the program I used:
                CTL-OPT dftactgrp(*No) actgrp(*NEW);
                CTL-OPT PGMINFO(*PCML : *DCLCASE);
                Dcl-Pi  Play  ExtPgm;
                  DSParm  LikeDS(testStuff);
                Dcl-DS  testStuff Qualified;
                  Dcl-DS  simple1;
                    schar  char(10);
                    sint   int(5);
                  dcl-ds  dsarray  Dim(3);
                    achar  char(10);
                    aint   int(5);
                    dcl-ds  nested;
                      nchar1    char(10);
                      nint1     int(5);
                  Char1       Char(10);
                  int1        Int(10);
                  Packed1     Packed(7:2);
                  Zoned1      Zoned(5:2);
                  Array1      Char(10)  Dim(20);
                *InLr = *On;
                And this is resulting PCML.

                <pcml version="6.0">
                   <!-- RPG program: PLAY  -->
                   <!-- created: 2019-04-11-11.55.32 -->
                   <!-- source: PARTNER400/QRPGLESRC(PLAY) -->
                   <!-- 17 -->
                   <struct name="nested">
                      <data name="nchar1" type="char" length="10" usage="inherit" />
                      <data name="nint1" type="int" length="2" precision="15" usage="inherit" />
                   <!-- 14 -->
                   <struct name="dsarray">
                      <data name="achar" type="char" length="10" usage="inherit" />
                      <data name="aint" type="int" length="2" precision="15" usage="inherit" />
                      <data name="nested" type="struct" struct="nested" usage="inherit" />
                   <!-- 10 -->
                   <struct name="simple1">
                      <data name="schar" type="char" length="10" usage="inherit" />
                      <data name="sint" type="int" length="2" precision="15" usage="inherit" />
                   <!-- 9 -->
                   <struct name="testStuff">
                      <data name="simple1" type="struct" struct="simple1" usage="inherit" />
                      <data name="dsarray" type="struct" struct="dsarray" count="3" usage="inherit" />
                      <data name="Char1" type="char" length="10" usage="inherit" />
                      <data name="int1" type="int" length="4" precision="31" usage="inherit" />
                      <data name="Packed1" type="packed" length="7" precision="2" usage="inherit" />
                      <data name="Zoned1" type="zoned" length="5" precision="2" usage="inherit" />
                      <data name="Array1" type="char" length="10" count="20" usage="inherit" />
                   <!-- 5 -->
                   <program name="Play" path="/QSYS.LIB/PARTNER400.LIB/PLAY.PGM">
                      <data name="DSParm" type="struct" struct="testStuff" usage="inputoutput" />


                • #10
                  Seems to me the biggest limitation is with arrays. When you define DIM(32767) that doesn't necessarily mean that you want the JSON document you create to have 32767 elements, for example. The value on DIM is the maximum -- so you need some way of passing the 'current length' to whatever tool is generating the JSON from the PCML. This, perhaps, becomes more complex when you have things like arrays nested inside of structures (possibly many levels deep).

                  I guess you'd probably want something like the reverse of 'countprefix' to solve it. (though, that feels like a kludge. It sure would be nice if RPG had variable-length arrays!!)


                  • #11
                    Originally posted by Scott Klement View Post
                    I guess you'd probably want something like the reverse of 'countprefix' to solve it. (though, that feels like a kludge. It sure would be nice if RPG had variable-length arrays!!)
                    We can but hope ... it has been on the shopping list for a long time.


                    • #12
                      Just to complete this thread. Varying length arrays are available in 7.4 (only) BUT only stand-alone arrays and DS arrays. Nested arrays at some future date hopefully.


                      • #13
                        Also, you can't (currently) define a parameter as a varying-length array.