Using DATA-GEN with YALDTAGEN Parser

[HMiller]

Hello,
I'm using Scott Klement's YALDTAGEN parser and it's really powerful.
Unfortunately I'm unable to use DATA-GEN with option "write to standard output" when all JSON data need to be saved on an IFS FILE.
I need it as it's a massive request and RPGLE won't compile due to *RNF0376 and *RNF7301 (variable greater than 16.773.104 byte).

data-gen *END
%data(vJsonVar:'doc=file output=continue')
%GEN( 'YAJLDTAGEN'
: '{"write to stdout":true, "http status":200 }');


I attach a simple RPGLE test program which always returns HTTP 500 , Internal server error.

Best regards,
Giovanni

[Scott Klement]

You need to specify the same %GEN options for each of the calls. Because you don't have "write to stdout" on the *START call, it doesn't open the connection to stdout, and therefore it isn't writing the results to the http server. Try it like this:

**free
ctl-opt main (Main) openopt(*nocvtdata) dftActGrp(*no) option(*noDebugIO:*srcStmt);
////////////////////////////////////////////////////////////////////////////////////////
// Main Procedure
////////////////////////////////////////////////////////////////////////////////////////
Dcl-Proc Main ;
   Dcl-Pi  Main;
   End-Pi;
      dcl-ds test ;
         week     char(20);
      end-ds;
   dcl-s i        int(10) inz;
   dcl-s vJsonVar varchar(1000) inz('/tmp/testfile.json');
   dcl-c options '{ "sequence type": "array", +
                    "write to stdout":true, +
                    "http status":200 +
                  }';


   data-gen *START
            %data(vJsonVar:'doc=file output=clear')
               %GEN( 'YAJLDTAGEN': options);
   for i = 0 by 7 to (3*7);
      exec sql
         values char(week_iso(current_date + :i days))
         into :week;
      data-gen test
               %data(vJsonVar:'doc=file output=continue')
               %GEN( 'YAJLDTAGEN':options);
   endfor;

   data-gen *END
            %data(vJsonVar:'doc=file output=continue')
            %GEN( 'YAJLDTAGEN': 'options');

   *inlr = *on;
   return;

End-Proc;

[HMiller]

Hi Scott,
It works absolutely fine now , thank you.
I'm then creating the json output file which has a nested level exactly like your example in "DATA-GEN Presentation".
I did try many alternatives but I always get error with REASON CODE 2 or 3 or I'm unable to close the first bracket at the very end of the json file.
The expected json file should be exactly like the one below where "invoices" arrays has more than 200.000 elements.

{
"customer": 5406,
"stmtDate": "2018-10-05",
"startDate": "2018-09-01",
"endDate": "2018-09-30",
"total": 6600.00,
"invoices":[
{ "invoice": "99001", "amount": 1000.00, "date": "2018-09-14" },
{ "invoice": "99309", "amount": 1500.00, "date": "2018-09-18" },
{ "invoice": "99447", "amount": 500.00, "date": "2018-09-23" },
{ "invoice": "99764", "amount": 3600.00, "date": "2018-09-14" }
]
}

[Scott Klement]

In order to help you with "reason code 2 or 3" I will need to know which error message you are getting. The reason codes mean something different in each message.

[HMiller]

Hi Scott,
my goal is to get a json file like the following:

{
"success" : "testObject",
"weeks" :
[{"week":"20"},{"week":"21"},{"week":"22"},{"wee k": "23"}]

}

Here is my first attempt that raises RNX0365 with Reason Code -3.
I did try to run two nested data-get *START , first is object ("success") , second is array ("weeks")

dcl-s vJsonVar varchar(1000) inz('/tmp/testfile.json');
dcl-c optionsObject '{ "sequence type": "object", +
"write to stdout":true, +
"http status":200 +
}';
dcl-c optionsArray '{ "sequence type": "array", +
"write to stdout":true, +
"http status":200 +
}';
data-gen *START
%data(vJsonVar:'doc=file output=clear')
%GEN( 'YAJLDTAGEN':optionsObject);
success = 'testObject';
data-gen res
%data(vJsonVar:'doc=file output=continue')
%GEN( 'YAJLDTAGEN':optionsObject);
data-gen *START
%data(vJsonVar:'doc=file output=append')
%GEN( 'YAJLDTAGEN':optionsArray);
for i = 0 by 7 to (3*7);
exec sql
values char(week_iso(current_date + :i days))
into :week;
data-gen test
%data(vJsonVar:'countprefix=num_')
%GEN( 'YAJLDTAGEN':optionsArray);
endfor;
data-gen *END
%data(vJsonVar:'doc=file output=continue')
%GEN( 'YAJLDTAGEN':optionsArray);
data-gen *END
%data(vJsonVar:'doc=file output=continue')
%GEN( 'YAJLDTAGEN':optionsObject);

Messaggi . . . : An error occurred with the sequence of DATA-GEN
operations. Reason code 3.

Cause . . . . . : The DATA-GEN operation is out of sequence. See the
description of the reason code for more information. The first operand is
*START. The output file is /tmp/testfile.json. The options are "doc=file
output=append". The reason code is 3.
Recovery . . . : Contact the person responsible for program maintenance to
determine the cause of the problem. To obtain a trace of the calls made to
the generator, add environment variable QIBM_RPG_DATA_GEN_TRACE. See the ILE
RPG Reference for the possible values for this environment variable.

The other attempt was to run separate sequences if *START and *END.
The file is created but the Json file is not closing with }.

dcl-s vJsonVar varchar(1000) inz('/tmp/testfile.json');
dcl-c optionsObject '{ "sequence type": "object", +
"write to stdout":true, +
"http status":200 +
}';
dcl-c optionsArray '{ "sequence type": "array", +
"write to stdout":true, +
"http status":200 +
}';
// OBJECT
data-gen *START
%data(vJsonVar:'doc=file output=clear')
%GEN( 'YAJLDTAGEN':optionsObject);
success = 'testObject';
data-gen res
%data(vJsonVar:'doc=file output=continue')
%GEN( 'YAJLDTAGEN':optionsObject);
data-gen *END
%data(vJsonVar:'doc=file output=continue')
%GEN( 'YAJLDTAGEN':optionsObject);
// ARRAY
data-gen *START
%data(vJsonVar:'doc=file output=append')
%GEN( 'YAJLDTAGEN':optionsArray);
for i = 0 by 7 to (3*7);
exec sql
values char(week_iso(current_date + :i days))
into :week;
data-gen test
%data(vJsonVar:'countprefix=num_')
%GEN( 'YAJLDTAGEN':optionsArray);
endfor;
data-gen *END
%data(vJsonVar:'doc=file output=continue')
%GEN( 'YAJLDTAGEN':optionsArray);

Furthermore with this second attempt I see strange characters in Postman.

[Scott Klement]

Yeah, this doesn't make any sense. You are telling it to create two different documents, but you are referencing the same file. So they will end up both writing to the same place. I don't know what the results will end up being... best case scenario you'll end up with two different documents concatenated in the file. But, I wouldn't rely on that, you may also end up with data from the two documents being mixed together.

The extra "strange characters" you see are the HTTP headers. They're being sent in EBCDIC whereas the rest of this document is in UTF-8, so they look funny.

You can't write two different documents to the same file like that.

[HMiller]

Hi Scott,
thank you for your reply.
It means then that there is no way to create a Json file, without using Data Structures definitions, like this one

{ "customer": 5406,
"stmtDate": "2018-10-05",
"startDate": "2018-09-01",
"endDate": "2018-09-30",
"total": 6600.00,
"invoices":[
{ "invoice": "99001", "amount": 1000.00, "date": "2018-09-14" }, { "invoice": "99309", "amount": 1500.00, "date": "2018-09-18" }, { "invoice": "99447", "amount": 500.00, "date": "2018-09-23" }, { "invoice": "99764", "amount": 3600.00, "date": "2018-09-14" }
]
}

with DATA-GEN %GEN( 'YAJLDTAGEN') using options doc=file and *START *END in order to get rid of *RNF0376 and *RNF7301 (variable greater than 16.773.104 byte) compilation error.