VARYING procedure return values

[RayGilliesJones]

Is it 'wasteful' to specify the maximum return value size (16773102) on procedures that return varying types? Is all of this memory allocated regardless of it I use or not, or does something super cool happen with the RETURN statement, and only the 'required' memory is used? Is the burden or memory allocating actually in the calling routine, rather than the procedure itself?

I'd be really interested to know.

[JonBoy]

I believe that the entire storage will be allocated regardless of the actual length used. There are however a few "wrinkles" under the covers when it comes to parms and return values. Barbara is better position to clarify this. But as a general rule a varying length field always occupies the full amount of storage regardless of the active content.

Personally I worry more about the performance of large return values than memory usage. This article discusses the topic https://www.itjungle.com/2009/09/30/fhg093009-story01/.

Clearly IBM does too or they would not have added the RTNPARM option - which is how you should be handling large return values on routines that are called with any frequency. Read this if you are unfamiliar with this option https://www.itjungle.com/2010/11/17/fhg111710-story01/.

Hopefully this answers your question.

[RayGilliesJones]

Cheers Jon. I certainly do worry about performance, and yes, I would definitely be using the RTNPARM too. I understand what you mean about varying fields always occupying the full amount of storage, but this is subtly different - it's the return value itself that is defined as 16-gazillion bytes, not a field within the procedure.... so I was wondering if all of that would be gobbled up if I only returned "Hello World"

[JonBoy]

It's going to get copied onto the stack - and then get copied again typically on the receiving end. And as I say I'm pretty sure that ALL of the string will be created/copied. Reason I think that is that otherwise RPG would have to have different internal processing rules for return values than for a comparable field within the program. Just doesn't seem likely to me. When you use RTNPARM you'll get vastly better performance because it is exactly the same as if you were passing it as a parm.

There's a proposed enhancement to RPG to allow the debugger to view the return value before leaving the procedure. If/when it happens might make it more obvious as to whether the full memory has been set up.

[Scott Klement]

To be honest, I've found that the performance of return values is quite good. I realize the system has to do more work with a return value vs. a parameter, but in almost all cases, the performance difference doesn't matter at all. IBM changed the way return values work back when they released IBM i 6.1, and the performance since then has been great. At least, on 100k return values. The extra convenience of working with it as a return value more than makes up for the tiny performance difference.

Now 16mb is another matter. The performance may be a problem there because its so huge. Furthermore, a bigger problem is that you're going to run out of automatic storage very quickly with variables that large. This is a case where using *VARSIZE parameters and passing the %SIZE in another parameter (or using OPDESC) is worthwhile, even if it makes your routine more cumbersome to call. This assumes, of course, that most of the time the caller won't need the data to be that large, but you want the capability to be there in case it ever is.

I really wish IBM would add an "expanding" string type. This is the default in almost all other programming languages today, you simply define your variable as a string, you don't need to give it a max size. The runtime takes care of making it smaller or larger as you change the data. This way, you can just pass a "string" to a routine, and not worry about its max capacity. It would save a lot of hassle for developers, and make it far easier to write general-purpose routines or work with documents like JSON or XML that do not specify sizes for their strings. IBM recently added this sort of support for arrays, but not for strings... would like to see it in strings, and would like these to be passed/retained when calling routines (arrays right now have to be passed as *VARSIZE, which makes their auto-expanding ability nearly useless unless you write old-style monolithic programs.)

[RayGilliesJones]

I agree Scott, it would be great to have expanding string functionality. Likewise I find the new arrays too cumbersome to be bothered with.

[JonBoy]

Well it is supported for strings in that that is effectively how varchar fields work and they are in widespread use on other platforms. But under the covers the growing/shrinking string is not as cost and trouble free as you seem to think. null terminated strings (which is what we're talking about) have their own issues. With a big string where's the end? You have to test every single character for hex 00 - that's a lot of tests. Varchar is much more efficient in that regard. Not to mention that just about every buffer overrun hack (read virus, etc.) can be attributed to the use of such strings and programmers not making sure that they are not too large (yet another test on each character).

Even a 1,000 byte return value can show significant performance gains from RTNPARM - it depends how often you call it! And you don't sacrifice the convenience of return value!

[Scott Klement]

I didn't suggest null-terminated strings?! Believe me, I've been coding in C for more than 20 years, I know all about the shortcomings of null-terminated strings. That would definitely be the wrong direction.

PHP, Python, Node, Java, Bash shell, Perl etc. The list goes on... none of them have predefined limits to how large a string can be, and none of them require you to always take up the maximum amount of memory that a string can be. You, as the programmer, don't have to worry about that stuff, you just define a string (or in the case of JavaScript, just a variable) and the runtime takes care of the rest. It makes programming so much easier. And, modern features sort of assume you'll be using a language like this, its very unusual to have to define the length of a string, so document formats like YAML, XML and JSON don't define them, either.

RTNPARM is nice, no argument there. The only disadvantage is interoperability with other languages. In some cases, though, I still support IBM i 6.1, so can't always use RTNPARM.

[JonBoy]

OK Scott - I didn't mean to imply that you had said null terminated. Languages that have such a capability have to either do the equivalent of a null termination or varchar. It is just transparent to the programmer - but there are going to be costs performance wise surely?

[Scott Klement]

OK Scott - I didn't mean to imply that you had said null terminated. Languages that have such a capability have to either do the equivalent of a null termination or varchar. It is just transparent to the programmer - but there are going to be costs performance wise surely?

Since RPG's existing support is varchar, I assume that would be how RPG would handle it. As I'm sure you know, varchar is more efficient than scanning the whole string for a null value, and also more efficient than the tradiitonal, fixed-length character fields that are constantly being filled up with blanks and then trimmed, and then re-filled up, etc. I think this is very different, however, from what I was discussing, which is the automatic expanding of the memory allocated.

Under the covers, when you try to add data to a string that makes its length exceed what has already been allocated, the runtime would simply reallocate it to the larger size. Would this take some CPU cycles to do? Of course. But, no more so than if it had to be done manually by my code, and surely with less risk of mistakes in allocating/deallocating the memory. I have written service programs that do this sort of thing in the past, and they work well. The down size, of course, is that everything has to be accessed via subprocedure, which is a lot more cumbersome than just accessing it as a string. Other languages are able to do this efficiently to the point where people use them all the time, and nobody is complaining about performance. Why can't RPG do this as well?