Dynamic Variables CPU Costs

, ,
1

We changed a Natural program in the middle of last month (around May 15th) to use dyanmic variables instead of a FOR Loop.

Looking at our billing system our CPU time has almost doubled.

Does using dynamaic variables in a Natural subprogram running under CICS increase CPU usage?

Should we revert to our For Loops?
Thanks for any help or information that you can give us.

Our Statistics:
May - Number of Transactions: = 63,349
May - CPU time := 4.0241

June - Number of Transactions: = 62,719
June CPU Time/Hrs := 9.1827

2

I would not expect a doubling of CPU time simply due to the (proper) use of Dynamic variables. Your results indicate mis-use. It is true that a fair bit of CPU time is required each time a dynamic variable is expanded, but you can minimize this overhead.

First, should you be using a Dynamic variable? Here’s a snippet from the LEN Consulting LLC’s Introduction to Natural training materials:

  • Reasons to choose STATIC or DYNAMIC
    Static
    . length is known
    . requires redefinition
    Note: over-allocations waste storage

    Dynamic
    . length is unknown
    . storage requirements fluctuate during execution
    . reduce total storage allocations
    Note: incurs CPU overhead for resizing and GETMAINs

Now let’s presume that you have determined that you have a valid reason to use a Dynamic variable.

Make some educated guesses as to how large your variable could be. Let’s presume that the maximum result string is 1,000 characters. Appending a single character at a time, for 1,000 iterations, will consume a lot of CPU (relatively speaking). You can reduce the CPU overhead through the use of the EXPAND statement.

Let’s say that 50% of the time, your result is up to 100 characters and 25% of the time it is up to 500 characters. At the start of your program code

EXPAND DYNAMIC #result TO 100     /* like starting with #result (A100)

Now you can append 100 characters to the result with none of the additional CPU overhead. Then add
code similar to this.

ASSIGN #length = *LENGTH (#result) 
DECIDE ON FIRST #length 
  VALUE 100 
    EXPAND DYNAMIC #result TO 500 
  VALUE 500 
    EXPAND DYNAMIC #result TO 1000 
  NONE 
    IGNORE 
END-DECIDE 
... append next character to #result ...

Also look into the REDUCE and RESIZE statements.

3

Actually, I have done a fair number of timing tests where DYNAMIC beat standard length variables. Hence your observation is quite interesting.

What I really do not understand is your comment that you have replaced a FOR loop with a DYNAMIC variable. Could you show us some code, or at least “pseudo-code” indicating what the FOR loop did and what you replaced it with.

steve

4

wow…doubled CPU with only one program changed? What sort of program is this - would you expect that it is such a large portion of your overall online usage that it could have such an impact?

Check your Buffer Pool statistics for signs of how heavily used the program is (or if there are other candidates).

5

Steve: I apolgize for not getting back to you earlier. It seems I did not get the reply email notification.

We do not have the SAG XML tools.

We are creating an XML string and replaced our old copycode (XML001C1) with new copycode XML001C4 using dynamic variables.

I am providing both copycodes here. We were not using EXPAND. Thanks for telling us about that.

I have included the text in the message and as a word document.

Can you tell us why the XML001C4 doubles the CPU time?

Original copycode XML001C1 which created the XML string.
14:28:22 Copycode XML001C1 Library DOLMLST User MXM0240 2009-07-15

0010 ************************************************************************
0020 DEFINE SUBROUTINE COPY-TEXT-TO-MAIN
0030 ************************************************************************
0040 #TEMP-XML-INDEX := 1
0050 FOR #X := 1 TO 250
0060 IF (#TEMP-XML-ARRAY(#X) = ’ ’ AND #TEMP-XML-ARRAY(#X + 1) = ’ ')
0070 #STOP := 250 - #X
0080 MOVE SUBSTRING(#TEMP-XML,#X,#STOP) TO #FIELD-CHECK
0090 IF #FIELD-CHECK = ’ ’
0100 ESCAPE BOTTOM
0110 END-IF
0120 END-IF
0130 #HOLD-XML-ARRAY(#HOLD-XML-INDEX) := #TEMP-XML-ARRAY(#TEMP-XML-INDEX)
0140 #HOLD-XML-INDEX := #HOLD-XML-INDEX + 1
0150 #TEMP-XML-INDEX := #TEMP-XML-INDEX + 1
0160 *** TO DETERMNE IF THE #HOLD-XML-ARRAY IS FULL
0170 IF #HOLD-XML-ARRAY(250) > ’ ’ OR #HOLD-XML-INDEX = 251
0180 #SEND-BUFFER(#XML-LINE) := #HOLD-XML
0190 #XML-LINE := #XML-LINE + 1
0200 #HOLD-XML-INDEX := 1
0210 RESET #HOLD-XML
0220 END-IF
0230 END-FOR
0240 IF #ERROR-CODE > 0
0250 #SEND-BUFFER(#XML-LINE) := #HOLD-XML
0260 END-IF
0270 RESET #TEMP-XML
0280 END-SUBROUTINE
0290 0

XML001C4: NEW VERSION of above copycode, using dynamic variable to create XML string.

14:28:22 Copycode XML001C4 Library DOLMLST User MXM0240 2009-07-15

0010 ************************************************************************
0020 DEFINE SUBROUTINE COPY-TEXT-TO-MAIN
0030 ************************************************************************
0040 #XML-LEN := *LENGTH(#XML-TEMP)
0050 #XML-START-IDX := 1
0060 IF #XML-LEN > 250
0070 FOR #X := 1 TO 60
0080 MOVE SUBSTRING(#XML-TEMP,#XML-START-IDX,250) TO #SEND-BUFFER(#X)
0090 #XML-START-IDX := #XML-START-IDX + 250
0100 IF (#XML-START-IDX + 250) > #XML-LEN
0110 ESCAPE BOTTOM
0120 END-IF
0130 END-FOR
0140 IF #XML-LEN < 15000
0150 #XML-STOP-IDX := #XML-LEN - #XML-START-IDX + 1
0160 MOVE SUBSTRING(#XML-TEMP,#XML-START-IDX,#XML-STOP-IDX) TO
0170 #SEND-BUFFER(#X + 1)
0180 END-IF
0190 ELSE
0200 #SEND-BUFFER(1) := #XML-TEMP
0210 END-IF
0220 END-SUBROUTINE
0230 **
XMLC004 replacing XMLC001.doc (24.5 KB)

6

Sorry, I don’t get it. :oops: I can’t understand, what your copycodes (especially XML001C1) are really doing. Maybe it’s best to provide us an executable sample with variable definitions etc.

7

Hi Michele
It is a bit hard to reconstruct what the copycode actually achieves,
but my guess is that it just copies from #xml-temp to #send-buffer ?!

The old copycode moved from an array to another, whereas the new just
moves from one dynamic var to an array - correct ?

As the programlogic is therefore quite different it is quite hard to determine
if the dynamic variable as such is causing the overhead (though i doubt it;-).

My approach to moving would be far more simple:
I assume the #send-buffer is an alfa array like below(?),
so a redefined alfa could be used as target for one single move instead.

DEFINE DATA LOCAL
1 #SEND-BUFFER (A250/60)
1 REDEFINE #SEND-BUFFER
2 #SEND-BUFFER-REDEF (A15000)

1 #XML-TEMP (A) DYNAMIC
1 #XML-LEN (I4)

END-DEFINE

#XML-LEN := *LENGTH(#XML-TEMP)
IF #XML-LEN >15000
#XML-LEN := 15000

    • some extra errorhandling ?!
      END-IF
      RESET #SEND-BUFFER-REDEF
      MOVE SUBSTRING(#XML-TEMP,1,#XML-LEN) TO #SEND-BUFFER-REDEF
      END
8

One uses a fixed-length variable (#TEMP-XML) the other uses a dynamic variable (#XML-TEMP). Both break down the large variable via SUBSTRing.

I ran CPU tests on a program similar to the second module (XML001C4). As I expected, there was little difference in CPU usage when I changed the variable from fixed to DYNAMIC (tests using Natural for Windows).

Michele, what about the code that populates the dynamic variable? That is where I would expect to find the problem. Did that code change, too?

9

Thank you all for your responses. It is our first time with Dynamic variables.

We have a 15000 buffer containing formatted XML data that we are passing backto a server. In earlier versions of Natural we could not have a buffer this size, so we redefined it into 60 buffers of 250 bytes.

In the calling program, we do not use EXPAND, RESIZE or REDUCE. We are continuing compressing data into this dynamic varriable #XML-TEMP. When we are finished with all our compressions we call this copycode, when we are ready to move the buffer to another buffer. It is only called one time at the end of the calling program.

It did do a test where I just coded

#XML-LEN := *LENGTH(#XML-TEMP)

MOVE SUBSTRING(#XML-TEMP,1,#XML-LEN) TO #SEND-AREA

Could this replace our copycode ? Do we need to do a ‘free’ of memory at the end? If so what is the code?

If we continue to compress into the same dynamic variable, does it expand every time, and this use more processing time?

How is the best way to use EXPAND, SIZE, RESIZE

Thanks for your help.

The input variables are defined as below in an LDA format:

 1 #RECEIVE-AREA                    A    1 (15000)

R 1 #RECEIVE-AREA /* REDEF. BEGIN : #RECEIVE-A
2 #RECEIVE-BUFFER A 250 (60)
R 1 #RECEIVE-AREA
2 #TRANS-ENVIR A 1 /* D=DEV, Q=QA, P=PROD
2 #TRANS-CODE A 5
1 #SEND-AREA A 1 (15000)
R 1 #SEND-AREA /* REDEF. BEGIN : #SEND-AREA
2 #SEND-BUFFER A 250 (60)

The other variables as defined in a LDA are

1 #XML-LEN N 5
1 #XML-START-IDX N 5
1 #XML-STOP-IDX N 5
1 #XML-LINE N 3 INIT<1>
1 #X N 3
1 #ERROR-CODE N 2
1 #D-QUOTE A 1 INIT
1 #VALUE-TO-ADD A 200
1 #TAG-NAME A 20
1 #TABLE-KEY-2 A 35
1 #TABLE-KEY-2 /* REDEF. BEGIN : #TA
2 #TBL-KEY-2-NUM A 3
2 #TBL-KEY-2-DATA A 32
1 #XML-TEMP A DYNAMIC
1 #XML-MISC-TEXT A DYNAMIC

10

Hi Michele,
I totally agree with Ralph: You should have a look at the code that populates the dynamic string !
I you subscribe to Steve’s “Inside Natural” I can really recommend you take a closer look at the Number 4 Issue from 2003, where Steve goes through all (and I mean ALL;-) the different aspects of dynamic variables !

And to correct myself (embarrassing!) there is no need for substring if you move from a dynamic to a standard alfa !!
So instead of

#XML-LEN := *LENGTH(#XML-TEMP)
MOVE SUBSTRING(#XML-TEMP,1,#XML-LEN) TO #SEND-AREA

you should just use:
MOVE #XML-TEMP TO #SEND-AREA

But be aware you still need another redefine of your #SEND-DATA as (A15000)
in order to do this.
Finn

11

Normally such a small field length wouldn’t warrant the use of a Dynamic variable. Neither does your logic seem to warrant the use of arrays, which can impact performance. All you need is

1 #XML-TEMP (A15000)    
1 #SEND-AREA (A15000)    
    
some-sort-of-loop    
  COMPRESS #XML-TEMP    
           #next-string    
      into #XML-TEMP leaving no space    
end-some-sort-of-loop    
ASSIGN #SEND-AREA = #XML-TEMP   /* couldn't you use #XML-TEMP instead of #SEND-AREA?    
RESET  #XML-TEMP                /* for next XML doc/iteration

The problem here is that, under the covers, Natural resets all 15,000 bytes of #XML-TEMP before filling it with the result of the COMPRESS statement. This will burn CPU, and the bigger the variable, the bigger the impact.

Change #XML-TEMP to a Dynamic variable to avoid this. With a Dynamic variable, there are no trailing bytes that Natural needs to reset. Just remember to set the size back to zero before you reiterate for the next XML document. If your program processes only one document at a time, then you don’t need to resize because #XML-TEMP will have a length of zero each time the program is invoked.

1 #XML-TEMP (A) DYNAMIC    
1 #SEND-AREA (A15000)  
  
some-sort-of-loop    
  COMPRESS ...    
end-some-sort-of-loop    
ASSIGN #SEND-AREA = #XML-TEMP    
RESIZE DYNAMIC #XML-TEMP TO 0   /* for next XML doc/iteration

Based on what you have told us and the code that you provided, the Dynamic variable is not the culprit. You might be able to reduce CPU consumption, but only slightly, by removing the SUBSTRing clauses and arrays as suggested by Finn. (I would do so for that reason and for the cleaner code.) Re-read Doug Kelly’s posting and look for the CPU problem elsewhere.

To answer the remaining question regarding EXPAND, REDUCE, and RESIZE statements, you would use them only if there were huge variations in the sizes of the XML documents, as I explained in my initial posting.

Tips to save CPU: When using arrays, define the indices as full-word integers. And turn on the Optimizing Compiler if you have it.

1 #I (I4)    
1 #J (I4)
12

Thank you Everyone! We appreciate your time that it took to answer this problem, and we learned a lot! :slight_smile:
Michele Marie