Area need: which handed-over bytes anyone reads

Read first: Fixpoints and worklists, Gen/kill and reaching definitions, Value sets and dynamic calls, Byte extents

To say where XCTL PROGRAM(WS-NEXT-PGM) goes, you have to know what the caller left in those bytes, so the analysis follows the caller's whole commarea forward into the callee. Commareas run to hundreds or thousands of bytes, and carrying all of them through every program that hands one over is most of what the cross-program analysis costs — yet almost none of those bytes ever name a program.

PROGA hands PROGB 100 bytes; PROGB reads 8 of them to decide where to go next. Area need finds those 8, for every program, before the expensive analysis runs, so that analysis carries 8 bytes across the arrow instead of 100. On CardDemo, the bytes it says anyone needs on entry add up, across all its programs, to 92.

New to commareas or XCTL? See the COBOL/CICS primer. Each program in the corpus is a member.

The example

PROGA hands its whole 100-byte commarea to PROGB, which transfers on to the name in bytes [8, 16):

flowchart TD
    A["PROGA
writes ws[0, 8) and ws[8, 16)"] -->|"XCTL, hands ws[0, 100)"| B["PROGB
reads bytes [8, 16)"]
    B -->|"XCTL PROGRAM(WS-NEXT-PGM)"| C(["'PROGC'"])

Here is that commarea, one cell per run of bytes from the offset above it (4 bytes a cell up to 16, then 21). PROGB's XCTL reads the two outlined cells. Nothing reads the other six.

PROGA's 100-byte commarea (outlined: read by PROGB's XCTL)

That is the whole answer for this example — one row per program, saying which bytes of what it receives anyone reads:

The lines that matter, with WS-NEXT-PGM at bytes [8, 16) in both programs:

* PROGA
01  WS-COMMAREA.
    05  WS-USER-ID   PIC X(8).
    05  WS-NEXT-PGM  PIC X(8).
    05  WS-REST      PIC X(84).
MOVE 'USER0001' TO WS-USER-ID.
MOVE 'PROGC' TO WS-NEXT-PGM.
EXEC CICS XCTL PROGRAM('PROGB')
     COMMAREA(WS-COMMAREA) END-EXEC.

* PROGB (WS-COMMAREA laid out as in PROGA)
01  DFHCOMMAREA  PIC X(100).
MOVE DFHCOMMAREA TO WS-COMMAREA.
EXEC CICS XCTL PROGRAM(WS-NEXT-PGM) END-EXEC.

       IDENTIFICATION DIVISION.
       PROGRAM-ID. PROGA.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01  WS-COMMAREA.
           05  WS-USER-ID       PIC X(8).
           05  WS-NEXT-PGM      PIC X(8).
           05  WS-REST          PIC X(84).
       PROCEDURE DIVISION.
       MAIN-PARA.
           MOVE 'USER0001' TO WS-USER-ID.
           MOVE 'PROGC' TO WS-NEXT-PGM.
           EXEC CICS XCTL PROGRAM('PROGB')
                COMMAREA(WS-COMMAREA) END-EXEC.

       IDENTIFICATION DIVISION.
       PROGRAM-ID. PROGB.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01  WS-COMMAREA.
           05  WS-USER-ID       PIC X(8).
           05  WS-NEXT-PGM      PIC X(8).
           05  WS-REST          PIC X(84).
       LINKAGE SECTION.
       01  DFHCOMMAREA          PIC X(100).
       PROCEDURE DIVISION.
       MAIN-PARA.
           MOVE DFHCOMMAREA TO WS-COMMAREA.
           EXEC CICS XCTL PROGRAM(WS-NEXT-PGM) END-EXEC.

Byte ranges are written region[start, end), counted from 0: ws[8, 16) is WS-NEXT-PGM, linkage[0, 100) is PROGB's DFHCOMMAREA.

Step by step

Only statements are shown; the graph's other nodes write nothing and hand their live set on unchanged. Both XCTLs transfer for good, so neither program reads anything back and both back rows stay empty.

Within a program the walk visits statements last to first (the worklist takes the highest position each time), which is why each table below lists the XCTL first. Each table shows what is live on entry to the statement, that is, just above it.

Fed the file from step 5, cobble-dataflow-fixpoint --call-targets-only --narrow-areas demands only ws[8, 16) at PROGA's XCTL instead of all of ws[0, 100). The table shows what that does to PROGA's results: it keeps one write target (WS-NEXT-PGM) instead of two, and its edge count drops from 2 to 1. The last row is the cross-program combine, the same in both runs: it still follows linkage[8, 16) in PROGB back to 'PROGC' in PROGA. These rows and counts are what cobble produced on these two programs.

Without the byte-for-byte copy rule, the MOVE in PROGB would make all of linkage[0, 100) live, as in the strip below, rather than the two amber cells of step 2. PROGB's entry row would be [0, 100), and PROGA would have to keep MOVE 'USER0001' too, along with whatever wrote the other 84 bytes.

Live at PROGB's entry if the MOVE needed its whole source

Why it runs backwards

Step 2 of the walk read PROGB bottom to top. Here are its two statements with what is live written between them:

                    live: {linkage[8, 16)}
MOVE DFHCOMMAREA TO WS-COMMAREA.
                    live: {ws[8, 16)}
EXEC CICS XCTL PROGRAM(WS-NEXT-PGM) END-EXEC.
                    ↑ walk from the bottom up

The XCTL needs ws[8, 16), so that is live just above it. The MOVE writes all of ws[8, 16), so above the MOVE the need moves to what it copied from: linkage[8, 16). This is ordinary backward liveness — the worklist of reaching definitions run in reverse, over byte extents instead of whole fields.

Where paths split, anything needed on either path is needed. Put the MOVE inside an IF, and above the IF both ranges are live:

                    live: {ws[8, 16), linkage[8, 16)}
IF WS-FLAG = 'Y'
                    live: {linkage[8, 16)}
    MOVE DFHCOMMAREA TO WS-COMMAREA
END-IF.
                    live: {ws[8, 16)}
EXEC CICS XCTL PROGRAM(WS-NEXT-PGM) END-EXEC.

The entry row is still [8, 16): ws[8, 16) is PROGB's own storage, not part of what it receives. (Cobble gives this variant the same row as the original.) The two ends of every walk are the seed and the summary:

• Seed: every dynamic call site needs its operand, as PROGB's XCTL needs WS-NEXT-PGM; so does a TRANSID data name. A call that hands an area over needs whatever the callee reads on entry — in step 4, PROGA's XCTL needs ws[8, 16) once PROGB's row is known.
• Summary: the bytes of a received area still live at the top are the member's entry row, [8, 16) for PROGB.

Both XCTLs transfer for good, so the walk's back rows stay empty. A LINK returns, and what the caller reads afterwards becomes the callee's back row. Change PROGA's transfer to a LINK followed by a transfer to WS-USER-ID:

EXEC CICS LINK PROGRAM('PROGB')
     COMMAREA(WS-COMMAREA) END-EXEC.
                    live just below: {ws[0, 8)}
EXEC CICS XCTL PROGRAM(WS-USER-ID) END-EXEC.
                    needs ws[0, 8)

Just below the LINK, ws[0, 8) is live. Inside the handed area that is offsets [0, 8), so cobble gives PROGB "back":[[["commarea",0],[0,8]]], needed at its GOBACK, EXIT PROGRAM, STOP RUN or EXEC CICS RETURN (charging STOP RUN, which never returns, is a harmless over-approximation). Its entry row stays [8, 16).

Rows are keyed by slot and hold offsets from the start of the area, which is why PROGA's ws[8, 16) and PROGB's linkage[8, 16) are the same two cells of the strip:

When the need crosses a program

In the walk, the need travelled back one arrow: PROGB's row in round 1, PROGA's charge in round 2. Put a pass-through program, PROGM, in the middle and it has to travel two:

PROGRAM-ID. PROGM.
    MOVE DFHCOMMAREA TO WS-COMMAREA.
    EXEC CICS XCTL PROGRAM('PROGB')
         COMMAREA(WS-COMMAREA) END-EXEC.

Now the need has to travel back along the chain, one dotted arrow at a time. PROGB's need on entry decides what PROGM must keep live at its XCTL; that makes PROGM's own commarea live on its entry, which PROGA must then keep. Each dotted arrow is one round: the solid arrows hand bytes forward, the dotted ones hand the need back.

flowchart TD
    A["PROGA
receives nothing"] -->|"XCTL, hands ws[0, 100)"| M["PROGM
copies DFHCOMMAREA"]
    M -->|"XCTL, hands ws[0, 100)"| B["PROGB
XCTL PROGRAM(WS-NEXT-PGM)"]
    B -.->|"round 1: PROGB entry [8, 16)"| M
    M -.->|"round 2: PROGM entry [8, 16)"| A

Round 3 walks PROGA once more and nothing changes, since PROGA receives no area and the need has nowhere further to go. That is what "until it stops changing" means here: three rounds for a chain of three. (These rows and the three rounds are what cobble-dataflow-area-need reported on these three programs.)

Why it matters on a real corpus

A dynamic transfer's target name comes from one of three places:

* 1: a VALUE
01  WS-NEXT-PGM  PIC X(8) VALUE 'PROGC'.
* 2: a MOVE of a literal
    MOVE 'PROGC' TO WS-NEXT-PGM.
* 3: read out of the commarea
    MOVE DFHCOMMAREA TO WS-COMMAREA.
    EXEC CICS XCTL PROGRAM(WS-NEXT-PGM) END-EXEC.

Shapes 1 and 2 are answered inside one program. Shape 3 is PROGB: the name is whatever the caller put in those bytes. On a large CICS corpus (about 3,800 programs and 19,000 jumps) about 5% of sites look like this, and they are why values are followed across programs at all. Following them means carrying every handed-over area forward through every caller — areas of hundreds or thousands of bytes, almost none of which name a program, as in the strip. Area need is the cheap backward question asked first. On CardDemo, folding LENGTH OF bounds and then resolving more sites brought its summed entry need down like this, and the narrowed run gave the same targets, links and graph as the run over whole areas:

Cobble runs it as cobble-dataflow-area-need, then cobble-dataflow-fixpoint --call-targets-only --narrow-areas, then the cross-program combine (commands under in cobble); the steps behind the CardDemo rows are under what goes wrong.

The rule

Most lines of the box have already run on this page. The table points each of those back at the place it was applied.

Byte-for-byte copies. A MOVE copies byte for byte when either side is a group, or both are alphanumeric, the source has no subscript, and its reference-modification start (if any) is known to the compiler. For such a write, the walk maps each live target byte back to the one source byte that landed on it, and keeps only those: that is why step 2's strip has two amber cells and not eight. Any other write needs all its sources.

LENGTH OF in bounds. A reference-modification bound such as WS-AREA(LENGTH OF WS-HDR + 1 : LENGTH OF WS-BODY) is folded to a number, because the compiler knows each field's length from its declaration. Literals, LENGTH OF and + - * / over them fold. With both bounds known the extent is exact and narrow. Without folding it would be a clamped extent over the whole base field. In PROGB's terms, the write below starts at position 8 + 8 + 1 = 17 (reference modification counts from 1, so offset 16) and is 20 bytes long: ws[16, 36), inside WS-REST, clear of the live ws[8, 16).

01  WS-SAVED             PIC X(20).
    MOVE WS-SAVED TO WS-COMMAREA(LENGTH OF WS-USER-ID
         + LENGTH OF WS-NEXT-PGM + 1 : LENGTH OF WS-SAVED).

START and RETRIEVE. A START hands over its FROM area the way a LINK hands over a commarea, in slot RETRIEVED/0. Its transaction id is resolved to a program through the CSD, as a RETURN TRANSID's is. The started program's need in that slot is the bytes of its RETRIEVE INTO area live right after the RETRIEVE. The RETRIEVE itself demands nothing: it writes the area, it does not read it. In the fragment below, worked by the rule rather than run, the XCTL makes ws[8, 16) live right after the RETRIEVE, so the started program's row is RETRIEVED/0: [8, 16), and the START is charged those bytes of its FROM area.

* the program that starts the transaction
     EXEC CICS START TRANSID('TRNB')
          FROM(WS-COMMAREA) END-EXEC.
* the program the CSD maps TRNB to
     EXEC CICS RETRIEVE INTO(WS-COMMAREA) END-EXEC.
     EXEC CICS XCTL PROGRAM(WS-NEXT-PGM) END-EXEC.

How the result narrows the fixpoint. --call-targets-only keeps only the writes a dynamic call target or a handed-over area depends on. Without --narrow-areas every handed-over area counts whole: the middle column of the table after the walk. With it, each call is charged only the bytes the area-need file says its callees read, so the forward run tracks those bytes and nothing else of the area: the right-hand column.

What goes wrong

Each bullet below comes with a small change to the example and what cobble reported for it. Every variant keeps PROGA as listed above unless it says otherwise.

• One unresolved site charges everyone. A dynamic call site that hands over an area and is not resolved is charged the union of every member's entry row. In the variant below, PROGB's XCTL also hands its commarea on, and the corpus gains a PROGD that nothing calls but that needs all 100 bytes. PROGB's site is not resolved inside its own program, so area need charges it PROGD's row, and the 100 bytes spread back to PROGA.

  * PROGB: the dynamic XCTL now hands an area over
       EXEC CICS XCTL PROGRAM(WS-NEXT-PGM)
            COMMAREA(WS-COMMAREA) END-EXEC.
  * PROGD: called by nobody
       MOVE DFHCOMMAREA(WS-POS : 8) TO WS-NEXT-PGM.
       EXEC CICS XCTL PROGRAM(WS-NEXT-PGM) END-EXEC.

  member	entry row	as listed
  PROGD	[0, 100)	not in the corpus
  PROGB	[0, 100)	[8, 16)
  PROGA edges	2	1

  In a corpus where some programs need thousands of bytes, that union is thousands of bytes, and it then makes the caller's own commarea live, which spreads to its callers, as it spread from PROGB to PROGA in the table. Before narrowing on the large CICS corpus, rows such as 4,500 bytes needed on entry appeared, mostly for this reason. Resolving as many sites as possible inside their own program, before the rounds start, is the fix; see resolving a dynamic site locally. A site resolved to no program at all charges nothing.
• Clamped extents keep whole areas live. A write whose target is not exact (a computed subscript, a reference-modification bound that does not fold) is weak: it kills nothing, and its whole target becomes live wherever it touches a live byte. The write may not land, so the forward run merges what it writes with the target's old contents, and it cannot tell which bytes those are; the old bytes of the whole target are therefore read. Put the LENGTH OF write from the rule section into PROGB, between its MOVE and its XCTL, then swap its folded start for a run-time one:

  MOVE WS-SAVED TO WS-COMMAREA(LENGTH OF WS-USER-ID
       + LENGTH OF WS-NEXT-PGM + 1 : LENGTH OF WS-SAVED).
  MOVE WS-SAVED TO WS-COMMAREA(WS-POS : LENGTH OF WS-SAVED).

  start	write's extent	PROGB entry row
  LENGTH OF … + 1	ws[16, 36), exact	[8, 16)
  WS-POS	ws[0, 100), clamped	[0, 100)

  The clamped write touches the live ws[8, 16), so all of ws[0, 100) becomes live above it, and the byte-for-byte MOVE then maps all 100 bytes back to linkage. Before LENGTH OF folded, a program that moved its saved state into part of a 2,000-byte commarea with LENGTH OF bounds made the whole area live, and its RETURN TRANSID to itself grew an 8-byte need to 2,000. That one fix took CardDemo's summed entry need from 4,394 bytes to 140. Resolving more sites, using the results of an earlier fixpoint and combine fed back through --fixpoint-dir and --combined-dir (see local site resolution), then took it to 92.
• Copies that are not byte-for-byte need everything. A MOVE into a numeric field, a subscripted source, or a source whose reference-modification start is a run-time value all fall back to needing the whole source. That is safe but wide. Replace PROGB's group MOVE with a move of just the name, first with a literal start, then with a run-time one:

  MOVE DFHCOMMAREA(9 : 8) TO WS-NEXT-PGM.
  MOVE DFHCOMMAREA(WS-POS : 8) TO WS-NEXT-PGM.

  source	byte for byte?	PROGB entry row
  DFHCOMMAREA(9 : 8)	yes	[8, 16)
  DFHCOMMAREA(WS-POS : 8)	no	[0, 100)

  Both write the same 8 bytes; the second reads from an offset nobody knows until run time, so every byte of DFHCOMMAREA might be the one, and the strip after the walk is what results. Before the copy rule, CardDemo's summed entry need was 32,767 bytes; with it, 1,033. That pair was measured separately from the 4,394 → 140 → 92 sequence above, so 1,033 is not that sequence's starting point.
• A capped run can be short. --max-rounds N stops after N rounds and warns how many members were still to walk. A need that has not settled can miss bytes that only arrive through longer call chains, and a fixpoint narrowed by it can then miss candidates. On the PROGA → PROGM → PROGB chain from the problem, --max-rounds 1 stops after the first dotted arrow:

  	settled (3 rounds)	--max-rounds 1
  PROGM entry row	[8, 16)	empty
  PROGA edges	1	0
  PROGB's XCTL	candidates ["PROGC"]	candidates [], reason not-recorded

  An uncapped run always settles, since rows only grow and are bounded by the areas' sizes.
• It is only as good as the corpus. A callee that is not in the corpus, or is but does not receive that slot, is charged every byte of the area: nothing is known about what it reads. Run PROGA alone, without PROGB, and its XCTL is charged all of ws[0, 100):

  corpus	PROGA's XCTL keeps	PROGA edges
  PROGA, PROGB	ws[8, 16)	1
  PROGA alone	ws[0, 100)	2

  An unreadable member adds nothing, so its own targets go unresolved later.

In cobble

The command is cobble-dataflow-area-need. It runs the rounds over the whole corpus and stops when a round changes no row, or after --max-rounds N. --sites adds a report of each dynamic site and where it sits in the source. The fixpoint reads the result under cobble-dataflow-fixpoint --call-targets-only --narrow-areas, and tracks only the bytes of each handed-over area that some program reads. Below are the three commands as run on this page's two programs, after cobble-parse-corpus and cobble-extents, with the two round lines the first one logged (timestamps and per-member lines left out): the same two rounds as the walk, 8 bytes needed on entry, and 1 then 0 members to walk next. The command is documented in field dataflow.

cobble-dataflow-area-need --asg-dir asg --extents extents.jsonl \
    --out need.jsonl
cobble-dataflow-fixpoint --asg-dir asg --extents extents.jsonl \
    --out-dir fp --call-targets-only --narrow-areas need.jsonl --all
cobble-dataflow-combine --config corpus.toml --fixpoint-dir fp \
    --out-dir comb --all

area need round 1: walked 2 members; 8 bytes needed on entry, 0 read back, summed over 2 members; 1 to walk next
area need round 2: walked 1 members; 8 bytes needed on entry, 0 read back, summed over 2 members; 0 to walk next

Next: resolving a dynamic call site inside its own program, and rounds across programs, and walking only what changed.