next up previous
Next: Blocking
Up: NDF SECTIONS
Previous: More Advanced Use: Partitioning

Chunking  

The technique of chunking is best introduced by an example. Consider a large 1-dimensional NDF (a spectrum, perhaps) which is to be processed in pieces in order to limit memory usage, and suppose that the maximum size of one of these pieces is to be 10000 pixels. The spectrum can be divided into pieces for processing by creating a section to refer to each piece. Each of these sections should follow on from the previous one, and each should contain 10000 pixels (except the last one, which may be smaller if the total number of pixels is not an exact multiple of 10000). Each of these sections is termed a chunk.

If we want to process each of these chunks in turn, we need to know how many there are, and to have a method of creating the appropriate sections. The NDF_ system provides these facilities through the routines NDF_NCHNK (which determines the number of chunks available) and NDF_CHUNK (which creates sections referring to successive chunks). The following illustrates how these routines might be used:

      INTEGER ICHUNK, ICH, MXPIX, NCHUNK
      PARAMETER ( MXPIX = 10000 )

      ...

*  Determine the number of chunks available.
      CALL NDF_NCHNK( INDF, MXPIX, NCHUNK, STATUS )

*  Loop through the chunks, creating a section to refer to each one.
      DO 1 ICHUNK = 1, NCHUNK
         CALL NDF_CHUNK( INDF, MXPIX, ICHUNK, ICH, STATUS )

         <access the resulting section/chunk>

*  Annul the section identifier.
         CALL NDF_ANNUL( ICH, STATUS )
 1    CONTINUE

Here, MXPIX is set equal to the maximum number of pixels which a chunk is to contain, and NDF_NCHNK is then called to determine how many such chunks are available in the NDF (the result is returned via the NCHUNK argument). The routine NDF_CHUNK is then called repeatedly to create a sequence of NDF sections which refer to each chunk in turn. These chunks are specified by the chunk index ICHUNK, which varies from 1 to NCHUNK. The NDF identifier for each section created by NDF_CHUNK is annulled when it is no longer required.

In the 1-dimensional case (as here), this process of chunking is straightforward, and could have been programmed directly without too much difficulty. However, in the N-dimensional case (with N > 1), the number of pixels in each NDF section (chunk) must be the product of N separate dimension sizes. Thus, not all sizes of chunk are available. In addition, each chunk must fit within the bounds of the NDF from which it is derived. As a result, the size and shape of each chunk returned by NDF_CHUNK may vary (although the sequence of chunks generated is always repeatable if several NDFs with the same shape are processed using the same value of MXPIX).

The purpose of chunking is to divide an NDF into pieces, each of which contains contiguous pixels (i.e. pixels whose storage locations in the NDF follow one after the other) with the chunks themselves also following each other contiguously in pixel-storage order.[*] NDF_CHUNK will accomplish this for any shape of NDF, and for any limit on the maximum number of pixels in a chunk. Thus, by appropriately defining MXPIX, a limit can be set on memory usage regardless of the total size of NDF being processed.

In fact, by setting MXPIX to certain special values it is possible to partition an NDF into chunks of pre-determined shape. For instance, if MXPIX is set equal to the first dimension size, then NDF_CHUNK will step through the NDF one ``line'' at a time. Similarly, if MXPIX is set to the product of the first two dimension sizes, then NDF_CHUNK will step through each ``plane'' of a 3-dimensional stack of images. The following example shows how this might be used to apply a smoothing algorithm to each image in a 3-dimensional stack without needing to access the entire NDF at once:

*  Obtain the input NDF shape and set MXPIX.
      CALL NDF_DIM( INDF, NDF__MXDIM, DIM, NDIM, STATUS )
      MXPIX = DIM( 1 ) * DIM( 2 )

*  Create the output NDF, using the input NDF as a template.
      CALL NDF_PROP( INDF1, ' ', 'OUT', INDF2, STATUS )

*  Determine the number of chunks (i.e. image planes) and loop through
*  them.
      CALL NDF_NCHNK( INDF1, MXPIX, NCHUNK, STATUS )
      DO 1 ICHUNK = 1, NCHUNK

*  Start a new NDF context and create sections for the input/output
*  image planes.
         CALL NDF_BEGIN
         CALL NDF_CHUNK( INDF1, MXPIX, ICHUNK, ICH1, STATUS )
         CALL NDF_CHUNK( INDF2, MXPIX, ICHUNK, ICH2, STATUS )

*  Access the data.
         CALL NDF_MAP( ICH1, 'Data', '_REAL', 'READ', PNTR1, EL, STATUS )
         CALL NDF_MAP( ICH2, 'Data', '_REAL', 'WRITE', PNTR2, EL, STATUS )

         <perform the smoothing operation>

*  End the NDF context.
         CALL NDF_END( STATUS )
 1    CONTINUE

Note the use of an NDF context to simplify ``cleaning up'' after processing each chunk.



next up previous
Next: Blocking
Up: NDF SECTIONS
Previous: More Advanced Use: Partitioning


Starlink User Note 33
R.F. Warren-Smith
11th January 2000
E-mail:rfws@star.rl.ac.uk

Copyright © 2000 Council for the Central Laboratory of the Research Councils