A typical representation of the block grid system for KIVA-3 is shown in Fig. 2. This represents a tensor-product grid with single-index cell numbering. Cell index increases in an x-y-z order sweeping x-y planes one at a time in increasing z-direction. As we have mentioned, an active cell is represented by its i4 vertex leading the loops to include ghost cells if they are not sorted out. Notice that the real physical domain is surrounded by ghost cells in all directions. The boundary vertices on the right, back, and top are real vertices, but they represent cell volumes that are not part of the physical domain. There are flags (F for cells, FV for vertices) assoicated with each cell or vertex to indicate if they are real or not. The flags take values of 1.0 or 0.0. Based on the flag values one can sort out the elements on such a grid in different groups. Elements with correspond to the ghost cells on the left, front, and bottom, whereas elements with correspond to the active cells, elements with correspond to ghost cells on the right, back, and top, and elements with F>0 correspond to real vertices as distinguished in Fig. 2. The storage of array elements according to such sorting in KIVA-3 is illustrated in Fig. 3.
Since the access pattern for vertices and cells on the boundaries need to be known in advance for communication requirements, one needs to further sort the groups within themselves separating those on the left, right, and so on. One would also have to assume that the boundary shared between neighbor processors has the same grid points on both sides to assure proper communication. The suggested sorting could be either done in the pre-processor or KIVA-3 program itself.
Figure 2: Grid Structure as shown in x-y planes.
Figure 3: Storage sorted by F and FV to maintain the shortest possible vector lengths.