KIVA-3  is written for vector computers, particularly CRAYs. It differs from KIVA-2  in several ways, most distinctly in the grid structure. KIVA-2 uses a tensor-product mesh and covers the whole problem domain in a single block whereas KIVA-3 offers the ability of covering the problem domain with as many blocks as possible as long as the connectivity between blocks is maintained properly. This block-structured mesh frees one from the constraint of using large numbers of ghost cells to cover complex geometries all in one piece. Instead, one can represent each part of the engine with a separate block and let the program patch them together with the connectivity arrays that describe who the neighbor points are in all directions (left, right, back, top, bottom and front). The connectivity arrays make it unnecessary to keep the cell centers in a particular order, since through indirect addressing one can have access to the rest of the mesh system. Although block-structured mesh leads to the presence of fewer ghost cells in general, there is another change in KIVA-3 that leads to the creation of previously unneeded ghost cells. Earlier versions of the code introduced the ghost cells only in the right, back and top, whereas the latest version includes ghost cells on the left, front and bottom as well to simplify the inflow/outflow boundary conditions. Also, cell-face boundary conditions for all six faces of a cell permit greater flexibility and simplification in the application of the boundary conditions. Altogether, the changes in KIVA-3 create a quite general receipe of physics, numerics and boundary conditions that can be applied to a part of the domain as well as the whole domain, and thereby provides for a convenient block-wise distributed implementation.
There are many potential areas in the code that can use the convenience of having the connectivity arrays even for non-distributed implementations. In KIVA-2, the computational loops have to sweep over all the ghost cells as well as the active cells to be able to cover the whole domain without violating the vectorization rules. For a system of 150,000 mesh points there may be as many as 50,000 ghost cells and KIVA-2 would waste a third of the execution time on ghost cells. However, in KIVA-3, the ghost cells are separated from the active cells and the computational loops need to sweep only the active cells, which saves significant execution time .
The use of connectivity arrays and indirect addressing is advantageous, but there are disadvantages associated with indirect addressing as well, particularly for parallel implementations that are based on loop decomposition. In fact, loop decomposition is prohibitive under these circumstances. Instead of dividing the loop among processors, one can divide the problem geometrically such that each processor's computation is localized to a block of meshes that are in physical contact. Indirect addressing, then, will not be a problem for processors to handle elements in their own region so long as they can identify the elements that must be exchanged with the neighbor processor during communication. That is to suggest one to stick to the block-structured spirit of original KIVA-3 and decompose the problem in a way that KIVA-3 operates on one of these blocks on each processor separately.
A detailed analysis of the code demands that one know what the computational cells are and the physics and numerics taking place in cell centers, vertices, momentum cells, and cell face volumes to be able to parallelize the computational loops. The outcome of our analysis is given in the following sections. The code has a great deal of physics in it including particle dynamics, fluid dynamics and chemistry, and thus introduces several types of computational cells to carry the conservation laws. Following a section on the description of the cells, we will discuss the dependencies for diffusion, advection and spray dynamics taking place in those volumes.