KIVA-3 is a finite-difference code and solves fluid, particle and chemistry equations over a mesh made up arbitrary hexahedrons. Each hexahedron is a cell with vertices located at the corners. Most of the physical quantities are cell-averaged and are computed via cell control volumes. The momentum field is computed via momentum cells that are centered at the vertex locations. Cell-face velocities are computed at cell faces via cell-face volumes centered around the face it involves. For a typical representation of these computational volumes (regular cells, momentum cells, and cell-face volumes) see Fig. 1.
A regular cell is represented by its lower left front vertex, namely i4. Notice that the loops over the regular cells will have to sweep only the vertices representing real cells. The vertices located on the right, back and top faces of the global mesh represent what we call ghost cells that could be used in applying the boundary conditions. In a distributed implementation, these ghost cells may correspond to the real cells of the adjacent blocks residing on other processors. Since the global mesh is covered with ghost cells on the left, front and bottom directions, those cells could also correspond to the real cells of adjacent processors in those directions.
Discretized equations for mass density, specific internal energy and turbulence equations are solved for cell-centered quantities and thus require both cell-center and cell-face values of variables involved. The evaluation of variables on the cell faces is done through averaging the cells that share the cell face in question. Thus, cell-face averaging is a potential for communication between adjacent processors for the cell faces on the boundary.
The momentum equation is solved at vertex locations and thus requires quantities both at vertex locations and momentum-cell faces. Momentum cells are centered around the vertices and occupy 1/8th volume of each cell volume that share the vertex in question. A momentum cell then has 24 cell faces, three of which reside in each cell that shares the vertex. Evaluating quantities on the boundary momentum cell faces will have to involve processor communication. Indeed, there will be contributions to the momentum of boundary vertices from adjacent processors and one would need to account for that.
The solution procedure also involves computing cell-face velocities that require pressure contributions from both sides of the faces. Again, for the cell faces on the boundary this means communication. A cell-face volume is centered around the face in question and has faces that coincide with regular cell faces surrounding it.
Advection of mass, energy, momentum and turbulence kinetic energy and dissipation also require quantities evaluated at regular and momentum cell faces. One can use either Quasi Second Order Upwind (QSOU) or first order Partial Donor Cell (PDC) Upwind to evaluate the quantities on the cell faces. For cell-centered variables (mass, energy, etc) this requires derivatives at cell centers on either sides of the face. For momentum, this requires derivatives of velocity field at vertex locations on both sides of the vertex at which the momentum cell is located.
Figure 1: Computational Volumes