I am working on implementing a boundary element method for hydrodynamic forces with gradients in OpenMDAO. I have had success with moderate sized problems (size here meaning the number of panels in the mesh, and thus size of the matrices in the problem), but have encountered a PETSc error when increasing the problem size. The memory/computational time requirements are fairly large, around 200GB total for this moderate size problem running on 24 processes, so I have been running on a cluster. The error is an index out of range in the PETSc scatter that occurs during setup of the transfers.
Specifically the error message (repeated from each process) is:
Traceback (most recent call last):
File "/cluster/projects/nn0000k/peterjr/MeshOpt/check_totals.py", line 293, in <module>
prob.run_model()
File "/cluster/projects/nn0000k/peterjr/venv_meshopt_310/lib/python3.10/site-packages/openmdao/core/problem.py", line 655, in run_model
self.final_setup()
File "/cluster/projects/nn0000k/peterjr/venv_meshopt_310/lib/python3.10/site-packages/openmdao/core/problem.py", line 1070, in final_setup
self.model._final_setup(self.comm, self._orig_mode)
File "/cluster/projects/nn0000k/peterjr/venv_meshopt_310/lib/python3.10/site-packages/openmdao/core/group.py", line 1325, in _final_setup
self._setup_transfers(self._abs_desvars, self._abs_responses)
File "/cluster/projects/nn0000k/peterjr/venv_meshopt_310/lib/python3.10/site-packages/openmdao/core/group.py", line 3258, in _setup_transfers
self._vector_class.TRANSFER._setup_transfers(self, desvars, responses)
File "/cluster/projects/nn0000k/peterjr/venv_meshopt_310/lib/python3.10/site-packages/openmdao/vectors/petsc_transfer.py", line 80, in _setup_transfers
subsys._setup_transfers(desvars, responses)
File "/cluster/projects/nn0000k/peterjr/venv_meshopt_310/lib/python3.10/site-packages/openmdao/vectors/petsc_transfer.py", line 83, in _setup_transfers
'fwd': PETScTransfer._setup_transfers_fwd(group, desvars, responses)
File "/cluster/projects/nn0000k/peterjr/venv_meshopt_310/lib/python3.10/site-packages/openmdao/vectors/petsc_transfer.py", line 136, in _setup_transfers_fwd
transfers[None] = PETScTransfer(group._vectors['input']['nonlinear'],
File "/cluster/projects/nn0000k/peterjr/venv_meshopt_310/lib/python3.10/site-packages/openmdao/vectors/petsc_transfer.py", line 59, in __init__
self._scatter = PETSc.Scatter().create(out_vec._petsc, out_indexset, in_vec._petsc,
File "petsc4py/PETSc/Scatter.pyx", line 46, in petsc4py.PETSc.Scatter.create
petsc4py.PETSc.Error: error code 63
[ 0] VecScatterCreate() at /cluster/work/users/hrn/build/PETSc/3.19.2/foss-2022b-ind64/petsc-3.19.2/src/vec/is/sf/interface/vscat.c:736
[ 0] Argument out of range
subsys._setup_transfers(desvars, responses)
File "/cluster/projects/nn0000k/peterjr/venv_meshopt_310/lib/python3.10/site-packages/openmdao/core/group.py", line 3258, in _setup_transfers
[ 0] Scatter indices in ix are out of range
subsys._setup_transfers(desvars, responses)
I am using OpenMDAO version 3.30 with:
- OpenMPI: 4.1.4
- PETSc: 3.19.2
- Python: 3.10.8
- numpy: 1.24.2
- scipy: 1.10.1
I have tried to replicate this problem using a very large beam problem, but have not managed to reproduce the same error yet therefore I haven't submitted this as a bug report.
I had a previous error on the larger problems with PETSc encountering overflow in integer operations, so I have reinstalled PETSc using 64-bit indices.
My next thought was that perhaps the larger problems I am attempting to solve are large enough that the matrix-free architecture is required? If so I assume that would then require a re-write of all compute_partials functions as compute_jacvec_product functions?