Spack and Uberenv

GEOS is transitioning to a new Spack and Uberenv system for building our dependencies. We refer the reader to the Spack documentation and Uberenv documentation, in particular the Spack documentation for specs and dependencies, manual compiler configuration and external packages are worth reading.

Building the dependencies can be as simple as running

./scripts/uberenv/uberenv.py

This will create a directory uberenv_libs (or a directory name you specify by adding --prefix directory-name) in the current working directory, clone Spack into uberenv_libs/spack and install the dependencies into uberenv_libs/system_dependent_path. It will then spit out a host-config file in the current directory which you can use to build GEOS. While the above command should work on every system, it should never be used. Invoked as such, Spack will ignore any system libraries you have installed and will go down a rabbit hole building dependencies. Furthermore this does not allow you to choose the compiler to build. Both of these are easily solved by creating a directory with a spack.yaml.

To prevent this from happening you’ll need to create a directory with a spack.yaml file. You can find working examples for commonly used systems in scripts/spack_configs.

Once you have these files setup you can run Uberenv again and instruct it to use them with. If for instance you added Clang 10.0.1 to the spack.yaml file the your command would look something like this:

./scripts/uberenv/uberenv.py --spack-config-dir=/path/to/your/config/directory/ --spec="%[email protected]"

It is worth noting that GEOS has two project json files (.uberenv_config.json and scripts/pygeosx_configs/pygeosx.json) and two configuration directories for LC systems (scripts/spack_configs and scripts/pygeosx_configs). The .uberenv_config.json project json file and scripts/spack_configs directory is for building GEOS. The scripts/pygeosx_configs/pygeosx.json project json file and scripts/pygeosx_configs directory is for building pygeosx. This is because pygeosx has a separate list of required compilers and packages to build from GEOS (e.g. pygeosx’s numpy dependency recommends building with gcc and using openblas for BLAS/LAPACK). However, when not building pygeosx other dependencies depend on python, but an existing system version works just fine, so it can be put in GEOS’s spack.yaml to prevent Spack from building it. By default, Uberenv will find and use .uberenv_config.json to build GEOS, but you can use the --project-json command line option to target scripts/pygeosx_configs/pygeosx.json to build pygeosx:

./scripts/uberenv/uberenv.py --spack-config-dir=/path/to/your/config/directory/ --spec="%[email protected]" --project-json="scripts/pygeosx_configs/pygeosx.json"

Note

When building pygeosx, Spack will build various python packages, however by default they are not installed in python. There are various ways of accomplishing this, but the recommended approach is to use spack environments. Once you build pygeosx using Uberenv, Spack will create a view that ensures the Spack-built python can find the built python packages. For example, with a default uberenv_libs directory of dependencies, the path to the view of python will be uberenv_libs/._view/*/bin/python3. If you want to use your pygeosx python3 executable in GEOS, you will need to update your host-config’s Python3_ROOT_DIR and Python3_EXECUTABLE to the path to Spack’s view of python.

Build Configuration

Warning

The spack build system is undergoing updates. The petsc variant and others are still a work in progress.

The GEOS Spack package has a lot of options for controlling which dependencies you would like to build and how you’d like them built. The GEOS Spack package file is at `scripts/spack_packages/packages/geosx/package.py <https://github.com/GEOS-DEV/GEOS/tree/develop/scripts/spack_packages/packages/geosx/package.py>`_. The variants for the package are as follows

    variant('shared', default=True, description='Build Shared Libs.')
    variant('caliper', default=True, description='Build Caliper support.')
    variant('vtk', default=True, description='Build VTK support.')
    variant('fesapi', default=False, description='Build fesapi support.')
    variant('trilinos', default=True, description='Build Trilinos support.')
    variant('hypre', default=True, description='Build HYPRE support.')
    variant('petsc', default=False, description='Build PETSc support.')
    variant('scotch', default=True, description='Build Scotch support.')
    variant('uncrustify', default=True, description='Build Uncrustify support.')
    variant('lai',
            default='hypre',
            description='Linear algebra interface.',
            values=('trilinos', 'hypre', 'petsc'),
            multi=False)
    variant('pygeosx', default=True, description='Enable pygeosx.')

For example if you wanted to build with GCC 8.3.1, without Caliper and with Hypre as the Linear Algebra Interface, your spec would be %gcc@8.3.1 ~caliper lai=hypre.

The GEOS Spack package lists out the libraries that GEOS depends ons. Currently these dependencies are

    depends_on('[email protected]:', type='build')

    depends_on('blt')

    #
    # Virtual packages
    #
    depends_on('mpi')
    depends_on('blas')
    depends_on('lapack')

    #
    # Performance portability
    #
    depends_on('raja +openmp~examples~exercises~shared')

    depends_on('umpire +c+openmp~examples+fortran~device_alloc~shared')

    depends_on('[email protected] +raja+openmp~examples~shared')

    depends_on('camp')

    with when('+cuda'):
        for sm_ in CudaPackage.cuda_arch_values:
            depends_on('raja+cuda cuda_arch={0}'.format(sm_), when='cuda_arch={0}'.format(sm_))
            depends_on('umpire+cuda cuda_arch={0}'.format(sm_), when='cuda_arch={0}'.format(sm_))
            depends_on('chai+cuda cuda_arch={0}'.format(sm_), when='cuda_arch={0}'.format(sm_))
            depends_on('camp+cuda cuda_arch={0}'.format(sm_), when='cuda_arch={0}'.format(sm_))

    #
    # IO
    #
    depends_on('[email protected]')
    depends_on('[email protected]~fortran')

    depends_on('[email protected]~test~fortran~hdf5_compat')

    depends_on('[email protected]', when='+caliper')
    depends_on('[email protected]~gotcha~sampler~libunwind~libdw', when='+caliper')

    depends_on('[email protected]')

    depends_on('[email protected] cxxstd=14')
    depends_on('[email protected]', when='+vtk')

    depends_on('fesapi', when='+fesapi')

    #
    # Math
    #
    depends_on('[email protected]+int64')

    depends_on('superlu-dist +int64+openmp')

    depends_on('[email protected] +mpi +int64', when='+scotch')

    depends_on('[email protected]+openmp')

    trilinos_build_options = '+openmp'
    trilinos_packages = '+aztec+stratimikos~amesos2~anasazi~belos~ifpack2~muelu~sacado+thyra'
    depends_on('[email protected] ' + trilinos_build_options + trilinos_packages, when='+trilinos')

    depends_on("hypre +superlu-dist+mixedint+mpi+openmp", when='+hypre~cuda')

    depends_on("hypre +cuda+superlu-dist+mixedint+mpi+openmp+umpire+unified-memory cxxflags='-fPIC'", when='+hypre+cuda')
    with when('+cuda'):
        for sm_ in CudaPackage.cuda_arch_values:
            depends_on('hypre+cuda cuda_arch={0}'.format(sm_), when='cuda_arch={0}'.format(sm_))

    depends_on('[email protected]~hdf5~hypre+int64', when='+petsc')
    depends_on('petsc+ptscotch', when='+petsc+scotch')

    #
    # Python
    #
    depends_on('python')


    #
    # Dev tools
    #
    depends_on('uncrustify', when='+uncrustify')

    #
    # Documentation
    #
    depends_on('[email protected]', when='+docs', type='build')
    depends_on('[email protected]:', when='+docs', type='build')

Using the Spack spec syntax you can inturn specify variants for each of the dependencies of GEOS. So for example if you could modify the spec above to build RAJA in debug by using %gcc@8.3.1 ~caliper lai=hypre ^raja build_type=Debug. When building with Uberenv Spack should print out a table containing the full spec for every dependency it will build. If you would like to look at the variants for say RAJA in more detail you can find the package file at uberenv_libs/spack/var/spack/repos/builtin/packages/raja/package.py.

Adding a Dependency (Advanced)

Adding a dependency to GEOS is straight forward if the dependency already builds with Spack. If that is the case then all you need to do is add a depends_on('cool-new-library') to the GEOS package.py file. If however the dependency doesn’t have a Spack package, you will have to add one by creating a cool-new-library/package.yaml file in the scripts/spack_packages/packages directory and adding the logic to build it there.

Oftentimes (unfortunately), even when a package already exists, it might not work out of the box for your system. In this case copy over the existing package.py file from the Spack repository into scripts/spack_packages/packages/cool-new-library/package.py, as if you were adding a new package, and perform your modifications there. Once you have the package working, copy the package back into the Spack repository (running Uberenv should do this for you) and commit+push your changes to Spack.