Spack and Uberenv

GEOS is transitioning to a new Uberenv and Spack system for building our dependencies. We refer the reader to the Uberenv documentation and Spack documentation, in particular the Spack documentation sections worth reading are:

Building the dependencies can be as simple as running:

./scripts/uberenv/uberenv.py

from the thirdPartyLibs directory. This will create a directory uberenv_libs (or a directory name you specify by adding --prefix directory-name) in the current working directory, clone Spack into uberenv_libs/spack and install the dependencies into uberenv_libs/system_dependent_path. It will then spit out host-config files (see Host-Config Generation) in the current directory which you can use to build GEOS or LvArray. While the above command should work on every system, it should never be used to build GEOS. Invoked as such, Spack will ignore any system libraries you have installed and will go down a rabbit hole building dependencies. Furthermore this does not allow you to choose the compiler to build with. Both of these are easily solved by creating a spack.yaml configuration file, also known in Spack as an environment file, to tell Spack where pre-installed system libraries and compiles are located. See spack.yaml for more on how to create a spack.yaml file.

Once you have the spack.yaml file setup, you can run Uberenv again and instruct it to use the environment file with the command line option --spack-env-file. If for instance you added Clang 14.0.6 to the spack.yaml file, then your command to build the dependencies would look something like this:

./scripts/uberenv/uberenv.py --spack-env-file=/path/to/your/spack.yaml --spec="%[email protected]" --prefix=/output/path/to/third-party/GEOS/libraries

For more Uberenv command-line options, you can run the uberenv.py script with the --help option or consult the command line options.

Note

There is no requirement that your environment file be named spack.yaml when it is passed to Uberenv using the --spack-env-file command line option.

Note

On LC systems only, there is not a requirement to specify --spack-env-file. This is because Uberenv uses the environment variable SYS_TYPE in combination with the .uberenv_config.json Uberenv configuration file to determine the folder name that contains the required spack.yaml file (e.g. scripts/spack_configs/blueos_3_ppc64le_ib_p9/spack.yaml). More information on Uberenv configuration behavior can be found in Uberenv spack configurations documentation.

spack.yaml

The spack.yaml configuration file tells Spack where it can find relevant packages and compilers to build GEOS third-party dependencies. Without spack.yaml, building the dependencies will take significantly longer.

There are many examples and resources available for constructing a spack.yaml file:

spack.yaml from scratch

If the examples and resources listed in spack.yaml are not applicable to your system, or you would like to see what packages are already installed on your system, you can call Uberenv with the following option:

./scripts/uberenv/uberenv.py --setup-and-env-only

This command will setup Spack and ask Spack create a spack.yaml environment file for you. Uberenv will invoke spack compiler find and spack external find to find pre-installed compilers and packages on your system.

Note

These spack commands called underneath are not perfect and are not guaranteed to find every compiler and every package available on your system. In fact, the commands can be non-deterministic, and you can get two different spack.yaml environment files from two separate invocations! Also, not all the packages found are relevant dependencies or sub-dependencies for GEOS. Of the packages that are relevant, they may not be the right version!

This command should be used as a first approximation of your system environment, to determine the paths where more suitable compilers and packages are potentially located.

Required package versions in spack.yaml

In the LC configuration files mentioned in spack.yaml section, you will see a list of packages with the require keyword:

hypre:
  require: "@git.06da35b1a4b1066a093bc0c6c48aee12bee74cd4"
...

This tells Spack that GEOS always requires a specific commit of hypre, a commit on the latest develop branch in this case. Ideally, package versions should be specified in the GEOS Spack package file. However, when a version of a package is newer than what Spack knows about or an unversioned commit is needed, the Spack package syntax cannot express that requirement. As a result:

Warning

Every spack.yaml file must have the list of packages with the “require” keyword to ensure these packages are built with the right versions or commits. See LC configuration files mentioned in spack.yaml section for the list.

Uberenv configuration file

Uberenv needs a .uberenv_config.json configuration file to function as a submodule. Details on the various configuration options can be found in Uberenv project configuration documentation. The most notable option for maintenance is spack_commit, which is the Spack commit that Uberenv checkouts to build the dependencies.

pygeosx

Warning

The spack build system for pygeosx is a work in progress.

It is worth noting that GEOS has two project json files (.uberenv_config.json and scripts/pygeosx_configs/pygeosx.json) and two configuration directories for LC systems (scripts/spack_configs and scripts/pygeosx_configs). The .uberenv_config.json project json file and scripts/spack_configs directory is for building GEOS dependencies. The scripts/pygeosx_configs/pygeosx.json project json file and scripts/pygeosx_configs directory is for building pygeosx dependencies.This is because pygeosx has a separate list of required compilers and packages to build from GEOS (e.g. pygeosx’s numpy dependency recommends building with gcc and using openblas for BLAS/LAPACK). When not building pygeosx, other dependencies of GEOS still depend on python. An existing system version of python will work just fine, and can be put in GEOS’s spack.yaml to prevent Spack from building its own verion of python. By default, Uberenv will find and use .uberenv_config.json to build GEOS, but you can use the --project-json command line option to target scripts/pygeosx_configs/pygeosx.json to build pygeosx:

./scripts/uberenv/uberenv.py --spack-config-dir=/path/to/your/config/directory/ --spec="%[email protected]" --project-json="scripts/pygeosx_configs/pygeosx.json"

Note

When building pygeosx, Spack will build various python packages, however by default they are not installed in python. There are various ways of accomplishing this, but the recommended approach is to use spack environments. Once you build pygeosx using Uberenv, Spack will create a view that ensures the Spack-built python can find the built python packages. For example, with a default uberenv_libs directory of dependencies, the path to the view of python will be uberenv_libs/._view/*/bin/python3. If you want to use your pygeosx python3 executable in GEOS, you will need to update your host-config’s Python3_ROOT_DIR and Python3_EXECUTABLE to the path to Spack’s view of python.

Build Configuration

Warning

The spack build system is undergoing updates. The petsc variant and others are still a work in progress.

The GEOS Spack package has a lot of options, or what Spack calls variants, for controlling which dependencies you would like to build and how you’d like them built. The GEOS Spack package file has variants that are marked with variant() in the file.

For example if you wanted to build with GCC 8.3.1, without Caliper and with Hypre as the Linear Algebra Interface, your spec would be %gcc@8.3.1 ~caliper lai=hypre.

The GEOS Spack package lists out the libraries that GEOS depends ons. These dependencies are marked with depends_on() in the file.

Using the Spack spec syntax, you can inturn specify variants for each of the dependencies of GEOS. For example, you could modify the spec above to build RAJA in debug mode by using %gcc@8.3.1 ~caliper lai=hypre ^raja build_type=Debug. When building with Uberenv, Spack should print out a table containing the full spec for every dependency it will build. If you would like to look at the variants for say RAJA in more detail, you can find the package file at uberenv_libs/spack/var/spack/repos/builtin/packages/raja/package.py, by using file finder on the Spack Github website, or by searching for the package at https://packages.spack.io/.

Host-Config Generation

The logic for generating the host-configs can be found in the GEOS spack recipe. The GEOS host-config is generated by the geos_hostconfig() function, while the LvArray host-config is generated by the lvarray_hostconfig() function. After successfully building all the third-party dependencies, Spack will call these two functions to populate two host-configs based on information it knows about the dependencies.

Note

The host-config generation is currently based on LC systems, and the generated host-config may be missing or have incorrect details for your system (e.g. choice of MPIEXEC_NUMPROC_FLAG). Please modify the python functions and/or host-configs generated as needed.

LC TPL Build Script

On LC systems, it is necessary to update the third-party library installations after a change to the configuration. The setupLC-TPL-uberenv.bash script is used to build the third-party libraries on multiple LC systems using uberenv:

./setupLC-TPL-uberenv.bash /path/to/shared/installation/directory

This command will also generate a LvArray and GEOS host-config for each specified machine and compiler combination.

Note

The terminal output from the command may fail to update. In that case, you can track the progress of the builds by looking at the generated *.log file associated with each machine and compiler combination (e.g. ruby-gcc-12.log).

Adding a Dependency (Advanced)

Adding a dependency to GEOS is straight forward if the dependency already builds with Spack. If that is the case, then all you need to do is add a depends_on('cool-new-library') to the GEOS package.py file. If however the dependency doesn’t have a Spack package, you will have to add one by creating a cool-new-library/package.py file in the scripts/spack_packages/packages directory and adding the logic to build it there. For instructions on how to create a package recipe from scratch, Spack has provided a Spack Packing Guide.

Oftentimes (unfortunately), even when a package already exists in Spack, it might not work out of the box for your system. In this case copy over the existing package.py file from the Spack repository into scripts/spack_packages/packages/cool-new-library/package.py, as if you were adding a new package, and perform your modifications there. Once you have the package working, copy the package back into the Spack repository and commit+push your changes to Spack.