Basic profiling with CALIPER

GEOS is equipped with Caliper timers. We integrate Caliper into GEOS by marking source-code sections of interest such as computational kernels or initialization steps. Caliper is included in the GEOS TPL library and is built by adding the following cmake configuration to a host-config file.

option( ENABLE_CALIPER "Enables CALIPER" On )

GEOS/Caliper Annotation Macros

The following macros may be used to annotate GEOS:

  • GEOS_MARK_SCOPE(name) - Marks a scope with the given name.

  • GEOS_MARK_FUNCTION - Marks a function with the name of the function. The name includes the namespace the function is in but not any of the template arguments or parameters. Therefore overloaded function all show up as one entry. If you would like to mark up a specific overload use GEOS_MARK_SCOPE with a unique name.

  • GEOS_MARK_BEGIN(name) - Marks the beginning of a user defined code region.

  • GEOS_MARK_END(name) - Marks the end of user defined code region.

The 2 first macros also generate annotations for NVTX is ENABLE_CUDA_NVTOOLSEXT is activated through CMake.

Configuring Caliper

Caliper configuration is done by specifying a string to initialize Caliper with via the -t option. A few options are listed below but we refer the reader to Caliper Config for the full Caliper tutorial.

  • -t runtime-report,max_column_width=200 Will make Caliper print aggregated timing information to standard out, with a column width large enought that it doesn’t truncate most function names.

  • -t runtime-report,max_column_width=200,profile.cuda Does the same as the above, but also instruments CUDA API calls. This is only an option when building with CUDA.

  • -t runtime-report,aggregate_across_ranks=false Will make Caliper write per rank timing information to standard out. This isn’t useful when using more than one rank but it does provide more information for single rank runs.

  • -t spot() Will make Caliper output a .cali timing file that can be viewed in the Spot web server.

Using Adiak

Adiak is a library that allows the addition of meta-data to the Caliper Spot output, it is enabled with Caliper. This meta-data allows you to easily slice and dice the timings available in the Spot web server. To export meta-data use the adiak::value function.

See Adiak API for the full Adiak documentation.

Using Spot

To use Spot you will need an LC account and a directory full of .cali files you would like to analyse. Point your browser to Spot and open up the directory containing the timing files.

Opening Spot caliper files in Python

An example Python program for analyzing Spot Caliper files in Python is provided below. Note that it requires pandas and hatchet both of which can be installed with a package manager. In addition it requires that cali-query is in the PATH variable, this is built with Caliper so we can just point it into the TPLs.

import sys
import subprocess
import json
import os

import pandas as pd
from IPython.display import display, HTML

# Import hatchet, on LC this can be done by adding hatchet to PYTHONPATH
sys.path.append('/usr/gapps/spot/live/hatchet')
import hatchet as ht

# Add cali-query to PATH
cali_query_path = "/usr/gapps/GEOSX/thirdPartyLibs/2020-06-12/[email protected]/caliper/bin"
os.environ["PATH"] += os.pathsep + cali_query_path

CALI_FILES = [
{ "cali_file": "/usr/gapps/GEOSX/timingFiles/200612-04342891243.cali", "metric_name": "avg#inclusive#sum#time.duration"},
{ "cali_file": "/usr/gapps/GEOSX/timingFiles/200611-044740108300.cali", "metric_name": "avg#inclusive#sum#time.duration"},
]

grouping_attribute = "prop:nested"
default_metric = "avg#inclusive#sum#time.duration"
query = "select %s,sum(%s) group by %s format json-split" % (grouping_attribute, default_metric, grouping_attribute)

gf1 = ht.GraphFrame.from_caliper(CALI_FILES[0]['cali_file'], query)
gf2 = ht.GraphFrame.from_caliper(CALI_FILES[1]['cali_file'], query)

# Print the tree representation using the default metric
# Also print the resulting dataframe with metadata
print(gf1.tree(color=True, metric="sum#"+default_metric))
display(HTML(gf1.dataframe.to_html()))

# Print the tree representation using the default metric
# Also print the resulting dataframe with metadata
print(gf2.tree(color=True, metric="sum#"+default_metric))
display(HTML(gf2.dataframe.to_html()))

# Compute the speedup between the first two cali files (exlusive and inclusive metrics only)
gf3 = (gf1 - gf2) / gf2
print(gf3.tree(color=True, metric="sum#"+default_metric))

# Compute the difference between the first two cali files (exclusive and inclusive metrics only)
# Print the resulting tree
gf4 = gf1 - gf2
print(gf4.tree(color=True, metric="sum#"+default_metric))

# Compute the sum of the first two cali files (exclusive and inclusive metrics only)
# Print the resulting tree
gf5 = gf1 + gf2
print(gf5.tree(color=True, metric="sum#"+default_metric))