# Launching and configuring HPX applications¶

## Configuring HPX applications¶

All HPX applications can be configured using special command line options and/or using special configuration files. This section describes the available options, the configuration file format, and the algorithm used to locate possible predefined configuration files. Additionally this section describes the defaults assumed if no external configuration information is supplied.

During startup any HPX application applies a predefined search pattern to locate one or more configuration files. All found files will be read and merged in the sequence they are found into one single internal database holding all configuration properties. This database is used during the execution of the application to configure different aspects of the runtime system.

In addition to the ini files, any application can supply its own configuration files, which will be merged with the configuration database as well. Moreover, the user can specify additional configuration parameters on the command line when executing an application. The HPX runtime system will merge all command line configuration options (see the description of the --hpx:ini, --hpx:config, and --hpx:app-config command line options).

### The HPX INI File Format¶

All HPX applications can be configured using a special file format which is similar to the well-known Windows INI file format. This is a structured text format allowing to group key/value pairs (properties) into sections. The basic element contained in an ini file is the property. Every property has a name and a value, delimited by an equals sign '='. The name appears to the left of the equals sign:

name=value


The value may contain equal signs as only the first '=' character is interpreted as the delimiter between name and value Whitespace before the name, after the value and immediately before and after the delimiting equal sign is ignored. Whitespace inside the value is retained.

Properties may be grouped into arbitrarily named sections. The section name appears on a line by itself, in square brackets [ and ]. All properties after the section declaration are associated with that section. There is no explicit “end of section” delimiter; sections end at the next section declaration, or the end of the file:

[section]


In HPX sections can be nested. A nested section has a name composed of all section names it is embedded in. The section names are concatenated using a dot '.':

[outer_section.inner_section]


Here inner_section is logically nested within outer_section.

It is possible to use the full section name concatenated with the property name to refer to a particular property. For example in:

[a.b.c]
d = e


the property value of d can be referred to as a.b.c.d=e.

In HPX ini files can contain comments. Hash signs '#' at the beginning of a line indicate a comment. All characters starting with the '#' until the end of line are ignored.

If a property with the same name is reused inside a section, the second occurrence of this property name will override the first occurrence (discard the first value). Duplicate sections simply merge their properties together, as if they occurred contiguously.

In HPX ini files, a property value ${FOO:default} will use the environmental variable FOO to extract the actual value if it is set and default otherwise. No default has to be specified. Therefore ${FOO} refers to the environmental variable FOO. If FOO is not set or empty the overall expression will evaluate to an empty string. A property value $[section.key:default] refers to the value held by the property section.key if it exists and default otherwise. No default has to be specified. Therefore $[section.key] refers to the property section.key. If the property section.key is not set or empty, the overall expression will evaluate to an empty string.

Note

Any property $[section.key:default] is evaluated whenever it is queried and not when the configuration data is initialized. This allows for lazy evaluation and relaxes initialization order of different sections. The only exception are recursive property values, e.g. values referring to the very key they are associated with. Those property values are evaluated at initialization time to avoid infinite recursion. ### Built-in Default Configuration Settings¶ During startup any HPX application applies a predefined search pattern to locate one or more configuration files. All found files will be read and merged in the sequence they are found into one single internal data structure holding all configuration properties. As a first step the internal configuration database is filled with a set of default configuration properties. Those settings are described on a section by section basis below. Note You can print the default configuration settings used for an executable by specifying the command line option --hpx:dump-config. #### The system configuration section¶ [system] pid = <process-id> prefix = <current prefix path of core HPX library> executable = <current prefix path of executable>   Property Description system.pid This is initialized to store the current OS-process id of the application instance. system.prefix This is initialized to the base directory HPX has been loaded from. system.executable_prefix This is initialized to the base directory the current executable has been loaded from. #### The hpx configuration section¶ [hpx] location =${HPX_LOCATION:$[system.prefix]} component_path =$[hpx.location]/lib/hpx:$[system.executable_prefix]/lib/hpx:$[system.executable_prefix]/../lib/hpx
master_ini_path = $[hpx.location]/share/hpx-<version>:$[system.executable_prefix]/share/hpx-<version>:$[system.executable_prefix]/../share/hpx-<version> ini_path =$[hpx.master_ini_path]/ini
localities = 1
program_name =
cmd_line =
lock_detection = ${HPX_LOCK_DETECTION:0} throw_on_held_lock =${HPX_THROW_ON_HELD_LOCK:1}
spinlock_deadlock_detection_limit = ${HPX_SPINLOCK_DEADLOCK_DETECTION_LIMIT:1000000} max_background_threads =${HPX_MAX_BACKGROUND_THREADS:$[hpx.os_threads]} max_idle_loop_count =${HPX_MAX_IDLE_LOOP_COUNT:<hpx_idle_loop_count_max>}
max_busy_loop_count = ${HPX_MAX_BUSY_LOOP_COUNT:<hpx_busy_loop_count_max>} max_idle_backoff_time =${HPX_MAX_IDLE_BACKOFF_TIME:<hpx_idle_backoff_time_max>}

[hpx.stacks]
small_size = ${HPX_SMALL_STACK_SIZE:<hpx_small_stack_size>} medium_size =${HPX_MEDIUM_STACK_SIZE:<hpx_medium_stack_size>}
large_size = ${HPX_LARGE_STACK_SIZE:<hpx_large_stack_size>} huge_size =${HPX_HUGE_STACK_SIZE:<hpx_huge_stack_size>}
use_guard_pages = ${HPX_THREAD_GUARD_PAGE:1}   Property Description hpx.location This is initialized to the id of the locality this application instance is running on. hpx.component_path Duplicates are discarded. This property can refer to a list of directories separated by ':' (Linux, Android, and MacOS) or using ';' (Windows). hpx.master_ini_path This is initialized to the list of default paths of the main hpx.ini configuration files. This property can refer to a list of directories separated by ':' (Linux, Android, and MacOS) or using ';' (Windows). hpx.ini_path This is initialized to the default path where HPX will look for more ini configuration files. This property can refer to a list of directories separated by ':' (Linux, Android, and MacOS) or using ';' (Windows). hpx.os_threads This setting reflects the number of OS-threads used for running HPX-threads. Defaults to number of detected cores (not hyperthreads/PUs). hpx.localities This setting reflects the number of localities the application is running on. Defaults to 1. hpx.program_name This setting reflects the program name of the application instance. Initialized from the command line argv[0]. hpx.cmd_line This setting reflects the actual command line used to launch this application instance. hpx.lock_detection This setting verifies that no locks are being held while a HPX thread is suspended. This setting is applicable only if HPX_WITH_VERIFY_LOCKS is set during configuration in CMake. hpx.throw_on_held_lock This setting causes an exception if during lock detection at least one lock is being held while a HPX thread is suspended. This setting is applicable only if HPX_WITH_VERIFY_LOCKS is set during configuration in CMake. This setting has no effect if hpx.lock_detection=0. hpx.minimal_deadlock_detection This setting enables support for minimal deadlock detection for HPX-threads. By default this is set to 1 (for Debug builds) or to 0 (for Release, RelWithDebInfo, RelMinSize builds), this setting is effective only if HPX_WITH_THREAD_DEADLOCK_DETECTION is set during configuration in CMake. hpx.spinlock_deadlock_detection This setting verifies that spinlocks don’t spin longer than specified using the hpx.spinlock_deadlock_detection_limit. This setting is applicable only if HPX_WITH_SPINLOCK_DEADLOCK_DETECTION is set during configuration in CMake. By default this is set to 1 (for Debug builds) or to 0 (for Release, RelWithDebInfo, RelMinSize builds). hpx.spinlock_deadlock_detection_limit This setting specifies the upper limit of allowed number of spins that spinlocks are allowed to perform. This setting is applicable only if HPX_WITH_SPINLOCK_DEADLOCK_DETECTION is set during configuration in CMake. By default this is set to 1000000. hpx.max_background_threads This setting defines the number of threads in the scheduler which are used to execute background work. By default this is the same as the number of cores used for the scheduler. hpx.max_idle_loop_count By default this is defined by the preprocessor constant HPX_IDLE_LOOP_COUNT_MAX. This is an internal setting which you should change only if you know exactly what you are doing. hpx.max_busy_loop_count This setting defines the maximum value of the busy-loop counter in the scheduler. By default this is defined by the preprocessor constant HPX_BUSY_LOOP_COUNT_MAX. This is an internal setting which you should change only if you know exactly what you are doing. hpx.max_idle_backoff_time This setting defines the maximum time (in milliseconds) for the scheduler to sleep after being idle for hpx.max_idle_loop_count iterations. This setting is applicable only if HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF is set during configuration in CMake. By default this is defined by the preprocessor constant HPX_IDLE_BACKOFF_TIME_MAX. This is an internal setting which you should change only if you know exactly what you are doing. hpx.stacks.small_size This is initialized to the small stack size to be used by HPX-threads. Set by default to the value of the compile time preprocessor constant HPX_SMALL_STACK_SIZE (defaults to 0x8000). This value is used for all HPX threads by default, except for the thread running hpx_main (which runs on a large stack). hpx.stacks.medium_size This is initialized to the medium stack size to be used by HPX-threads. Set by default to the value of the compile time preprocessor constant HPX_MEDIUM_STACK_SIZE (defaults to 0x20000). hpx.stacks.large_size This is initialized to the large stack size to be used by HPX-threads. Set by default to the value of the compile time preprocessor constant HPX_LARGE_STACK_SIZE (defaults to 0x200000). This setting is used by default for the thread running hpx_main only. hpx.stacks.huge_size This is initialized to the huge stack size to be used by HPX-threads. Set by default to the value of the compile time preprocessor constant HPX_HUGE_STACK_SIZE (defaults to 0x2000000). hpx.stacks.use_guard_pages This entry controls whether the coroutine library will generate stack guard pages or not. This entry is applicable on Linux only and only if the HPX_USE_GENERIC_COROUTINE_CONTEXT option is not enabled and the HPX_WITH_THREAD_GUARD_PAGE is set to 1 while configuring the build system. It is set by default to 1. #### The hpx.threadpools configuration section¶ [hpx.threadpools] io_pool_size =${HPX_NUM_IO_POOL_SIZE:2}
parcel_pool_size = ${HPX_NUM_PARCEL_POOL_SIZE:2} timer_pool_size =${HPX_NUM_TIMER_POOL_SIZE:2}

 Property Description hpx.threadpools.io_pool_size The value of this property defines the number of OS-threads created for the internal I/O thread pool. hpx.threadpools.parcel_pool_size The value of this property defines the number of OS-threads created for the internal parcel thread pool. hpx.threadpools.timer_pool_size The value of this property defines the number of OS-threads created for the internal timer thread pool.

#### The hpx.thread_queue configuration section¶

Important

These setting control internal values used by the thread scheduling queues in the HPX scheduler. You should not modify these settings except if you know exactly what you are doing]

[hpx.thread_queue]
min_tasks_to_steal_pending = ${HPX_THREAD_QUEUE_MIN_TASKS_TO_STEAL_PENDING:0} min_tasks_to_steal_staged =${HPX_THREAD_QUEUE_MIN_TASKS_TO_STEAL_STAGED:10}
min_add_new_count = ${HPX_THREAD_QUEUE_MIN_ADD_NEW_COUNT:10} max_add_new_count =${HPX_THREAD_QUEUE_MAX_ADD_NEW_COUNT:10}
max_delete_count = ${HPX_THREAD_QUEUE_MAX_DELETE_COUNT:1000}   Property Description hpx.thread_queue.min_tasks_to_steal_pending The value of this property defines the number of pending HPX threads which have to be available before neighboring cores are allowed to steal work. The default is to allow stealing always. hpx.thread_queue.min_tasks_to_steal_staged The value of this property defines the number of staged HPX tasks have which to be available before neighboring cores are allowed to steal work. The default is to allow stealing only if there are more tan 10 tasks available. hpx.thread_queue.min_add_new_count The value of this property defines the minimal number tasks to be converted into HPX threads whenever the thread queues for a core have run empty. hpx.thread_queue.max_add_new_count The value of this property defines the maximal number tasks to be converted into HPX threads whenever the thread queues for a core have run empty. hpx.thread_queue.max_delete_count The value of this property defines the number number of terminated HPX threads to discard during each invocation of the corresponding function. #### The hpx.components configuration section¶ [hpx.components] load_external =${HPX_LOAD_EXTERNAL_COMPONENTS:1}

 Property Description hpx.components.load_external This entry defines whether external components will be loaded on this locality. This entry normally is set to 1 and usually there is no need to directly change this value. It is automatically set to 0 for a dedicated AGAS server locality.

Additionally, the section hpx.components will be populated with the information gathered from all found components. The information loaded for each of the components will contain at least the following properties:

[hpx.components.<component_instance_name>]
name = <component_name>
path = <full_path_of_the_component_module>
enabled = $[hpx.components.load_external]   Property Description hpx.components..name This is the name of a component, usually the same as the second argument to the macro used while registering the component with HPX_REGISTER_COMPONENT. Set by the component factory. hpx.components..path This is either the full path file name of the component module or the directory the component module is located in. In this case, the component module name will be derived from the property hpx.components..name. Set by the component factory. hpx.components..enabled This setting explicitly enables or disables the component. This is an optional property, HPX assumed that the component is enabled if it is not defined. The value for <component_instance_name> is usually the same as for the corresponding name property. However generally it can be defined to any arbitrary instance name. It is used to distinguish between different ini sections, one for each component. #### The hpx.parcel configuration section¶ [hpx.parcel] address =${HPX_PARCEL_SERVER_ADDRESS:<hpx_initial_ip_address>}
port = ${HPX_PARCEL_SERVER_PORT:<hpx_initial_ip_port>} bootstrap =${HPX_PARCEL_BOOTSTRAP:<hpx_parcel_bootstrap>}
max_connections = ${HPX_PARCEL_MAX_CONNECTIONS:<hpx_parcel_max_connections>} max_connections_per_locality =${HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY:<hpx_parcel_max_connections_per_locality>}
max_message_size = ${HPX_PARCEL_MAX_MESSAGE_SIZE:<hpx_parcel_max_message_size>} max_outbound_message_size =${HPX_PARCEL_MAX_OUTBOUND_MESSAGE_SIZE:<hpx_parcel_max_outbound_message_size>}
array_optimization = ${HPX_PARCEL_ARRAY_OPTIMIZATION:1} zero_copy_optimization =${HPX_PARCEL_ZERO_COPY_OPTIMIZATION:$[hpx.parcel.array_optimization]} async_serialization =${HPX_PARCEL_ASYNC_SERIALIZATION:1}
message_handlers = ${HPX_PARCEL_MESSAGE_HANDLERS:0}   Property Description hpx.parcel.address This property defines the default IP address to be used for the parcel layer to listen to. This IP address will be used as long as no other values are specified (for instance using the --hpx:hpx command line option). The expected format is any valid IP address or domain name format which can be resolved into an IP address. The default depends on the compile time preprocessor constant HPX_INITIAL_IP_ADDRESS ("127.0.0.1"). hpx.parcel.port This property defines the default IP port to be used for the parcel layer to listen to. This IP port will be used as long as no other values are specified (for instance using the --hpx:hpx command line option). The default depends on the compile time preprocessor constant HPX_INITIAL_IP_PORT (7910). hpx.parcel.bootstrap This property defines which parcelport type should be used during application bootstrap. The default depends on the compile time preprocessor constant HPX_PARCEL_BOOTSTRAP ("tcp"). hpx.parcel.max_connections This property defines how many network connections between different localities are overall kept alive by each of locality. The default depends on the compile time preprocessor constant HPX_PARCEL_MAX_CONNECTIONS (512). hpx.parcel.max_connections_per_locality This property defines the maximum number of network connections that one locality will open to another locality. The default depends on the compile time preprocessor constant HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY (4). hpx.parcel.max_message_size This property defines the maximum allowed message size which will be transferrable through the parcel layer. The default depends on the compile time preprocessor constant HPX_PARCEL_MAX_MESSAGE_SIZE (1000000000 bytes). hpx.parcel.max_outbound_message_size This property defines the maximum allowed outbound coalesced message size which will be transferrable through the parcel layer. The default depends on the compile time preprocessor constant HPX_PARCEL_MAX_OUTBOUND_MESSAGE_SIZE (1000000 bytes). hpx.parcel.array_optimization This property defines whether this locality is allowed to utilize array optimizations during serialization of parcel data. The default is 1. hpx.parcel.zero_copy_optimization This property defines whether this locality is allowed to utilize zero copy optimizations during serialization of parcel data. The default is the same value as set for hpx.parcel.array_optimization. hpx.parcel.async_serialization This property defines whether this locality is allowed to spawn a new thread for serialization (this is both for encoding and decoding parcels). The default is 1. hpx.parcel.message_handlers This property defines whether message handlers are loaded. The default is 0. The following settings relate to the TCP/IP parcelport. [hpx.parcel.tcp] enable =${HPX_HAVE_PARCELPORT_TCP:$[hpx.parcel.enabled]} array_optimization =${HPX_PARCEL_TCP_ARRAY_OPTIMIZATION:$[hpx.parcel.array_optimization]} zero_copy_optimization =${HPX_PARCEL_TCP_ZERO_COPY_OPTIMIZATION:$[hpx.parcel.zero_copy_optimization]} async_serialization =${HPX_PARCEL_TCP_ASYNC_SERIALIZATION:$[hpx.parcel.async_serialization]} parcel_pool_size =${HPX_PARCEL_TCP_PARCEL_POOL_SIZE:$[hpx.threadpools.parcel_pool_size]} max_connections =${HPX_PARCEL_TCP_MAX_CONNECTIONS:$[hpx.parcel.max_connections]} max_connections_per_locality =${HPX_PARCEL_TCP_MAX_CONNECTIONS_PER_LOCALITY:$[hpx.parcel.max_connections_per_locality]} max_message_size =${HPX_PARCEL_TCP_MAX_MESSAGE_SIZE:$[hpx.parcel.max_message_size]} max_outbound_message_size =${HPX_PARCEL_TCP_MAX_OUTBOUND_MESSAGE_SIZE:$[hpx.parcel.max_outbound_message_size]}   Property Description hpx.parcel.tcp.enable Enable the use of the default TCP parcelport. Note that the initial bootstrap of the overall HPX application will be performed using the default TCP connections. This parcelport is enabled by default. This will be disabled only if MPI is enabled (see below). hpx.parcel.tcp.array_optimization This property defines whether this locality is allowed to utilize array optimizations in the TCP/IP parcelport during serialization of parcel data. The default is the same value as set for hpx.parcel.array_optimization. hpx.parcel.tcp.zero_copy_optimization This property defines whether this locality is allowed to utilize zero copy optimizations in the TCP/IP parcelport during serialization of parcel data. The default is the same value as set for hpx.parcel.zero_copy_optimization. hpx.parcel.tcp.async_serialization This property defines whether this locality is allowed to spawn a new thread for serialization in the TCP/IP parcelport (this is both for encoding and decoding parcels). The default is the same value as set for hpx.parcel.async_serialization. hpx.parcel.tcp.parcel_pool_size The value of this property defines the number of OS-threads created for the internal parcel thread pool of the TCP parcel port. The default is taken from hpx.threadpools.parcel_pool_size. hpx.parcel.tcp.max_connections This property defines how many network connections between different localities are overall kept alive by each of locality. The default is taken from hpx.parcel.max_connections. hpx.parcel.tcp.max_connections_per_locality This property defines the maximum number of network connections that one locality will open to another locality. The default is taken from hpx.parcel.max_connections_per_locality. hpx.parcel.tcp.max_message_size This property defines the maximum allowed message size which will be transferrable through the parcel layer. The default is taken from hpx.parcel.max_message_size. hpx.parcel.tcp.max_outbound_message_size This property defines the maximum allowed outbound coalesced message size which will be transferrable through the parcel layer. The default is taken from hpx.parcel.max_outbound_connections. The following settings relate to the MPI parcelport. These settings take effect only if the compile time constant HPX_HAVE_PARCELPORT_MPI is set (the equivalent cmake variable is HPX_WITH_PARCELPORT_MPI and has to be set to ON. [hpx.parcel.mpi] enable =${HPX_HAVE_PARCELPORT_MPI:$[hpx.parcel.enabled]} env =${HPX_HAVE_PARCELPORT_MPI_ENV:MV2_COMM_WORLD_RANK,PMI_RANK,OMPI_COMM_WORLD_SIZE,ALPS_APP_PE}
multithreaded = ${HPX_HAVE_PARCELPORT_MPI_MULTITHREADED:0} rank = <MPI_rank> processor_name = <MPI_processor_name> array_optimization =${HPX_HAVE_PARCEL_MPI_ARRAY_OPTIMIZATION:$[hpx.parcel.array_optimization]} zero_copy_optimization =${HPX_HAVE_PARCEL_MPI_ZERO_COPY_OPTIMIZATION:$[hpx.parcel.zero_copy_optimization]} use_io_pool =${HPX_HAVE_PARCEL_MPI_USE_IO_POOL:$1} async_serialization =${HPX_HAVE_PARCEL_MPI_ASYNC_SERIALIZATION:$[hpx.parcel.async_serialization]} parcel_pool_size =${HPX_HAVE_PARCEL_MPI_PARCEL_POOL_SIZE:$[hpx.threadpools.parcel_pool_size]} max_connections =${HPX_HAVE_PARCEL_MPI_MAX_CONNECTIONS:$[hpx.parcel.max_connections]} max_connections_per_locality =${HPX_HAVE_PARCEL_MPI_MAX_CONNECTIONS_PER_LOCALITY:$[hpx.parcel.max_connections_per_locality]} max_message_size =${HPX_HAVE_PARCEL_MPI_MAX_MESSAGE_SIZE:$[hpx.parcel.max_message_size]} max_outbound_message_size =${HPX_HAVE_PARCEL_MPI_MAX_OUTBOUND_MESSAGE_SIZE:$[hpx.parcel.max_outbound_message_size]}   Property Description hpx.parcel.mpi.enable Enable the use of the MPI parcelport. HPX tries to detect if the application was started within a parallel MPI environment. If the detection was succesful, the MPI parcelport is enabled by default. To explicitly disable the MPI parcelport, set to 0. Note that the initial bootstrap of the overall HPX application will be performed using MPI as well. hpx.parcel.mpi.env This property influences which environment variables (comma separated) will be analyzed to find out whether the application was invoked by MPI. hpx.parcel.mpi.multithreaded This property is used to determine what threading mode to use when initializing MPI. If this setting is 0 HPX will initialize MPI with MPI_THREAD_SINGLE if the value is not equal to 0 HPX will initialize MPI with MPI_THREAD_MULTI. hpx.parcel.mpi.rank This property will be initialized to the MPI rank of the locality. hpx.parcel.mpi.processor_name This property will be initialized to the MPI processor name of the locality. hpx.parcel.mpi.array_optimization This property defines whether this locality is allowed to utilize array optimizations in the MPI parcelport during serialization of parcel data. The default is the same value as set for hpx.parcel.array_optimization. hpx.parcel.mpi.zero_copy_optimization This property defines whether this locality is allowed to utilize zero copy optimizations in the MPI parcelport during serialization of parcel data. The default is the same value as set for hpx.parcel.zero_copy_optimization. hpx.parcel.mpi.use_io_pool This property can be set to run the progress thread inside of HPX threads instead of a separate thread pool. The default is 1. hpx.parcel.mpi.async_serialization This property defines whether this locality is allowed to spawn a new thread for serialization in the MPI parcelport (this is both for encoding and decoding parcels). The default is the same value as set for hpx.parcel.async_serialization. hpx.parcel.mpi.parcel_pool_size The value of this property defines the number of OS-threads created for the internal parcel thread pool of the MPI parcel port. The default is taken from hpx.threadpools.parcel_pool_size. hpx.parcel.mpi.max_connections This property defines how many network connections between different localities are overall kept alive by each of locality. The default is taken from hpx.parcel.max_connections. hpx.parcel.mpi.max_connections_per_locality This property defines the maximum number of network connections that one locality will open to another locality. The default is taken from hpx.parcel.max_connections_per_locality. hpx.parcel.mpi.max_message_size This property defines the maximum allowed message size which will be transferrable through the parcel layer. The default is taken from hpx.parcel.max_message_size. hpx.parcel.mpi.max_outbound_message_size This property defines the maximum allowed outbound coalesced message size which will be transferrable through the parcel layer. The default is taken from hpx.parcel.max_outbound_connections. #### The hpx.agas configuration section¶ [hpx.agas] address =${HPX_AGAS_SERVER_ADDRESS:<hpx_initial_ip_address>}
port = ${HPX_AGAS_SERVER_PORT:<hpx_initial_ip_port>} service_mode = hosted dedicated_server = 0 max_pending_refcnt_requests =${HPX_AGAS_MAX_PENDING_REFCNT_REQUESTS:<hpx_initial_agas_max_pending_refcnt_requests>}
use_caching = ${HPX_AGAS_USE_CACHING:1} use_range_caching =${HPX_AGAS_USE_RANGE_CACHING:1}
enabled = $[hpx.components.load_external] default = 1  The values in this section correspond to the expected configuration information for a component as described in the section Built-in Default Configuration Settings. In order to locate component shared libraries, HPX will try loading all shared libraries (files with the platform specific extension of a shared library, Linux: *.so, Windows: *.dll, MacOS: *.dylib found in the directory referenced by the ini property hpx.component_path). This first step corresponds to step 1) during the process of filling the internal configuration database with default information as described in section Loading INI files. After all of the configuration information has been loaded, HPX performs the second step in terms of loading components. During this step, HPX scans all existing configuration sections [hpx.component.<some_component_instance_name>] and instantiates a special factory object for each of the successfully located and loaded components. During the application’s life time, these factory objects will be responsible to create new and discard old instances of the component they are associated with. This step is performed after step 11) of the process of filling the internal configuration database with default information as described in section Loading INI files. ### Application specific component example¶ In this section we assume to have a simple application component which exposes one member function as a component action. The header file app_server.hpp declares the C++ type to be exposed as a component. This type has a member function print_greeting() which is exposed as an action print_greeting_action. We assume the source files for this example are located in a directory referenced by $APP_ROOT:

// file: $APP_ROOT/app_server.hpp #include <hpx/hpx.hpp> #include <hpx/include/iostreams.hpp> namespace app { // Define a simple component exposing one action 'print_greeting' class HPX_COMPONENT_EXPORT server : public hpx::components::component_base<server> { void print_greeting () { hpx::cout << "Hey, how are you?\n" << hpx::flush; } // Component actions need to be declared, this also defines the // type 'print_greeting_action' representing the action. HPX_DEFINE_COMPONENT_ACTION(server, print_greeting, print_greeting_action); }; } // Declare boilerplate code required for each of the component actions. HPX_REGISTER_ACTION_DECLARATION(app::server::print_greeting_action);  The corresponding source file contains mainly macro invocations which define boilerplate code needed for HPX to function properly: // file:$APP_ROOT/app_server.cpp
#include "app_server.hpp"

// Define boilerplate required once per component module.
HPX_REGISTER_COMPONENT_MODULE();

// Define factory object associated with our component of type 'app::server'.
HPX_REGISTER_COMPONENT(app::server, app_server);

// Define boilerplate code required for each of the component actions. Use the
// same argument as used for HPX_REGISTER_ACTION_DECLARATION above.
HPX_REGISTER_ACTION(app::server::print_greeting_action);


The following gives an example of how the component can be used. We create one instance of the app::server component on the current locality and invoke the exposed action print_greeting_action using the global id of the newly created instance. Note, that no special code is required to delete the component instance after it is not needed anymore. It will be deleted automatically when its last reference goes out of scope, here at the closing brace of the block surrounding the code:

// file: $APP_ROOT/use_app_server_example.cpp #include <hpx/hpx_init.hpp> #include "app_server.hpp" int hpx_main() { { // Create an instance of the app_server component on the current locality. hpx::naming:id_type app_server_instance = hpx::create_component<app::server>(hpx::find_here()); // Create an instance of the action 'print_greeting_action'. app::server::print_greeting_action print_greeting; // Invoke the action 'print_greeting' on the newly created component. print_greeting(app_server_instance); } return hpx::finalize(); } int main(int argc, char* argv[]) { return hpx::init(argc, argv); }  In order to make sure that the application will be able to use the component app::server, special configuration information must be passed to HPX. The simples way to allow HPX to ‘find’ the component is to provide special ini configuration files, which add the necessary information to the internal configuration database. The component should have a special ini file containing the information specific to the component app_server. # file:$APP_ROOT/app_server.ini
[hpx.components.app_server]
name = app_server
path = $APP_LOCATION/  Here $APP_LOCATION is the directory where the (binary) component shared library is located. HPX will attempt to load the shared library from there. The section name hpx.components.app_server reflects the instance name of the component (app_server is an arbitrary, but unique name). The property value for hpx.components.app_server.name should be the same as used for the second argument to the macro HPX_REGISTER_COMPONENT above.

Additionally a file .hpx.ini which could be located in the current working directory (see step 3 as described in the section Loading INI files) can be used to add to the ini search path for components:

# file: $PWD/.hpx.ini [hpx] ini_path =$[hpx.ini_path]:$APP_ROOT/  This assumes that the above ini file specific to the component is located in the directory $APP_ROOT.

Note

It is possible to reference the defined property from inside its value. HPX will gracefully use the previous value of hpx.ini_path for the reference on the right hand side and assign the overall (now expanded) value to the property.

## Logging¶

HPX uses a sophisticated logging framework allowing to follow in detail what operations have been performed inside the HPX library in what sequence. This information proves to be very useful for diagnosing problems or just for improving the understanding what is happening in HPX as a consequence of invoking HPX API functionality.

### Default logging¶

Enabling default logging is a simple process. The detailed description in the remainder of this section explains different ways to customize the defaults. Default logging can be enabled by using one of the following:

• a command line switch --hpx:debug-hpx-log, which will enable logging to the console terminal

• the command line switch --hpx:debug-hpx-log=<filename>, which enables logging to a given file <filename>, or

• setting an environment variable HPX_LOGLEVEL=<loglevel> while running the HPX application. In this case <loglevel> should be a number between (or equal to) 1 and 5 where 1 means minimal logging and 5 causes to log all available messages. When setting the environment variable the logs will be written to a file named hpx.<PID>.lo in the current working directory, where <PID> is the process id of the console instance of the application.

### Customizing logging¶

Generally, logging can be customized either using environment variable settings or using by an ini configuration file. Logging is generated in several categories, each of which can be customized independently. All customizable configuration parameters have reasonable defaults, allowing to use logging without any additional configuration effort. The following table lists the available categories.

 Category Category shortcut Information to be generated Environment variable General None Logging information generated by different subsystems of HPX, such as thread-manager, parcel layer, LCOs, etc. HPX_LOGLEVEL AGAS AGAS Logging output generated by the AGAS subsystem HPX_AGAS_LOGLEVEL Application APP Logging generated by applications. HPX_APP_LOGLEVEL

By default, all logging output is redirected to the console instance of an application, where it is collected and written to a file, one file for each logging category.

Each logging category can be customized at two levels, the parameters for each are stored in the ini configuration sections hpx.logging.CATEGORY and hpx.logging.console.CATEGORY (where CATEGORY is the category shortcut as listed in the table above). The former influences logging at the source locality and the latter modifies the logging behaviour for each of the categories at the console instance of an application.

### Levels¶

All HPX logging output has seven different logging levels. These levels can be set explicitly or through environmental variables in the main HPX ini file as shown below. The logging levels and their associated integral values are shown in the table below, ordered from most verbose to least verbose. By default, all HPX logs are set to 0, e.g. all logging output is disabled by default.

Table 19 Logging levels

Logging level

Integral value

<debug>

5

<info>

4

<warning>

3

<error>

2

<fatal>

1

No logging

0

Tip

The easiest way to enable logging output is to set the environment variable corresponding to the logging category to an integral value as described in the table above. For instance, setting HPX_LOGLEVEL=5 will enable full logging output for the general category. Please note that the syntax and means of setting environment variables varies between operating systems.

### Configuration¶

Logs will be saved to destinations as configured by the user. By default, logging output is saved on the console instance of an application to hpx.<CATEGORY>.<PID>.lo (where CATEGORY and PID> are placeholders for the category shortcut and the OS process id). The output for the general logging category is saved to hpx.<PID>.log. The default settings for the general logging category are shown here (the syntax is described in the section The HPX INI File Format):

[hpx.logging]
level = ${HPX_LOGLEVEL:0} destination =${HPX_LOGDESTINATION:console}
format = ${HPX_LOGFORMAT:(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%) P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%]|\\n}  The logging level is taken from the environment variable HPX_LOGLEVEL and defaults to zero, e.g. no logging. The default logging destination is read from the environment variable HPX_LOGDESTINATION On any of the localities it defaults to console which redirects all generated logging output to the console instance of an application. The following table lists the possible destinations for any logging output. It is possible to specify more than one destination separated by whitespace.  Logging destination Description file() Direct all output to a file with the given . cout Direct all output to the local standard output of the application instance on this locality. cerr Direct all output to the local standard error output of the application instance on this locality. console Direct all output to the console instance of the application. The console instance has its logging destinations configured separately. android_log Direct all output to the (Android) system log (available on Android systems only). The logging format is read from the environment variable HPX_LOGFORMAT and it defaults to a complex format description. This format consists of several placeholder fields (for instance %locality% which will be replaced by concrete values when the logging output is generated. All other information is transferred verbatim to the output. The table below describes the available field placeholders. The separator character | separates the logging message prefix formatted as shown and the actual log message which will replace the separator.  Name Description locality The id of the locality on which the logging message was generated. hpxthread The id of the HPX-thread generating this logging output. hpxphase The phase 1 of the HPX-thread generating this logging output. hpxcomponent The local virtual address of the component which the current HPX-thread is accessing. parentloc The id of the locality where the HPX thread was running which initiated the current HPX-thread. The current HPX-thread is generating this logging output. hpxparent The id of the HPX-thread which initiated the current HPX-thread. The current HPX-thread is generating this logging output. hpxparentphase The phase of the HPX-thread when it initiated the current HPX-thread. The current HPX-thread is generating this logging output. time The time stamp for this logging outputline as generated by the source locality. idx The sequence number of the logging output line as generated on the source locality. osthread The sequence number of the OS-thread which executes the current HPX-thread. Note Not all of the field placeholder may be expanded for all generated logging output. If no value is available for a particular field it is replaced with a sequence of '-' characters.] Here is an example line from a logging output generated by one of the HPX examples (please note that this is generated on a single line, without line break): (T00000000/0000000002d46f90.01/00000000009ebc10) P--------/0000000002d46f80.02 17:49.37.320 [000000000000004d] <info> [RT] successfully created component {0000000100ff0001, 0000000000030002} of type: component_barrier[7(3)]  The default settings for the general logging category on the console is shown here: [hpx.logging.console] level =${HPX_LOGLEVEL:$[hpx.logging.level]} destination =${HPX_CONSOLE_LOGDESTINATION:file(hpx.$[system.pid].log)} format =${HPX_CONSOLE_LOGFORMAT:|}


These settings define how the logging is customized once the logging output is received by the console instance of an application. The logging level is read from the environment variable HPX_LOGLEVEL (as set for the console instance of the application). The level defaults to the same values as the corresponding settings in the general logging configuration shown before. The destination on the console instance is set to be a file which name is generated based from its OS process id. Setting the environment variable HPX_CONSOLE_LOGDESTINATION allows customization of the naming scheme for the output file. The logging format is set to leave the original logging output unchanged, as received from one of the localities the application runs on.

## HPX Command Line Options¶

The predefined command line options for any application using hpx::init are described in the following subsections.

### HPX options (allowed on command line only)¶

--hpx:help

print out program usage (default: this message), possible values: full (additionally prints options from components)

--hpx:version

print out HPX version and copyright information

--hpx:info

print out HPX configuration information

--hpx:options-file arg

specify a file containing command line options (alternatively: @filepath)

### HPX options (additionally allowed in an options file)¶

--hpx:worker

run this instance in worker mode

--hpx:console

run this instance in console mode

--hpx:connect

run this instance in worker mode, but connecting late

--hpx:run-agas-server

run AGAS server as part of this runtime instance

--hpx:run-hpx-main

run the hpx_main function, regardless of locality mode

--hpx:hpx arg

the IP address the HPX parcelport is listening on, expected format: address:port (default: 127.0.0.1:7910)

--hpx:agas arg

the IP address the AGAS root server is running on, expected format: address:port (default: 127.0.0.1:7910)

--hpx:run-agas-server-only

run only the AGAS server

--hpx:nodefile arg

the file name of a node file to use (list of nodes, one node name per line and core)

--hpx:nodes arg

the (space separated) list of the nodes to use (usually this is extracted from a node file)

--hpx:endnodes

this can be used to end the list of nodes specified using the option --hpx:nodes

--hpx:ifsuffix arg

suffix to append to host names in order to resolve them to the proper network interconnect

--hpx:ifprefix arg

prefix to prepend to host names in order to resolve them to the proper network interconnect

--hpx:iftransform arg

sed-style search and replace (s/search/replace/) used to transform host names to the proper network interconnect

--hpx:localities arg

the number of localities to wait for at application startup (default: 1)

--hpx:node arg

number of the node this locality is run on (must be unique)

--hpx:ignore-batch-env

ignore batch environment variables

--hpx:expect-connecting-localities

this locality expects other localities to dynamically connect (this is implied if the number of initial localities is larger than 1)

--hpx:pu-offset

the first processing unit this instance of HPX should be run on (default: 0)

--hpx:pu-step

the step between used processing unit numbers for this instance of HPX (default: 1)

--hpx:threads arg

the number of operating system threads to spawn for this HPX locality. Possible values are: numeric values 1, 2, 3 and so on, all (which spawns one thread per processing unit, includes hyperthreads), or cores (which spawns one thread per core) (default: cores).

--hpx:cores arg

the number of cores to utilize for this HPX locality (default: all, i.e. the number of cores is based on the number of threads --hpx:threads assuming --hpx:bind=compact

--hpx:affinity arg

the affinity domain the OS threads will be confined to, possible values: pu, core, numa, machine (default: pu)

--hpx:bind arg

the detailed affinity description for the OS threads, see More details about HPX command line options for a detailed description of possible values. Do not use with --hpx:pu-step, --hpx:pu-offset or --hpx:affinity options. Implies --hpx:numa-sensitive (--hpx:bind=none) disables defining thread affinities).

--hpx:use-process-mask

use the process mask to restrict available hardware resources (implies --hpx:ignore-batch-env)

--hpx:print-bind

print to the console the bit masks calculated from the arguments specified to all --hpx:bind options.

--hpx:queuing arg

the queue scheduling policy to use, options are local, local-priority-fifo, local-priority-lifo, static, static-priority, abp-priority-fifo and abp-priority-lifo (default: local-priority-fifo)

--hpx:high-priority-threads arg

the number of operating system threads maintaining a high priority queue (default: number of OS threads), valid for --hpx:queuing=abp-priority, --hpx:queuing=static-priority and --hpx:queuing=local-priority only

--hpx:numa-sensitive

makes the scheduler NUMA sensitive

### HPX configuraton options¶

--hpx:app-config arg

load the specified application configuration (ini) file

--hpx:config arg

load the specified hpx configuration (ini) file

--hpx:ini arg

add a configuration definition to the default runtime configuration

--hpx:exit

exit after configuring the runtime

### HPX debugging options¶

--hpx:list-symbolic-names

list all registered symbolic names after startup

--hpx:list-component-types

list all dynamic component types after startup

--hpx:dump-config-initial

print the initial runtime configuration

--hpx:dump-config

print the final runtime configuration

--hpx:debug-hpx-log [arg]

enable all messages on the HPX log channel and send all HPX logs to the target destination (default: cout)

--hpx:debug-agas-log [arg]

enable all messages on the AGAS log channel and send all AGAS logs to the target destination (default: cout)

--hpx:debug-parcel-log [arg]

enable all messages on the parcel transport log channel and send all parcel transport logs to the target destination (default: cout)

--hpx:debug-timing-log [arg]

enable all messages on the timing log channel and send all timing logs to the target destination (default: cout)

--hpx:debug-app-log [arg]

enable all messages on the application log channel and send all application logs to the target destination (default: cout)

--hpx:debug-clp

debug command line processing

--hpx:attach-debugger arg

wait for a debugger to be attached, possible arg values: startup or exception (default: startup)

### Command line argument shortcuts¶

Additionally, the following shortcuts are available from every HPX application.

Table 22 Predefined command line option shortcuts

Shortcut option

Equivalent long option

-a

--hpx:agas

-c

--hpx:console

-h

--hpx:help

-I

--hpx:ini

-l

--hpx:localities

-p

--hpx:app-config

-q

--hpx:queuing

-r

--hpx:run-agas-server

-t

--hpx:threads

-v

--hpx:version

-w

--hpx:worker

-x

--hpx:hpx

-0

--hpx:node=0

-1

--hpx:node=1

-2

--hpx:node=2

-3

--hpx:node=3

-4

--hpx:node=4

-5

--hpx:node=5

-6

--hpx:node=6

-7

--hpx:node=7

-8

--hpx:node=8

-9

--hpx:node=9

It is possible to define your own shortcut options. In fact, all of the shortcuts listed above are pre-defined using the technique described here. Also, it is possible to redefine any of the pre-defined shortcuts to expand differently as well.

Shortcut options are obtained from the internal configuration database. They are stored as key-value properties in a special properties section named hpx.commandline. You can define your own shortcuts by adding the corresponding definitions to one of the ini configuration files as described in the section Configuring HPX applications. For instance, in order to define a command line shortcut --p which should expand to -hpx:print-counter, the following configuration information needs to be added to one of the ini configuration files:

[hpx.commandline.aliases]
--pc = --hpx:print-counter


Note

Any arguments for shortcut options passed on the command line are retained and passed as arguments to the corresponding expanded option. For instance, given the definition above, the command line option:

--pc=/threads{locality#0/total}/count/cumulative


would be expanded to:

--hpx:print-counter=/threads{locality#0/total}/count/cumulative


Important

Any shortcut option should either start with a single '-' or with two '--' characters. Shortcuts starting with a single '-' are interpreted as short options (i.e. everything after the first character following the '-' is treated as the argument). Shortcuts starting with '--' are interpreted as long options. No other shortcut formats are supported.

### Specifying options for single localities only¶

For runs involving more than one locality it is sometimes desirable to supply specific command line options to single localities only. When the HPX application is launched using a scheduler (like PBS, for more details see section How to use HPX applications with PBS), specifying dedicated command line options for single localities may be desirable. For this reason all of the command line options which have the general format --hpx:<some_key> can be used in a more general form: --hpx:<N>:<some_key>, where <N> is the number of the locality this command line options will be applied to, all other localities will simply ignore the option. For instance, the following PBS script passes the option --hpx:pu-offset=4 to the locality '1' only.

#!/bin/bash
#
#PBS -l nodes=2:ppn=4

APP_PATH=~/packages/hpx/bin/hello_world_distributed
APP_OPTIONS=

pbsdsh -u $APP_PATH$APP_OPTIONS --hpx:1:pu-offset=4 --hpx:nodes=cat $PBS_NODEFILE  Caution If the first application specific argument (inside $APP_OPTIONS is a non-option (i.e. does not start with a - or a --, then it must be placed before the option --hpx:nodes, which, in this case, should be the last option on the command line.

Alternatively, use the option --hpx:endnodes to explicitly mark the end of the list of node names:

pbsdsh -u $APP_PATH --hpx:1:pu-offset=4 --hpx:nodes=cat$PBS_NODEFILE --hpx:endnodes \$APP_OPTIONS


### More details about HPX command line options¶

This section documents the following list of the command line options in more detail:

#### The command line option --hpx:bind¶

This command line option allows one to specify the required affinity of the HPX worker threads to the underlying processing units. As a result the worker threads will run only on the processing units identified by the corresponding bind specification. The affinity settings are to be specified using --hpx:bind=<BINDINGS>, where <BINDINGS> have to be formatted as described below.

In addition to the syntax described below one can use --hpx:bind=none to disable all binding of any threads to a particular core. This is mostly supported for debugging purposes.

The specified affinities refer to specific regions within a machine hardware topology. In order to understand the hardware topology of a particular machine it may be useful to run the lstopo tool which is part of Portable Hardware Locality (HWLOC) to see the reported topology tree. Seeing and understanding a topology tree will definitely help in understanding the concepts that are discussed below.

Affinities can be specified using HWLOC (Portable Hardware Locality (HWLOC)) tuples. Tuples of HWLOC objects and associated indexes can be specified in the form object:index, object:index-index or object:index,...,index. HWLOC objects represent types of mapped items in a topology tree. Possible values for objects are socket, numanode, core and pu (processing unit). Indexes are non-negative integers that specify a unique physical object in a topology tree using its logical sequence number.

Chaining multiple tuples together in the more general form object1:index1[.object2:index2[...]] is permissible. While the first tuple’s object may appear anywhere in the topology, the Nth tuple’s object must have a shallower topology depth than the (N+1)th tuple’s object. Put simply: as you move right in a tuple chain, objects must go deeper in the topology tree. Indexes specified in chained tuples are relative to the scope of the parent object. For example, socket:0.core:1 refers to the second core in the first socket (all indices are zero based).

Multiple affinities can be specified using several --hpx:bind command line options or by appending several affinities separated by a ';' By default, if multiple affinities are specified, they are added.

"all" is a special affinity consisting in the entire current topology.

Note

All ‘names’ in an affinity specification, such as thread, socket, numanode, pu or all can be abbreviated. Thus the affinity specification threads:0-3=socket:0.core:1.pu:1 is fully equivalent to its shortened form t:0-3=s:0.c:1.p:1.

Here is a full grammar describing the possible format of mappings:

mappings     ::=  distribution | mapping (";" mapping)*
distribution ::=  "compact" | "scatter" | "balanced" | "numa-balanced"
mapping      ::=  thread_spec "=" pu_specs
thread_spec  ::=  "thread:" range_specs
pu_specs     ::=  pu_spec ("." pu_spec)*
pu_spec      ::=  type ":" range_specs | "~" pu_spec
range_specs  ::=  range_spec ("," range_spec)*
range_spec   ::=  int | int "-" int | "all"
type         ::=  "socket" | "numanode" | "core" | "pu"


The following example assumes a system with at least 4 cores, where each core has more than 1 processing unit (hardware threads). Running hello_world_distributed with 4 OS-threads (on 4 processing units), where each of those threads is bound to the first processing unit of each of the cores, can be achieved by invoking:

hello_world_distributed -t4 --hpx:bind=thread:0-3=core:0-3.pu:0


Here thread:0-3 specifies the OS threads for which to define affinity bindings, and core:0-3.pu: defines that for each of the cores (core:0-3) only their first processing unit pu:0 should be used.

Note

The command line option --hpx:print-bind can be used to print the bitmasks generated from the affinity mappings as specified with --hpx:bind. For instance, on a system with hyperthreading enabled (i.e. 2 processing units per core), the command line:

hello_world_distributed -t4 --hpx:bind=thread:0-3=core:0-3.pu:0 --hpx:print-bind


will cause this output to be printed:

0: PU L#0(P#0), Core L#0, Socket L#0, Node L#0(P#0)
1: PU L#2(P#2), Core L#1, Socket L#0, Node L#0(P#0)
2: PU L#4(P#4), Core L#2, Socket L#0, Node L#0(P#0)
3: PU L#6(P#6), Core L#3, Socket L#0, Node L#0(P#0)


where each bit in the bitmasks corresponds to a processing unit the listed worker thread will be bound to run on.

The difference between the four possible predefined distribution schemes (compact, scatter, balanced and numa-balanced) is best explained with an example. Imagine that we have a system with 4 cores and 4 hardware threads per core on 2 sockets. If we place 8 threads the assignments produced by the compact, scatter, balanced and numa-balanced types are shown in the figure below. Notice that compact does not fully utilize all the cores in the system. For this reason it is recommended that applications are run using the scatter or balanced/numa-balanced options in most cases.

Fig. 7 Schematic of thread affinity type distributions.

In addition to the predefined distributions it is possible to restrict the resources used by HPX to the process CPU mask. The CPU mask is typically set by e.g. MPI and batch environments. Using the command line option --hpx:use-process-mask makes HPX act as if only the processing units in the CPU mask are available for use by HPX. The number of threads is automatically determined from the CPU mask. The number of threads can still be changed manually using this option, but only to a number less than or equal to the number of processing units in the CPU mask. The option --hpx:print-bind is useful in conjunction with --hpx:use-process-mask to make sure threads are placed as expected.

1

The phase of a HPX-thread counts how often this thread has been activated.