python3-pathos Subpackage of python-pathos

The pathos package provides a few basic tools to make parallel and distributed computing more accessible to the end user. The goal of pathos is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. pathos provides methods for configuring, launching, monitoring, and controlling a service on a remote host. One of the most basic features of pathos is the ability to configure and launch a RPC-based service on a remote host. pathos seeds the remote host with the portpicker script, which allows the remote host to inform the localhost of a port that is available for communication. Beyond the ability to establish a RPC service, and then post requests, is the ability to launch code in parallel. Unlike parallel computing performed at the node level (typically with MPI), pathos enables the user to launch jobs in parallel across heterogeneous distributed resources. pathos provides distributed map and pipe algorithms, where a mix of local processors and distributed workers can be selected. pathos also provides a very basic automated load balancing service, as well as the ability for the user to directly select the resources. The high-level pool.map interface, yields a map implementation that hides the RPC internals from the user. With pool.map, the user can launch their code in parallel, and as a distributed service, using standard python and without writing a line of server or parallel batch code. RPC servers and communication in general is known to be insecure. However, instead of attempting to make the RPC communication itself secure, pathos provides the ability to automatically wrap any distributed service or communication in a ssh-tunnel. Ssh is a universally trusted method. Using ssh-tunnels, pathos has launched several distributed calculations on national lab clusters, and to date has performed test calculations that utilize node-to-node communication between several national lab clusters and a user's laptop. pathos allows the user to configure and launch at a very atomic level, through raw access to ssh and scp. pathos is the core of a python framework for heterogeneous computing. pathos is in active development, so any user feedback, bug reports, comments, or suggestions are highly appreciated. A list of issues is located at https://github.com/uqfoundation/pathos/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query.

Releases Overview