Introduction

Welcome to ChoRus! This website is a guide to using the ChoRus library.

What is ChoRus?

ChoRus is a library that enables Choreographic Programming in Rust.

In distributed programming, it is necessary to coordinate the behavior of multiple nodes. This coordination is typically achieved by writing a program that runs on each node. However, writing a program for each node can lead to bugs and inconsistencies.

Choreographic Programming is a programming paradigm that allows programmers to write "choreographies" that describe the desired behavior of a system as a whole. These choreographies can then be used to generate programs for each node in the system though a process called "end-point projection," or "EPP" for short.

Choreographic Programming as a Library

In the past, choreographic programming has been implemented as a standalone programming language. While this approach allows for flexibility for language designs, it makes it difficult to integrate choreographic programming into existing ecosystems.

ChoRus takes a different approach. Instead of implementing choreographic programming as a language, ChoRus implements choreographic programming as a library. ChoRus can be installed as a Cargo package and used in any Rust project. This allows choreographic programming to be used in any Rust project, including projects that use other libraries.

ChoRus is built on top of the "End-point Projection as Dependency Injection" (EPP-as-DI) approach. ChoRus also takes advantage of Rust's type system and macro system to provide a safe and ergonomic choreographic programming experience.

Features

At high level, ChoRus provides the following features:

• Define choreographies by implementing the Choreography trait.
  • Passing located arguments to / receiving located return values from choreographies.
  • Location polymorphism.
  • Higher-order choreographies.
  • Efficient conditional with choreographic enclaves.
• Performing end-point projection.
• Pluggable message transports.
  • Two built-in transports: Local and HTTP.
  • Creating custom transports.
• Macros for defining locations and choreographies.

Getting Started

Installation

ChoRus is still under active development. We are still expecting to make breaking changes to the API.

# create a binary crate
cargo new chorus_hello_world
cd chorus_hello_world
# install ChoRus as a dependency
cargo add chorus_lib

Running Hello World example

Once you have installed ChoRus, you can run the Hello World example by copy-pasting the following code into main.rs:

extern crate chorus_lib;

use std::thread;

use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, LocationSet, Projector};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};

// --- Define two locations (Alice and Bob) ---

#[derive(ChoreographyLocation)]
struct Alice;

#[derive(ChoreographyLocation)]
struct Bob;

// --- Define a choreography ---
struct HelloWorldChoreography;

// Implement the `Choreography` trait for `HelloWorldChoreography`
impl Choreography for HelloWorldChoreography {
    // Define the set of locations involved in the choreography.
    // In this case, the set consists of `Alice` and `Bob` and
    // the choreography can use theses locations.
    type L = LocationSet!(Alice, Bob);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        // Create a located value at Alice
        let msg_at_alice = op.locally(Alice, |_| {
            println!("Hello from Alice!");
            "Hello from Alice!".to_string()
        });
        // Send the located value to Bob
        let msg_at_bob = op.comm(Alice, Bob, &msg_at_alice);
        // Print the received message at Bob
        op.locally(Bob, |un| {
            let msg = un.unwrap(&msg_at_bob);
            println!("Bob received a message: {}", msg);
            msg
        });
    }
}

fn main() {
    let mut handles: Vec<thread::JoinHandle<()>> = Vec::new();
    // Create a transport channel
    let transport_channel = LocalTransportChannelBuilder::new()
        .with(Alice)
        .with(Bob)
        .build();
    // Run the choreography in two threads
    {
        let transport = LocalTransport::new(Alice, transport_channel.clone());
        handles.push(thread::spawn(move || {
            let p = Projector::new(Alice, transport);
            p.epp_and_run(HelloWorldChoreography);
        }));
    }
    {
        let transport = LocalTransport::new(Bob, transport_channel.clone());
        handles.push(thread::spawn(move || {
            let p = Projector::new(Bob, transport);
            p.epp_and_run(HelloWorldChoreography);
        }));
    }
    for h in handles {
        h.join().unwrap();
    }
}

Guide

This section will guide you through the concepts and features of ChoRus.

Locations

Before we can start writing choreographies, we need to define locations. A location is a place where a choreography can be executed. A location can be a physical location, such as a computer, or a logical location, such as a thread.

To define a location, we need to create a struct and derive the ChoreographyLocation trait.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::ChoreographyLocation;

#[derive(ChoreographyLocation)]
struct Alice;

#[derive(ChoreographyLocation)]
struct Bob;
}

The ChoreographyLocation trait provides the name method, which returns the name of the location as a &'static str. The name of a location is used to identify the location when performing end-point projection.

use chorus_lib::core::ChoreographyLocation;

#[derive(ChoreographyLocation)]
struct Alice;

#[derive(ChoreographyLocation)]
struct Bob;

let name = Alice::name();
assert_eq!(name, "Alice");

Location Set

A LocationSet is a special type representing a set of ChoreographyLocation types. It's used to ensure type safety within the system, and you'll see its application in future sections. To build a LocationSet type, you can use the LocationSet macro from the chorus_lib crate.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::ChoreographyLocation;
#[derive(ChoreographyLocation)]
struct Alice;

#[derive(ChoreographyLocation)]
struct Bob;
use chorus_lib::core::LocationSet;

type L = LocationSet!(Alice, Bob);
}

Some operators, such as multicast, requires a value of LocationSet to be passed as an argument. You can obtain a value of LocationSet by using the new method.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::ChoreographyLocation;
#[derive(ChoreographyLocation)]
struct Alice;

#[derive(ChoreographyLocation)]
struct Bob;
use chorus_lib::core::LocationSet;

let alice_and_bob = <LocationSet!(Alice, Bob)>::new();
}

Choreography

Choreography is a program that describes the behavior of a distributed system as a whole. To define a choreography, create a struct and implement the Choreography trait.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
// 1. Define a struct
struct HelloWorldChoreography;

// 2. Implement the `Choreography` trait
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        // 3. Use the `op` parameter to access operators
        op.locally(Alice, |_| {
            println!("Hello, World!");
        });
    }
}
}

Choreography must implement the run method which defines the behavior of the system. The run method takes a reference to an object that implements the ChoreoOp trait. The ChoreoOp trait provides choreographic operators such as locally and comm.

Also, each Choreography has an associated LocationSet type, L; this is the LocationSet that the Choreography can operate on.

Choreographic Operators

Inside the run method, you can use the op parameter to access choreographic operators.

locally

The locally operator is used to perform a computation at a single location. It takes two parameters: a location and a closure. The closure is executed only at the specified location.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

struct HelloWorldChoreography;
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
op.locally(Alice, |_| {
    println!("Hello, World!");
});
    }
}
}

The closure can return a value to create a located value. Located values are values that are only available at a single location. When the computation closure returns a located value, the locally operator returns a located value at the same location.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

struct HelloWorldChoreography;
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
// This value is only available at Alice
let num_at_alice: Located<i32, Alice> = op.locally(Alice, |_| {
    42
});
    }
}
}

The computation closure takes Unwrapper. Using the Unwrapper, you can get a reference out of a located value.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

struct HelloWorldChoreography;
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
let num_at_alice: Located<i32, Alice> = op.locally(Alice, |_| {
    42
});
op.locally(Alice, |un| {
    let num: &i32 = un.unwrap(&num_at_alice);
    println!("The number at Alice is {}", num);
    assert_eq!(*num, 42);
});
    }
}
}

Note that you can unwrap a located value only at the location where the located value is available. If you try to unwrap a located value at a different location, the program will fail to compile.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

struct HelloWorldChoreography;
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice, Bob);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
// This code will fail to compile
let num_at_alice = op.locally(Alice, |_| { 42 });
op.locally(Bob, |un| {
    // Only values located at Bob can be unwrapped here
    let num_at_alice: &i32 = un.unwrap(&num_at_alice);
});
    }
}
}

We will discuss located values in more detail in the Located Values section.

comm

The comm operator is used to perform a communication between two locations. It takes three parameters: a source location, a destination location, and a located value at the source location. The located value is sent from the source location to the destination location, and the operator returns a located value at the destination location.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

struct HelloWorldChoreography;
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice, Bob);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
// This value is only available at Alice
let num_at_alice: Located<i32, Alice> = op.locally(Alice, |_| {
    42
});
// Send the value from Alice to Bob
let num_at_bob: Located<i32, Bob> = op.comm(Alice, Bob, &num_at_alice);
// Bob can now access the value
op.locally(Bob, |un| {
    let num_at_bob: &i32 = un.unwrap(&num_at_bob);
    println!("The number at Bob is {}", num_at_bob);
});
    }
}
}

broadcast

The broadcast operator is used to perform a broadcast from a single location to multiple locations. It takes two parameters: a source location and a located value at the source location. The located value is sent from the source location to all other locations, and the operator returns a normal value.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

struct HelloWorldChoreography;
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
// This value is only available at Alice
let num_at_alice: Located<i32, Alice> = op.locally(Alice, |_| {
    42
});
// Broadcast the value from Alice to all other locations
let num: i32 = op.broadcast(Alice, num_at_alice);
    }
}
}

Because all locations receive the value, the return type of the broadcast operator is a normal value, not a located value. This means that the value can be used for control flow.

if num == 42 {
    println!("The number is 42!");
} else {
    println!("The number is not 42!");
}

Multicast

The multicast operator is similar to the broadcast operator, but it allows you to manually specify the recipients instead of sending the value to all locations. The operator returns a MultiplyLocated value that is available at the all recipient locations.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

struct HelloWorldChoreography;
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice, Bob, Carol);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
// This value is only available at Alice
let num_at_alice: Located<i32, Alice> = op.locally(Alice, |_| {
    42
});

// Send the value from Alice to Bob and Carol
let num_at_bob_and_carol: MultiplyLocated<i32, LocationSet!(Bob, Carol)> =
    op.multicast(Alice, <LocationSet!(Bob, Carol)>::new(), &num_at_alice);

// Bob and Carol can now access the value
op.locally(Bob, |un| {
    let num_at_bob: &i32 = un.unwrap(&num_at_bob_and_carol);
    println!("The number at Bob is {}", num_at_bob);
});
op.locally(Carol, |un| {
    let num_at_carol: &i32 = un.unwrap(&num_at_bob_and_carol);
    println!("The number at Carol is {}", num_at_carol);
});
    }
}
}

The second parameter of the multicast is a value of the LocationSet type. You can use the new() method of the LocationSet type to obtain a value representation of the location set.

Both Bob and Carol can access the value sent from Alice inside their local computation using the same unwrap method.

Note on invalid values for Choreography::L

You'll get a compile error if you try to work with a ChoreographyLocation that is not a member of L.

#![allow(unused)]
fn main() {
# extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
// 1. Define a struct
struct HelloWorldChoreography;

// 2. Implement the `Choreography` trait
// ...
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        // this will fail
        op.locally(Bob, |_| {
            println!("Hello, World!");
        });
    }
}
}

Located Values

As we have seen in the Choreography section, a located value is a value that is available only at a specific location. In this section, we will discuss located values in more detail.

MultiplyLocated struct

The MultiplyLocated struct represents a multiply located value (MLV) that is available at multiple locations. It is a generic struct that takes two type parameters: a type parameter V that represents the type of the value, and a type parameter L that represents the location set where the value is available.

pub struct MultiplyLocated<V, L>
where
    L: LocationSet,
{
    // ...
}

The MultiplyLocated struct can be in one of the two states: Local and Remote. The Local state represents a located value that is available at the current location (that is, the current location is a member of the location set L). The Remote state represents a located value that is available at a different location.

Located type

In some cases, we may want to represent a located value that is available at a single location. For example, the return value of the locally operator is a (singly) located value that is available at the location given as an argument to the operator. The Located is used to represent such located values. It is a type alias for the MultiplyLocated struct with the location set containing only one location.

type Located<V, L1> = MultiplyLocated<V, LocationSet!(L1)>;

Portable trait

Located values can be sent from one location to another using the comm operator and unwrapped using the broadcast operator if the value type implements the Portable trait.

trait Portable: Serialize + DeserializeOwned {}

The Portable is defined as above. The Serialize and DeserializeOwned traits are from the serde crate and are used to serialize and deserialize the value for communication.

The chorus_lib crate re-exports the Serialize and Deserialize from serde. In many cases, those traits can automatically be derived using the #[derive(Serialize, Deserialize)] attribute.

For the complete list of types that supports automatic derivation of Serialize and Deserialize, see the serde documentation. The documentation also explains how to implement Serialize and Deserialize for custom types.

Transport

In order to execute choreographies, we need to be able to send messages between locations. ChoRus provides a trait Transport that abstracts over message transports.

Built-in Transports

ChoRus provides two built-in transports: local and http.

The Local Transport

The local transport is used to execute choreographies on the same machine on different threads. This is useful for testing and prototyping.

To use the local transport, we first need to create a LocalTransportChannel, which works as a channel between threads and allows them to send messages to each other. To do so, we use the LocalTransportChannelBuilder struct from the chorus_lib crate.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreographyLocation, LocationSet};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
use chorus_lib::transport::local::LocalTransportChannelBuilder;

let transport_channel = LocalTransportChannelBuilder::new()
    .with(Alice)
    .with(Bob)
    .build();
}

Using the with method, we add locations to the channel. When we call build, it will create an instance of LocalTransportChannel.

Then, create a transport by using the LocalTransport::new function, which takes a target location (explained in the Projector section) and the LocalTransportChannel.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreographyLocation, LocationSet};
#[derive(ChoreographyLocation)]
struct Alice;
use chorus_lib::transport::local::LocalTransportChannelBuilder;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).build();
use chorus_lib::transport::local::{LocalTransport};

let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
}

Because of the nature of the Local transport, you must use the same LocalTransportChannel instance for all locations. You can clone the LocalTransportChannel instance and pass it to each Projector::new constructor.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
use std::thread;
use chorus_lib::core::{ChoreographyLocation, ChoreoOp, Choreography, Projector, LocationSet};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
struct HelloWorldChoreography;
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice, Bob);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
    }
}
let transport_channel = LocalTransportChannelBuilder::new()
    .with(Alice)
    .with(Bob)
    .build();
let mut handles: Vec<thread::JoinHandle<()>> = Vec::new();
{
    // create a transport for Alice
    let transport = LocalTransport::new(Alice, transport_channel.clone());
    handles.push(thread::spawn(move || {
        let p = Projector::new(Alice, transport);
        p.epp_and_run(HelloWorldChoreography);
    }));
}
{
    // create another for Bob
    let transport = LocalTransport::new(Bob, transport_channel.clone());
    handles.push(thread::spawn(move || {
        let p = Projector::new(Bob, transport);
        p.epp_and_run(HelloWorldChoreography);
    }));
}
}

The HTTP Transport

The http transport is used to execute choreographies on different machines. This is useful for executing choreographies in a distributed system.

To use the http transport, import HttpTransport and HttpTransportConfigBuilder from the chorus_lib crate.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::transport::http::{HttpTransport, HttpTransportConfigBuilder};
}

We need to construct a HttpTransportConfig using the HttpTransportConfigBuilder. First, we specify the target location and the hostname and port to listen on using the for_target method. Then, we specify the other locations and their (hostname, port) pairs using the with method.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
use chorus_lib::transport::http::{HttpTransport, HttpTransportConfigBuilder};
// `Alice` listens on port 8080 on localhost
let config = HttpTransportConfigBuilder::for_target(Alice, ("localhost", 8080))
                // Connect to `Bob` on port 8081 on localhost
                .with(Bob, ("localhost", 8081))
                .build();
let transport = HttpTransport::new(config);
}

In the above example, the transport will start the HTTP server on port 8080 on localhost. If Alice needs to send a message to Bob, it will use http://localhost:8081 as the destination.

Creating a Custom Transport

You can also create your own transport by implementing the Transport trait. It might be helpful to first build a TransportConfig to have the the information that you need for each ChoreographyLocation, and then have a constructor that takes the TransportConfig and builds the Transport based on it. While the syntax is similar to HttpTransportConfig, which is HttpTransportConfigBuilder::for_target(target_location, target_information), chained with information about other locations using the .with(other_location, other_location_information), the type of information for each ChoreographyLocation might diverge from the (host_name, port) format presented in HttpTransport. In some cases, the target_information could even have a different type than the following other_location_information types. But all the other_location_informations should have the same type.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
use chorus_lib::transport::TransportConfigBuilder;
let config = TransportConfigBuilder::for_target(Alice, ())
                .with(Bob, ("localhost", 8081))
                .with(Carol, ("localhost", 8082))
                .build();
}

See the API documentation for more details.

Note on the location set of the Choreography

Note that when calling epp_and_run on a Projector, you will get a compile error if the location set of the Choreography is not a subset of the location set of the Transport. In other words, the Transport should have information about every ChoreographyLocation that Choreography can talk about. So this will fail:

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
use chorus_lib::core::{ChoreographyLocation, Projector, Choreography, ChoreoOp, LocationSet};

#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
struct HelloWorldChoreography;
impl Choreography for HelloWorldChoreography {
     type L = LocationSet!(Alice, Bob);
     fn run(self, op: &impl ChoreoOp<Self::L>) {
     }
}

let transport_channel = LocalTransportChannelBuilder::new().with(Alice).build();
let transport = LocalTransport::new(Alice, transport_channel.clone());
let projector = Projector::new(Alice, transport);
projector.epp_and_run(HelloWorldChoreography);
}

Projector

Projector is responsible for performing the end-point projection and executing the choreography.

Creating a Projector

To create a Projector, you need to provide the target location and the transport.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
use chorus_lib::core::{ChoreographyLocation, Projector, LocationSet};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let projector = Projector::new(Alice, alice_transport);
}

Notice that the Projector is parameterized by its target location type. You will need one projector for each location to execute choreography.

Executing a Choreography

To execute a choreography, you need to call the epp_and_run method on the Projector instance. The epp_and_run method takes a choreography, performs the end-point projection, and executes the choreography.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
use chorus_lib::core::{ChoreographyLocation, Projector, Choreography, ChoreoOp, LocationSet};
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
struct HelloWorldChoreography;
impl Choreography for HelloWorldChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
    }
}
let projector = Projector::new(Alice, alice_transport);
projector.epp_and_run(HelloWorldChoreography);
}

If the choreography has a return value, the epp_and_run method will return the value. We will discuss the return values in the Input and Output section.

Input and Output

To use ChoRus as part of a larger system, you need to be able to write a choreography that takes input and returns output.

Moreover, you may want to write a choreography that takes located values as input and returns located values as output. In such cases, you need to be able to construct/unwrap located values outside of the run method.

In this section, we will show you how to write a choreography that takes input and returns output.

Input

To take input, you can use fields of the struct that implements the Choreography trait. For example, the following choreography takes a String as input.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

struct DemoChoreography {
    input: String,
}

impl Choreography for DemoChoreography {
    type L = LocationSet!();
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        println!("Input: {}", self.input);
    }
}
}

You can construct an instance of the choreography with the input and pass it to the epp_and_run function.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct DemoChoreography {
    input: String,
}
impl Choreography for DemoChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        println!("Input: {}", self.input);
    }
}

let choreo = DemoChoreography {
    input: "World".to_string(),
};

let projector = Projector::new(Alice, alice_transport);
projector.epp_and_run(choreo);
}

Located Input

Input of normal types such as String must be available at all locations. However, you may want to take input that is only available at a single location. You can do so by using a located value as a field of the choreography struct.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct DemoChoreography {
    input: Located<String, Alice>,
}

impl Choreography for DemoChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        op.locally(Alice, |un| {
            let input = un.unwrap(&self.input);
            println!("Input at Alice: {}", input);
        });
    }
}
}

Because the input field is located at Alice, you can only access the string at Alice using the Unwrapper.

To construct this choreography, you must pass a located value. The Projector struct provides two methods to construct located values: local and remote.

local constructs a located value that is available at the projection target. You must provide an actual value as an argument. The location will be the same as the target of the projector.

remote constructs a located value that is available at a different location. You must provide a location of the value. Note that this location must be different from the target of the projector. As of now, ChoRus does not check this at compile time. If you pass the same location as the target of the projector, the program will panic at runtime.

To run the sample choreography above at Alice, we use the local method to construct the located value.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct DemoChoreography {
    input: Located<String, Alice>,
}

impl Choreography for DemoChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        op.locally(Alice, |un| {
            let input = un.unwrap(&self.input);
            println!("Input at Alice: {}", input);
        });
    }
}
let projector_for_alice = Projector::new(Alice, alice_transport);
// Because the target of the projector is Alice, the located value is available at Alice.
let string_at_alice: Located<String, Alice> = projector_for_alice.local("Hello, World!".to_string());
// Instantiate the choreography with the located value
let choreo = DemoChoreography {
    input: string_at_alice,
};
projector_for_alice.epp_and_run(choreo);
}

For Bob, we use the remote method to construct the located value.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct DemoChoreography {
    input: Located<String, Alice>,
}

impl Choreography for DemoChoreography {
    type L = LocationSet!(Alice, Bob);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        op.locally(Alice, |un| {
            let input = un.unwrap(&self.input);
            println!("Input at Alice: {}", input);
        });
    }
}
let projector_for_bob = Projector::new(Bob, bob_transport);
// Construct a remote located value at Alice. The actual value is not required.
let string_at_alice = projector_for_bob.remote(Alice);
// Instantiate the choreography with the located value
let choreo = DemoChoreography {
    input: string_at_alice,
};
projector_for_bob.epp_and_run(choreo);
}

Output

Similarly, we can get output from choreographies by returning a value from the run method.

To do so, we specify the output type to the Choreography trait and return the value of the type from the run method.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct DemoChoreography;

impl Choreography<String> for DemoChoreography {
    type L = LocationSet!();
    fn run(self, op: &impl ChoreoOp<Self::L>) -> String {
        "Hello, World!".to_string()
    }
}
}

epp_and_run returns the value returned from the run method.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct DemoChoreography;

impl Choreography<String> for DemoChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) -> String {
        "Hello, World!".to_string()
    }
}
let choreo = DemoChoreography;
let projector = Projector::new(Alice, alice_transport);
let output = projector.epp_and_run(choreo);
assert_eq!(output, "Hello, World!".to_string());
}

Located Output

You can use the Located<V, L1> as a return type of the run method to return a located value. The projector provides a method unwrap to unwrap the output located values.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct DemoChoreography;

impl Choreography<Located<String, Alice>> for DemoChoreography {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) -> Located<String, Alice> {
        op.locally(Alice, |_| {
            "Hello, World!".to_string()
        })
    }
}

let projector = Projector::new(Alice, alice_transport);
let output = projector.epp_and_run(DemoChoreography);
let string_at_alice = projector.unwrap(output);
assert_eq!(string_at_alice, "Hello, World!".to_string());
}

Because projectors are parametric over locations, you can only unwrap located values at the target location.

You can return multiple located values by returning a tuple or struct that contains multiple located values. They don't have to be located at the same location, but you can only unwrap them at the correct location.

Runner

Runner is a struct that can be used to run a Choreography without doing end-point projection. It gives semantics to the Choreography and allows it to be run in a way that is similar to a function call.

To use Runner, construct an instance using the new constructor, and then call the run method with the Choreography.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct DemoChoreography;
impl Choreography for DemoChoreography {
    type L = LocationSet!();
    fn run(self, op: &impl ChoreoOp<Self::L>) {
    }
}
let runner = Runner::new();
runner.run(DemoChoreography);
}

As described in the Input and Output section, Runner can also pass values to the Choreography and receive values from it.

Because Runner executes the Choreography at all locations, all located inputs must be provided. Also, Runner can unwrap any located values returned by the Choreography.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct SumChoreography {
    x_at_alice: Located<u32, Alice>,
    y_at_bob: Located<u32, Bob>,
}
impl Choreography<Located<u32, Carol>> for SumChoreography {
    type L = LocationSet!(Alice, Bob, Carol);
    fn run(self, op: &impl ChoreoOp<Self::L>) -> Located<u32, Carol> {
        let x_at_carol = op.comm(Alice, Carol, &self.x_at_alice);
        let y_at_carol = op.comm(Bob, Carol, &self.y_at_bob);
        op.locally(Carol, |un| {
            let x = un.unwrap(&x_at_carol);
            let y = un.unwrap(&y_at_carol);
            x + y
        })
    }
}

let runner = Runner::new();
let x_at_alice = runner.local(1);
let y_at_bob = runner.local(2);
let sum_at_carol = runner.run(SumChoreography {
    x_at_alice,
    y_at_bob,
});
assert_eq!(runner.unwrap(sum_at_carol), 3);
}

Higher-order Choreography

Higher-order choreography is a choreography that takes another choreography as an argument. Just like higher-order functions, higher-order choreographies are useful for abstracting over common patterns.

This section describes how to define and execute higher-order choreographies.

Defining a Higher-order Choreography

To define a higher-order choreography, you need to create a generic struct that takes a type parameter that implements the Choreography trait.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct HigherOrderChoreography<C: Choreography> {
    sub_choreo: C,
};
}

When you implement the Choreography trait, you have access to the sub_choreo field. You can use the call method to execute the sub-choreography.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct HigherOrderChoreography<C: Choreography> {
    sub_choreo: C,
};
impl<C: Choreography<(), L = LocationSet!(Alice, Bob)>> Choreography for HigherOrderChoreography<C> {
    type L = LocationSet!(Alice, Bob);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        op.call(self.sub_choreo);
    }
}
}

Passing values to a sub-choreography

It is often useful to pass values to a sub-choreography. To do so, instead of storing the sub-choreography object as a field, you associate the sub-choreography trait with the choreography using the std::marker::PhantomData type.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
use std::marker::PhantomData;

trait SubChoreography {
    fn new(arg: Located<i32, Alice>) -> Self;
}

struct HigherOrderChoreography<C: Choreography<Located<bool, Alice>> + SubChoreography> {
    _marker: PhantomData<C>,
};

impl<C: Choreography<Located<bool, Alice>, L = LocationSet!(Alice)> + SubChoreography> Choreography for HigherOrderChoreography<C> {
    type L = LocationSet!(Alice);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        let num_at_alice = op.locally(Alice, |_| {
            42
        });
        let sub_choreo = C::new(num_at_alice);
        op.call(sub_choreo);
    }
}
}

Here, the HigherOrderChoreography struct takes a type parameter C that implements both the Choreography trait and the SubChoreography trait. The SubChoreography trait ensures that the C type can be constructed with a located integer at Alice using the new constructor.

Location Polymorphism

Another feature of ChoRus is location polymorphism. Location polymorphism allows choreographies to be defined using generic locations. These locations can then be instantiated with concrete locations, allowing the choreography to be executed on different locations.

To define a location-polymorphic choreography, you need to create a generic struct that takes a type parameter that implements the ChoreographyLocation trait. When instantiating the choreography, you can pass a concrete location.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());
struct LocationPolymorphicChoreography<L1: ChoreographyLocation> {
    location: L1,
}

impl<L1: ChoreographyLocation> Choreography for LocationPolymorphicChoreography<L1> {
    type L = LocationSet!(L1);
    fn run(self, op: &impl ChoreoOp<Self::L>) {
        op.locally(self.location, |_| {
            println!("Hello, World!");
        });
    }
}

let alice_say_hello = LocationPolymorphicChoreography {
    location: Alice,
};
let bob_say_hello = LocationPolymorphicChoreography {
    location: Bob,
};
}

Efficient Conditionals with Enclaves and MLVs

broadcast incurs unnecessary communication

In the previous section, we discussed how the broadcast operator can be used to implement a conditional behavior in a choreography. In short, the broadcast operator sends a located value from a source location to all other locations, making the value available at all locations. The resulting value is a normal (not Located) value and it can be used to make a branch.

However, the broadcast operator can incur unnecessary communication when not all locations need to receive the value. Consider a simple key-value store where a client sends either a Get or Put request to a primary server, and the primary server forwards the request to a backup server if the request is a Put. The backup server does not need to receive the request if the request is a Get.

Using the broadcast operator, this protocol can be implemented as follows:

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

fn read_request() -> Request {
    Request::Put("key".to_string(), "value".to_string())
}
fn get_value(key: &Key) -> Option<Value> {
    Some("value".to_string())
}
fn set_value(key: &Key, value: &Value) {
    println!("Saved key: {} and value: {}", key, value);
}

#[derive(ChoreographyLocation)]
struct Client;

#[derive(ChoreographyLocation)]
struct Primary;

#[derive(ChoreographyLocation)]
struct Backup;

type Key = String;
type Value = String;

#[derive(Serialize, Deserialize)]
enum Request {
    Get(Key),
    Put(Key, Value),
}

#[derive(Serialize, Deserialize)]
enum Response {
    GetOk(Option<Value>),
    PutOk,
}

struct KeyValueStoreChoreography;

impl Choreography<Located<Response, Client>> for KeyValueStoreChoreography {
    type L = LocationSet!(Client, Primary, Backup);
    fn run(self, op: &impl ChoreoOp<Self::L>) -> Located<Response, Client> {
        // Read the request from the client
        let request_at_client: Located<Request, Client> = op.locally(Client, |_| read_request());
        // Send the request to the primary server
        let request_at_primary: Located<Request, Primary> =
            op.comm(Client, Primary, &request_at_client);
        // Check if the request is a `Put`
        let is_put_at_primary: Located<bool, Primary> = op.locally(Primary, |un| {
            matches!(un.unwrap(&request_at_primary), Request::Put(_, _))
        });
        // Broadcast the `is_put_at_primary` to all locations so it can be used for branching
        let is_put: bool = op.broadcast(Primary, is_put_at_primary); // <-- Incurs unnecessary communication
        // Depending on the request, set or get the value
        let response_at_primary = if is_put {
            let request_at_backup: Located<Request, Backup> =
                op.comm(Primary, Backup, &request_at_primary);
            op.locally(Backup, |un| match un.unwrap(&request_at_backup) {
                Request::Put(key, value) => set_value(key, value),
                _ => (),
            });
            op.locally(Primary, |_| Response::PutOk)
        } else {
            op.locally(Primary, |un| {
                let key = match un.unwrap(&request_at_primary) {
                    Request::Get(key) => key,
                    _ => &"".to_string(),
                };
                Response::GetOk(get_value(key))
            })
        };
        // Send the response from the primary to the client
        let response_at_client = op.comm(Primary, Client, &response_at_primary);
        response_at_client
    }
}
}

While this implementation works, it incurs unnecessary communication. When we branch on is_put, we broadcast the value to all locations. This is necessary to make sure that the value is available at all locations so it can be used as a normal, non-located value. However, notice that the client does not need to receive the value. Regardless of whether the request is a Put or Get, the client should wait for the response from the primary server.

Changing the census with enclave

To avoid unnecessary communication, we can use the enclave operator. The enclave operator is similar to the call operator but executes a sub-choreography only at locations that are included in its location set. Inside the sub-choreography, broadcast only sends the value to the locations that are included in the location set. This allows us to avoid unnecessary communication.

Let's refactor the previous example using the enclave operator. We define a sub-choreography HandleRequestChoreography that describes how the primary and backup servers (but not the client) handle the request and use the enclave operator to execute the sub-choreography.

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

fn read_request() -> Request {
    Request::Put("key".to_string(), "value".to_string())
}
fn get_value(key: &Key) -> Option<Value> {
    Some("value".to_string())
}
fn set_value(key: &Key, value: &Value) {
    println!("Saved key: {} and value: {}", key, value);
}

#[derive(ChoreographyLocation)]
struct Client;

#[derive(ChoreographyLocation)]
struct Primary;

#[derive(ChoreographyLocation)]
struct Backup;

type Key = String;
type Value = String;

#[derive(Serialize, Deserialize)]
enum Request {
    Get(Key),
    Put(Key, Value),
}

#[derive(Serialize, Deserialize)]
enum Response {
    GetOk(Option<Value>),
    PutOk,
}

struct HandleRequestChoreography {
    request: Located<Request, Primary>,
}

// This sub-choreography describes how the primary and backup servers handle the request
impl Choreography<Located<Response, Primary>> for HandleRequestChoreography {
    type L = LocationSet!(Primary, Backup);
    fn run(self, op: &impl ChoreoOp<Self::L>) -> Located<Response, Primary> {
        let is_put_request: Located<bool, Primary> = op.locally(Primary, |un| {
            matches!(un.unwrap(&self.request), Request::Put(_, _))
        });
        let is_put: bool = op.broadcast(Primary, is_put_request);
        let response_at_primary = if is_put {
            let request_at_backup: Located<Request, Backup> =
                op.comm(Primary, Backup, &self.request);
            op.locally(Backup, |un| match un.unwrap(&request_at_backup) {
                Request::Put(key, value) => set_value(key, value),
                _ => (),
            });
            op.locally(Primary, |_| Response::PutOk)
        } else {
            op.locally(Primary, |un| {
                let key = match un.unwrap(&self.request) {
                    Request::Get(key) => key,
                    _ => &"".to_string(),
                };
                Response::GetOk(get_value(key))
            })
        };
        response_at_primary
    }
}

struct KeyValueStoreChoreography;

impl Choreography<Located<Response, Client>> for KeyValueStoreChoreography {
    type L = LocationSet!(Client, Primary, Backup);
    fn run(self, op: &impl ChoreoOp<Self::L>) -> Located<Response, Client> {
        let request_at_client: Located<Request, Client> = op.locally(Client, |_| read_request());
        let request_at_primary: Located<Request, Primary> =
            op.comm(Client, Primary, &request_at_client);
        // Execute the sub-choreography only at the primary and backup servers
        let response: MultiplyLocated<Located<Response, Primary>, LocationSet!(Primary, Backup)> =
            op.enclave(HandleRequestChoreography {
                request: request_at_primary,
            });
        let response_at_primary: Located<Response, Primary> = response.flatten();
        let response_at_client = op.comm(Primary, Client, &response_at_primary);
        response_at_client
    }
}
}

In this refactored version, the HandleRequestChoreography sub-choreography describes how the primary and backup servers handle the request. The enclave operator executes the sub-choreography only at the primary and backup servers. The broadcast operator inside the sub-choreography sends the value only to the primary and backup servers, avoiding unnecessary communication.

The enclave operator returns a return value of the sub-choreography wrapped as a MultiplyLocated value. Since HandleRequestChoreography returns a Located<Response, Primary>, the return value of the enclave operator is a MultiplyLocated<Located<Response, Primary>, LocationSet!(Primary, Backup)>. To get the located value at the primary server, we can use the locally operator to unwrap the MultiplyLocated value on the primary. Since this is a common pattern, we provide the flatten method on MultiplyLocated to simplify this operation.

With the enclave operator, we can avoid unnecessary communication and improve the efficiency of the choreography.

Reusing Knowledge of Choice in Enclaves

The key idea behind the enclave operator is that a normal value inside a choreography is equivalent to a (multiply) located value at all locations executing the choreography. This is why a normal value in a sub-choreography becomes a multiply located value at all locations executing the sub-choreography when returned from the enclave operator.

It is possible to perform this conversion in the opposite direction as well. If we have a multiply located value at some locations, and those are the only locations executing the choreography, then we can obtain a normal value out of the multiply located value. This is useful when we want to reuse the already known information about a choice in an enclave.

Inside a choreography, we can use the naked operator to convert a multiply located value at locations S to a normal value if the census of the choreography is a subset of S.

For example, the above choreography can be written as follows:

#![allow(unused)]
fn main() {
extern crate chorus_lib;
use chorus_lib::core::{ChoreoOp, Choreography, ChoreographyLocation, Projector, Located, MultiplyLocated, Runner, LocationSet, Serialize, Deserialize};
use chorus_lib::transport::local::{LocalTransport, LocalTransportChannelBuilder};
#[derive(ChoreographyLocation)]
struct Alice;
#[derive(ChoreographyLocation)]
struct Bob;
#[derive(ChoreographyLocation)]
struct Carol;
let transport_channel = LocalTransportChannelBuilder::new().with(Alice).with(Bob).with(Carol).build();
let alice_transport = LocalTransport::new(Alice, transport_channel.clone());
let bob_transport = LocalTransport::new(Bob, transport_channel.clone());
let carol_transport = LocalTransport::new(Carol, transport_channel.clone());

fn read_request() -> Request {
    Request::Put("key".to_string(), "value".to_string())
}
fn get_value(key: &Key) -> Option<Value> {
    Some("value".to_string())
}
fn set_value(key: &Key, value: &Value) {
    println!("Saved key: {} and value: {}", key, value);
}

#[derive(ChoreographyLocation)]
struct Client;

#[derive(ChoreographyLocation)]
struct Primary;

#[derive(ChoreographyLocation)]
struct Backup;

type Key = String;
type Value = String;

#[derive(Serialize, Deserialize)]
enum Request {
    Get(Key),
    Put(Key, Value),
}

#[derive(Serialize, Deserialize)]
enum Response {
    GetOk(Option<Value>),
    PutOk,
}

struct HandleRequestChoreography {
    request: Located<Request, Primary>,
    is_put: MultiplyLocated<bool, LocationSet!(Primary, Backup)>,
}

impl Choreography<Located<Response, Primary>> for HandleRequestChoreography {
    type L = LocationSet!(Primary, Backup);
    fn run(self, op: &impl ChoreoOp<Self::L>) -> Located<Response, Primary> {
        // obtain a normal boolean because {Primary, Backup} is the census of the choreography
        let is_put: bool = op.naked(self.is_put);
        let response_at_primary = if is_put {
            // ...
            let request_at_backup: Located<Request, Backup> =
                op.comm(Primary, Backup, &self.request);
            op.locally(Backup, |un| match un.unwrap(&request_at_backup) {
                Request::Put(key, value) => set_value(key, value),
                _ => (),
            });
            op.locally(Primary, |_| Response::PutOk)
        } else {
            // ...
            op.locally(Primary, |un| {
                let key = match un.unwrap(&self.request) {
                    Request::Get(key) => key,
                    _ => &"".to_string(),
                };
                Response::GetOk(get_value(key))
            })
        };
        response_at_primary
    }
}

struct KeyValueStoreChoreography;

impl Choreography<Located<Response, Client>> for KeyValueStoreChoreography {
    type L = LocationSet!(Client, Primary, Backup);
    fn run(self, op: &impl ChoreoOp<Self::L>) -> Located<Response, Client> {
        let request_at_client: Located<Request, Client> = op.locally(Client, |_| read_request());
        let request_at_primary: Located<Request, Primary> =
            op.comm(Client, Primary, &request_at_client);
        let is_put_at_primary: Located<bool, Primary> = op.locally(Primary, |un| {
            matches!(un.unwrap(&request_at_primary), Request::Put(_, _))
        });
        // get a MLV by multicasting the boolean to the census of the sub-choreography
        let is_put: MultiplyLocated<bool, LocationSet!(Primary, Backup)> = op.multicast(
            Primary,
            <LocationSet!(Primary, Backup)>::new(),
            &is_put_at_primary,
        );
        let response: MultiplyLocated<Located<Response, Primary>, LocationSet!(Primary, Backup)> =
            op.enclave(HandleRequestChoreography {
                is_put,
                request: request_at_primary,
            });
        let response_at_primary: Located<Response, Primary> = response.flatten();
        let response_at_client = op.comm(Primary, Client, &response_at_primary);
        response_at_client
    }
}
}

In this version, we first multicast the boolean value to the census of the sub-choreography (Primary and Client) and we pass the MLV to the sub-choreography. Inside the sub-choreography, we use the naked operator to obtain a normal boolean value. This allows us to reuse the already known information about the choice in the sub-choreography.

Links

Here are some links you might find useful:

• Chorus website
• Chorus API documentation
• Chorus GitHub repository