Developer setup

Set up your environment to develop FlowMatrix

Dependencies

To build FlowMatrix, you will need to install some dependencies.

Ubuntu

Follow these command to get the necessary dependencies to build FlowMatrix on Ubuntu:

$ sudo apt-get install pkg-config build-essential \
  libssl-dev openssl cmake curl postgresql postgresql-client protobuf-compiler
$ sudo systemctl start postgresql
$ curl https://sh.rustup.rs -sSf | sh -s -- -y
$ cargo install refinery_cli
$ curl -fsSL https://get.pnpm.io/install.sh | sh -

MacOS

First, install Homebrew if it’s not already installed. Then, run the following commands:

$ brew install postgresql
$ brew install protobuf
$ brew install pnpm

We also need to install CMake, which is used by some of the C and C++ libraries we’re dependant on, like librdkafka. The current version in homebrew (4.0.x) is too new, so a 3.x.x version must be installed manually:

$ curl -OL https://github.com/Kitware/CMake/releases/download/v3.31.7/cmake-3.31.7-macos-universal.tar.gz
$ tar xvfz cmake-3.*-macos-universal.tar.gz
$ mv cmake-3.*/Cmake.app /Applications

Then /Applications/Cmake.app/Contents/bin must be added your path, for example by putting this line in your .zshrc or .bashrc:

export PATH=$PATH:/Applications/CMake.app/Contents/bin

Next install Rust:

$ curl https://sh.rustup.rs -sSf | sh -s -- -y

Finally, install the following packages:

$ cargo install refinery_cli
$ cargo install cargo-nextest

Get the source

The FlowMatrix source can be checked out from GitHub with:

$ git clone https://github.com/FlowMatrixSystems/flowmatrix.git

Postgres

Developing FlowMatrix requires running a properly configured postgres instance. By default, it expects a database called flowmatrix, a user flowmatrix with password flowmatrix, although that can be changed by setting the following environment variables:

• DATABASE_NAME
• DATABASE_HOST
• DATABASE_PORT
• DATABASE_USER
• DATABASE_PASSWORD

On Ubuntu, you can set up a compatible database like this:

$ sudo -u postgres psql -c "CREATE USER flowmatrix WITH PASSWORD 'flowmatrix' SUPERUSER;"
$ sudo -u postgres createdb flowmatrix

On MacOS:

$ psql postgres -c "CREATE USER flowmatrix WITH PASSWORD 'flowmatrix' SUPERUSER;"
$ createdb flowmatrix

Migrations are managed using refinery. Once Postgres is set up, you can initialize it with

$ refinery setup # follow the prompts
$ mv refinery.toml ~/
$ refinery migrate -c ~/refinery.toml -p crates/flowmatrix-api/migrations

We use cornucopia for typed-checked SQL queries. Our build is set up to automatically re-generate the rust code for those queries on build, so you will need a DB set up for compilation as well.

Building the frontend

We use pnpm and vite for frontend development. Once pnpm is installed (following the OS-specific) instructions above, you should be able to build the console like this:

$ cd webui
$ pnpm install
$ pnpm build

This must be done before running the services, as the API service will expect to find the compiled webui source so it can serve it.

Building the services

FlowMatrix is built via Cargo, the Rust build tool. All services are built into a single binary, flowmatrix, which can be built with

$ cargo build --package flowmatrix

By default, Cargo builds in debug mode which gives you shorter build times in exchange for slower execution speeds. This is great for development, but you should make sure to always build in release mode (cargo build --release) before deploying or benchmarking.

This will build the binary in target/debug/flowmatrix. Running that you should see the available subcommands:

$ target/debug/flowmatrix
Usage: flowmatrix <COMMAND>

Commands:
  api         Starts an FlowMatrix API server
  controller  Starts an FlowMatrix Controller
  cluster     Starts a complete FlowMatrix cluster
  worker      Starts an FlowMatrix worker
  compiler    Starts an FlowMatrix compiler
  node        Starts an FlowMatrix node server
  migrate     Runs database migrations on the configure Postgres database
  help        Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

Running a cluster

All of the services can be run together in a single cluster, or individually. For development, it’s most typical to run them together using the cluster subcommand:

$ target/debug/flowmatrix cluster

This will start a new FlowMatrix cluster with the default “process” scheduler, which runs each pipeline as a separate process on the same node.

You can then open http://localhost:5115 to use your new cluster.

Cargo also allows combining the build-run process into a single step, with the cargo run command, for example:

$ cargo run --bin flowmatrix -- cluster

On MacOS, you may see this error in the controller logs when trying to run a job:

Child ("/tmp/flowmatrix-process/m2lgldjw") exited with status Ok(ExitStatus(unix_wait_status(9)))

This means that MacOS gatekeeper is killing the pipeline binary. To allow the pipeline to run, open System Settings and navigate to Privacy & Security -> Developer Tools. In the list, add and enable your terminal (for example Terminal.app or iTerm).

Front-end development

When developing the frontend, you can take advantage of vite’s dev server to shorten the dev cycle:

$ cd webui
$ pnpm run dev

Then visit http://localhost:5173/. This requires the cluster is running on http://localhost:5115.

Updating frontend definitions

After altering protobuf definitions in flowmatrix-rpc you need to regenerate the Typescript code used by the frontend:

$ pnpm run generate

Similarly, after altering REST API types in flowmatrix-api you need to run:

$ pnpm run openapi

Testing

FlowMatrix includes a large suite of tests covering most parts of the system. We use the standard Rust test framework, which can be invoked via

$ cargo test

We also recommend cargo-nextest, which provides a better UX on top of the standard test runner.

$ cargo nextest run

See the nextest docs for all options.

Connector tests may require a running local instance of that system to pass; for example the full test suite currently requires kafka and an mqtt broker.

Building docker

The FlowMatrix repo includes a Dockerfile to package FlowMatrix for development and deployment via Kubernetes or Docker Compose.

For efficiency, we use targets to build images for different purposes. The available targets are

• flowmatrix — includes only the FlowMatrix binary; most useful for deploying on Kubernetes or in Docker Compose
• flowmatrix-full — also includes a Rust compile environment; this is needed for building UDFs within the Web UI; the compiler service will dynamically fetch the necessary dependencies, but you may choose to use flowmatrix-full if you’re running in an environment that does not allow external internet access

To build a particular target, run

$ docker build . -f docker/Dockerfile -t {target} --target {target}

for example,

$ docker build . -f docker/Dockerfile -t flowmatrix-full --target flowmatrix-full

Note that this requires a fairly recent version of Docker that supports buildx.