Running Reports locally

For background information about reports, (click here)[./infrastructure].

Required

• Dev Container setup

Setting up the postgres database

After setting up the dev container, reports needs postgres setup with data before it can start generating reports.

Run in local machine terminal

docker exec -it meta_devcontainer-postgres-1  psql -h localhost -U postgres

Run the following commands

CREATE DATABASE reports;

-- We'll create two read-only roles for the reports-generator and
-- reports-orchestrator components respectively, and a more privileged role for
-- the postgres-sink component.
--
-- All of the roles are members of the rds_iam role so that they can connect
-- using IAM database authentication.
CREATE USER reports_generator;
GRANT CONNECT ON DATABASE reports TO reports_generator;

CREATE USER reports_orchestrator;
GRANT CONNECT ON DATABASE reports TO reports_orchestrator;

CREATE USER reports_postgres_sink;
GRANT ALL PRIVILEGES ON DATABASE reports TO reports_postgres_sink;

\connect reports;

-- Allow our read-only roles to read the schema, and the postgres-sink component
-- to create tables in the schema.
GRANT USAGE ON SCHEMA public TO reports_generator;
GRANT USAGE ON SCHEMA public TO reports_orchestrator;
GRANT ALL ON SCHEMA public TO reports_postgres_sink;

-- The following commands must be run as the role that will create the tables on
-- which we wish to grant default privileges (i.e. reports_postgres_sink).
SET ROLE reports_postgres_sink;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO reports_generator;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO reports_orchestrator;

Getting data into Postgres

For local testing, use datasets/reports for the scenarios needed. To write the data into the queue, run pnpm reset-data /reports/<specific folder>

As data is written in the queue, postgres-sink will start adding the messages into Postgres once it is running. Postgres-sink's only job is to insert records into the database.

Run in dev container terminal or shell:

pnpm start:reports-postgres-sink

Postgres-sink is working when "Inserted XX records in X ms" is printed in the logs. Kill the service once there hasn't been a new inserted message in a while. You can leave it running but it is not necessary.

View data in tables

To view the data in postgres, use the following command in terminal or use a database app to view the tables

docker exec -it meta_devcontainer-postgres-1  psql -h localhost -U postgres
\connect reports

\dt will show all tables in postgres shell.

For database app, this is the information to connect to postgres in dev container:

host: localhost
port: 39141
user: postgres
password: password

Orchestrator

The orchestrator reads messages from the queue topic and inserts jobs into a redis queue. Redis setup is part of the dev container setup and should be running in Docker. Run in dev container terminal:

pnpm start:reports-orchestrator

To add messages to the queue, use redpanda or use pnpm reset-data /reports/messages/<specific folder>

Generator

The generator will pick up a job from the redis queue, gather the data, and generate the files. The files will be saved to the local reports directory. Run in dev container terminal:

pnpm start:reports-generator