Workflows

Audience: Programmer

Overview

pyck combines Temporal workflow engine with our CRUD+ API to create complete, customized warehouse software solutions. This document explains the workflow architecture, integration points, and development patterns.

System Architecture

Key Components

1. pyck Core Services: Our CRUD+ microservices providing foundational warehouse functionality

2. Temporal Service: Durable execution service integrated within our core

3. Database: Persistent storage for both pyck services and Temporal

4. API Layer: GraphQL API from pyck services and gRPC API from Temporal

5. External Workflows: Custom workflow implementations that operate outside the core system

6. Frontend Applications: Mobile and web UIs that interact with the system

Workflow Architecture

                         ┌─────────────────────────────────────────────┐
                         │              pyck Core                      │
                         │                                             │
                         │  ┌─────────────┐    ┌─────────────────┐     │
                         │  │             │    │                 │     │
                         │  │ Database    │◄───┤ Core Services   │     │
                         │  │             │    │                 │     │
                         │  └─────────────┘    └────────┬────────┘     │
                         │                             ▲               │
                         │                             │               │
                         │  ┌─────────────┐            │               │
                         │  │             │            │               │
                         │  │ Temporal    │────────────┘               │
                         │  │             │                            │
                         │  └─────────────┘                            │
                         │                                             │
                         └─────────────┬──────────────┬────────────────┘
                                       │              │
                                       ▼              ▼
                         ┌─────────────────┐    ┌────────────────┐
                         │                 │    │                │
                         │  GraphQL API    │    │  gRPC API      │
                         │                 │    │  (Temporal)    │
                         └────────┬────────┘    └────────┬───────┘
                                  │                      │
                                  ▼                      ▼
           ┌─────────────────────────────────────────────────────────────┐
           │                     External Systems                         │
           │                                                             │
           │  ┌─────────────────┐                  ┌─────────────────┐   │
           │  │                 │                  │                 │   │
           │  │ Workflows       │────────────────►│ Frontend Apps   │   │
           │  │                 │                  │                 │   │
           │  └─────────────────┘                  └─────────────────┘   │
           │                                                             │
           └─────────────────────────────────────────────────────────────┘

Why Temporal?

Temporal provides critical capabilities for warehouse operations:

1. Crash-proof Execution: Workflows never lose state, even when systems fail

2. Fault Tolerance: Business logic continues despite network failures or API outages

3. Long-running Processes: Support for workflows that run for days, weeks, or months

4. Task Retries: Automatic retries with configurable policies for intermittent failures

5. Visibility: Complete history and real-time status of workflow executions

Workflow Development

Setup and Configuration

1. Project Structure: A typical warehouse workflow project includes directories for workflow definitions, activity implementations, data models, API clients, worker service entry point, and tests.

2. Dependencies: Workflow projects require dependencies for the Temporal SDK and pyck client libraries.

Workflow Registration

Workflows must be registered with both Temporal and the pyck API. This two-step registration process happens during application startup:

1. Temporal Registration: Workflows and activities are registered with the Temporal worker

2. pyck API Registration: Workflows are registered with the pyck GraphQL API to associate them with specific events

The registration process maps workflows to specific event types in the system. For example, a workflow named "TemporalWorkflowPicking_OrderCreate" would be triggered by the "OrderCreate" event.

Workflow ID Generation

Workflow IDs in pyck are generated deterministically to ensure consistency and prevent duplicate executions:

• With pyck_data_id: When pyck_data_id is present in the search attributes, it is used as the suffix for the workflow ID (e.g., WorkflowName_<pyck_data_id>)

• Without pyck_data_id: When no pyck_data_id is provided, a new UUID is generated as the suffix (e.g., WorkflowName_<uuid>)

This deterministic ID generation ensures that:

• Workflows tied to specific data entities can be uniquely identified

• Duplicate workflows for the same data entity are prevented

• Workflow state can be reliably tracked across the system

Workflow Definition

Workflows in Temporal are defined as functions that orchestrate a series of activities. A typical picking workflow would include steps like:

• Fetching order details

• Allocating inventory

• Generating picking tasks

• Updating mobile app UI for picker

• Waiting for picking completion

• Processing results and updating inventory

Frontend Control Points

Workflows control frontend behavior through a combination of:

1. Activity Properties: Each workflow activity can define UI properties that should be displayed

2. Signal Channels: Workflows wait for user input through signal channels

3. NextActivityName: Defining the next activity tells the frontend what screen to show

Event Triggering and Signal Routing

Workflows are triggered by events from the system through an advanced signal routing mechanism that differentiates between START and INTERMEDIATE signals:

1. START Signals: Create new workflow instances when triggered

2. INTERMEDIATE Signals: Forward signals to already running workflow instances

When an event occurs in the pyck system:

• The system uses JetStream-based routing to determine which workflows should be triggered

• START signals will create new workflow instances (or fail if a workflow with the same ID is already running)

• INTERMEDIATE signals will be forwarded to matching running workflows

For example, when an order is created in the system, the "OrderCreate" event is fired, which then triggers the "TemporalWorkflowPicking_OrderCreate" workflow.

Important Breaking Change: As of the latest version, running workflows cannot be reassigned on re-trigger. Attempting to start a workflow that is already running will result in an error. This ensures workflow state consistency and prevents accidental duplicate executions.

User Input Communication Flow

The communication between Temporal workflows and the frontend when waiting for user input follows a specific stateful pattern:

1. Workflow State Management

Workflows maintain their state through a query mechanism that allows external systems to inspect their current state without affecting execution:

1. Query Handler: Each workflow exposes its current UI state through a query handler for "nextActivityProperties"

2. Activity Definition: The workflow defines what activity the user should perform next

3. Signal Channels: The workflow creates channels to receive signals from the frontend

4. Waiting for Input: The workflow pauses execution while waiting for user input

2. Frontend Interaction Cycle

The frontend interacts with workflows through a well-defined cycle:

1. Discover Workflows: Frontend finds workflows assigned to the current user via the checkAssignedWorkflows GraphQL mutation

2. Get Current Activity: The system retrieves the current activity information from the workflow

3. Render UI Components: Frontend displays the appropriate UI based on the activity properties

4. Collect User Input: User interacts with the UI and provides the requested information

5. Submit Input and Get Next Step: When the user submits input, the frontend calls a single GraphQL mutation (userDataInput) that:

   • Sends the user input to the workflow

   • Waits for the workflow to process the input

   • Returns the next activity in the same response

This means the frontend doesn't need to separately poll for the next activity after submitting user input - it gets both acknowledgment of the submitted data and instructions for the next step in a single GraphQL mutation.

3. GraphQL Mutation Synchronization

When the userDataInput GraphQL mutation is called:

1. Signal Sending: The GraphQL resolver sends the user input to the workflow via a Temporal signal

2. Synchronous Waiting: The resolver then waits for the workflow to process the input

3. State Transition Detection: The resolver monitors the workflow state, waiting for it to change from the current activity to the next activity

4. Response with Next Step: Once the workflow updates its state, the GraphQL mutation returns with information about the next activity the frontend should display

4. Workflow Processing

Inside the workflow, when it receives user input:

1. Signal Reception: The workflow's signal channel receives the user input

2. State Update: The workflow updates its activity state to indicate the current activity is passed

3. Input Processing: The workflow processes the input, often calling an activity

4. Next Activity Definition: The workflow defines the next activity for the user

5. State Exposure: The updated state becomes available to the GraphQL resolver

This continuous cycle allows the workflow to drive the frontend state while maintaining the reliability guarantees provided by Temporal's durable execution model.

BPMN to Code Translation

We use BPMN for visual workflow design and translate it to Temporal code:

1. BPMN Process: Designed with logistics team for business requirements

2. Activities: BPMN tasks become Temporal activities

3. Gateways: BPMN gateways become workflow conditions

4. Events: BPMN events map to Temporal signals/triggers

The workflow directory structure typically follows this pattern:

processes/
  {priority}_{domain}/
    {id}_{event}/
      activities/      # Activity implementation
      domain/          # Data models
      workflows/       # Workflow definition
      {event}.bpmn     # Visual workflow design

Reference Implementation

For a practical example, see our reference implementation: https://github.com/pyck-ai/pyck-customer-template

Best Practices

1. Idempotency: Ensure all activities are idempotent for reliable retries

2. Deterministic Workflows: Keep workflows deterministic by moving non-deterministic code to activities

3. Error Handling: Use appropriate retry policies for different failure types

4. Testing: Write comprehensive tests for workflows and activities

5. Monitoring: Set up proper monitoring and alerting using Temporal metrics

6. Versioning: Use versioning for workflow changes to handle in-flight workflows