Overview
  • Updated on 23 Oct 2019
  • 1 minute to read
  • Contributors
  • Print
  • Share
  • Dark
    Light

Overview

  • Print
  • Share
  • Dark
    Light

Macrometa Pipelines automate the flow of data continuously and in realtime between systems possibly transforming the data along the way. Broadly the pipelines service provide following capabilities:

  • Real-time operations on data like filters, transforms, enrichment, DAG workflowsetc.
  • Event driven orchestration for serverless functions & microservices.
  • Connectors to glue a wide range of sources and sinks together.
  • Supports multiple topologies i.e., fan-in, fan_out, fan_out_sequential, greedy, round_robin and try.
  • Config driven pipelines with no code.
  • Automated provisioning of C8 collections & streams in the pipeline.

Architecture

Pipelines architecture is fairly simple to understand.

pipeline_architecture.png

The main sections that make up a pipeline are inputs, processors and outputs.

Following are few other ways one can configure the pipelines:

pipeline_patterns.png

Create Pipeline

Pipeline Configuration

Say we want to filter in real-time, all documents with status == processed from a collection (ordersCollection) and write to a stream (shipmentStream). Following is the configuration spec for this pipeline:

{
    "input": {
        "c8db": {
            "local": false,
            "name": "ordersCollection"
        }
    },
    "pipeline": {
        "processors": [
            {
                "type": "filter",
                "filter": {
                    "type": "jmespath",
                    "jmespath": {
                        "part": 0,
                        "query": "status == 'processed'"
                    }
                }
            }
        ]
    },
    "output": {
        "c8streams": {
            "local": false,
            "name": "shipmentStream"
        }
    }
}
Note
The ordersCollection and shipmentStream are automatically created by pipeline service if they do not exist.

Now login to your account via browser and create a pipeline as follows:

create_pipeline.png

Manage Pipeline

From GUI, you can update and delete pipelines as follows:

update_pipelines.png

REST API

Following are the available REST API for managing pipelines.

pipelines_rest_api.png

Next steps

Rest of the articles in this chapter cover in more detail various aspects of the pipeline i.e.,

Was this article helpful?