Moving from CloverDX

A CloverDX graph becomes a Trinium pipeline and a jobflow becomes a Flow. This guide maps the components and ideas, and says where the two differ.

The ideas

In CloverDX	In Trinium
Graph	Pipeline: nodes wired together on a canvas. Every node runs at once and rows stream between them.
Component	Node.
Edge	Wire.
Metadata	Columns worked out as you build: every node shows its output columns without a run, so there is no metadata to define by hand.
CTL2	One SQL expression language for every rule and formula.
Graph parameters	Parameters: values supplied when a pipeline runs, by a person, a schedule or a Flow step.
Jobflow	Flow: steps that run whole pipelines in order or side by side, branching on how each one ended.
Subgraph	No equivalent: there are no reusable parts of a pipeline. A shared pipeline run from several Flows covers reuse at the job level.
Server schedule	Schedule.
File event listener	A schedule that runs when files arrive in S3, Azure Blob Storage, Google Cloud Storage or SFTP.

Component by component

Readers

Theirs	Trinium	Notes
FlatFileReader, UniversalDataReader	CSV Files	Delimited text. Fixed-width text is not read.
SpreadsheetDataReader	Excel Files
JSONReader, JSONExtract	JSON Files, JSON	JSON Parse reads JSON held in a column.
XMLExtract, XMLReader, XMLXPathReader	XML Files, XML	Records are found by path; XPath expressions are not supported.
ParquetReader	Parquet Files
DatabaseReader	Postgres input, MySQL input, SQL Server input, Redshift input, Snowflake input, BigQuery input, Databricks input, ClickHouse input, Trino / Starburst input	One input node per database or warehouse.
MongoDBReader	MongoDB input
SalesforceReader, SalesforceBulkReader	Salesforce input
DataGenerator	Generate Rows, Text Input
KafkaReader, JMSReader, EmailReader	No node	Not supported: Trinium runs batch pipelines over tables and files, not message streams.

Writers

Theirs	Trinium	Notes
FlatFileWriter, UniversalDataWriter	CSV File
SpreadsheetDataWriter	Excel Report	Writes a new one-sheet workbook.
JSONWriter	JSON File
ParquetWriter	Parquet File
XMLWriter	No node	No equivalent: Trinium does not write XML files.
DatabaseWriter and the bulk writers	Postgres output, MySQL output, SQL Server output, Redshift output, Snowflake output, BigQuery output, Databricks output, ClickHouse output, Trino / Starburst output	One output node per database or warehouse, each loading in the database's own way.
MongoDBWriter	MongoDB output
SalesforceWriter, SalesforceBulkWriter	Salesforce output
EmailSender	Send Email
Trash	No node	No node needed: rows on an output with nothing wired to it go nowhere.

Transformers

Theirs	Trinium	Notes
Map	Formula, Manage Columns	Formula computes columns; Manage Columns keeps, renames and retypes them.
Filter	Filter
Partition	Route	Up to ten named branches, first match wins, plus one for unmatched rows.
Dedup	Unique
ExtSort, FastSort, SortWithinGroups	Sort
Aggregate	Summarize
Rollup	Summarize, Formula Multi-Row	Custom per-group code has no equivalent; Summarize and window functions cover the usual cases.
Denormalizer, Pivot	Cross Tab
Normalizer, MetaPivot	Transpose, Text to Columns
Concatenate, SimpleGather, Merge	Union
SimpleCopy	No node	No node needed: wire one output to several inputs and every wire gets every row.
DataSampler	Sample
DataIntersection	Join, Data Compare	Join keeps left-only, matched and right-only rows on outputs of their own.

Joiners

Theirs	Trinium	Notes
ExtHashJoin, ExtMergeJoin, RelationalJoin	Join, Join Multiple	Trinium chooses how to join; there is no hash or merge choice to make.
LookupJoin	Join
DBJoin	Database Lookup	Sends a stream's keys to the database table.
CrossJoin	Append Fields

Data Quality

Theirs	Trinium	Notes
Validator	Data Validation	Rows that break a rule leave on an output of their own, naming the rule.
ProfilerProbe	Data Profile

Others

Theirs	Trinium	Notes
HTTPConnector, RESTConnector	REST API (Read), API Lookup, REST API (Write)
WebServiceClient	API Lookup, XML	A SOAP call is an API Lookup with an XML body, its response parsed with XML Parse.
DBExecute	No node	Pre-SQL and post-SQL statements on the database nodes. There is no standalone SQL node.

File Operations

Theirs	Trinium	Notes
ListFiles	List Files
CopyFiles, MoveFiles, DeleteFiles	File Action	One File Action copies, moves or deletes files on S3, Azure Blob, GCS, SFTP or a network share, before the data is processed or after the run succeeds or fails. It names no folder of its own: the input, List Files or File from URL wired to its Files anchor hands it the files.

Job Control

Theirs	Trinium	Notes
ExecuteGraph	No node	A Run pipeline step in a Flow.
Condition	No node	A Flow wire's condition: on success, on failure or on complete. Inside a pipeline, Filter and Route branch on values.
Barrier	No node	A Flow step with several incoming wires waits for all of them.
Loop, Sleep	No node	Not supported: a Flow does not loop and has no wait step.
Subgraph	No node	No reusable part of a pipeline. A shared pipeline run from several Flows, each step handing it its own parameters, covers reuse at the job level.

What does not carry over

Custom Java components and transforms. Trinium has no Java extension point. Put the logic behind an HTTPS endpoint of your own, and API Batch Lookup sends it the rows many at a time.
Message streams and EDI: Kafka, JMS, HL7, EDIFACT and X12 are not read or written.
Loops and waits in a jobflow, and passing a value one step produced to a later step. Hand data between pipelines through a table or a folder.
Fixed-width files and XML output.

Moving a graph

Create a connection for each system the graph reads or writes.
Rebuild the components with the table above. There is no metadata to declare: each node's columns follow from its settings and the node before it.
Rewrite CTL transforms as SQL expressions in Formula, Filter and the other nodes that take a rule.
Press Test for a 50-row sample, deploy to Staging, and compare a full run with the CloverDX output using Data Compare.
Rebuild each jobflow as a Flow, one Run pipeline step per graph.

CloverDX component and job names are as of CloverDX 7.6.1, from CloverDX's public documentation: the component reference, job types, subgraphs and listeners. CloverDX is a trademark of its owner. Trinium is not affiliated with CloverDX.