Skip to main content

BladePipe

BladePipe is a real-time end-to-end data replication tool, moving data between 30+ databases, message queues, search engines, caching, real-time data warehouses, data lakes and more, with ultra-low latency less than 3 seconds. It features efficiency, stability and scalability, compatibility with diverse database engines, one-stop management, enhanced security, and complex data transformation.

Functions​

BladePipe presents a visual management interface, allowing you to easily create DataJobs to realize schema migration, data migration, synchronization, verification and correction, etc. In addition, more refined and customized configurations are supported by setting parameters. Now BladePipe supports data movement from the following source DataSources to Doris:

Source DataSourceSchema MigrationData MigrationData SyncVerification & Correction
MySQL/MariaDB/AuroraMySQLYesYesYesYes
OracleYesYesYesYes
PostgreSQL/AuroraPostgreSQLYesYesYesYes
SQL ServerYesYesYesYes
KafkaNoNoYesNo
AutoMQNoNoYesNo
TiDBYesYesYesYes
HanaYesYesYesYes
PolarDB-XYesYesYesYes
info

For more functions and parameter settings, please refer to BladePipe Connections.

Installation​

Follow the instructions in Install Worker (Docker) or Install Worker (Binary) to download and install a BladePipe Worker.

Example​

Taking a MySQL instance as an example, the following part describes how to move data from MySQL to Doris.

Add DataSources​

  1. Log in to the BladePipe Cloud. Click DataSource > Add DataSource.

  2. Select MySQL and Doris as the Type respectively, and fill in the setup form accordingly. Add DataSources-1

  3. Click Test Connection. After successful connection, click Add DataSource to add the DataSource. Add DataSources-2

Create a DataJob​

  1. Click DataJob > Create DataJob.

  2. Select the source and target DataSources, and click Test Connection to ensure the connection to the source and target DataSources are both successful. Create a DataJob-1

  3. Select Incremental for DataJob Type, together with the Full Data option. Create a DataJob-2

  4. Select the tables to be replicated. Create a DataJob-3

  5. Select the columns to be replicated. Create a DataJob-4

  6. Confirm the DataJob creation.

  7. The DataJob runs automatically. BladePipe will automatically run the following DataTasks:

  • Schema Migration: The schemas of the source tables will be migrated to the target instance.
  • Full Data: All existing data of the source tables will be fully migrated to the target instance.
  • Incremental: Ongoing data changes will be continuously synchronized to the target instance. Create a DataJob-5