Skip to content

Introduction to ConnectX

What is ETL?

Extract/Transform/Load (ETL) is the process of collecting data from one or more sources, processing/transforming it, and then loading it into one or more destinations.

What is ConnectX?

ConnectX is a proprietary ETL framework fully developed and maintained by Impresoft Engage.

  • Enables fast and effective integration of different data sources
  • Designed to handle large volumes of data with a parallel execution engine
  • Built to integrate with any technology
  • Ready to connect with widely used business applications

Built on

  • Python: a widely adopted programming language known for its performance and versatility.
  • Polars: a high-performance data manipulation library, considered one of the fastest solutions for data processing.

Available Connectors

  • Generic/File: CSV, Excel, JSON, XML, SFTP, SMTP
  • Web Services: REST API, SOAP API
  • Databases: Oracle, MS SQL Server, MySQL, MariaDB, PostgreSQL, SAP HANA, MS Dataverse
  • Applications: SugarCRM, Salesforce, Dynamics 365 CRM, Hubspot, Monday.com, Sage X3, Starnet, Zoho

Developing on ConnectX

Development takes place in a local environment using PyCharm, based on a non-modifiable core framework shared across all ETL projects for a given major release.

Features provided by Impresoft Engage

  • Orchestration of flows
  • Log management
  • Error handling
  • Field mapping for data loading
  • GET/LOAD DATA calls to databases and applications (MySQL, SugarCRM, …)

Features provided by Polars

  • GET/LOAD DATA calls to base connectors (e.g., JSON)
  • Data manipulation, deduplication, and transformation

Security Measures

For all aspects related to security measures implemented in ConnectX, with specific reference to flow execution via AWS Batch, please refer to the dedicated document:
IE | ConnectX - Technical Security Measures

Versioning

ConnectX follows the format:

  • major_release
    Significant changes that may break backward compatibility.
    Example: upgrading from 3.2 to 4.0 may require adjustments in existing configuration or jobs.

  • minor_release
    Introduces new features while maintaining backward compatibility.
    Example: users of 4.0 can upgrade to 4.1 without modifying existing jobs.

  • build_release
    Bugfix version within a specific minor release.
    Example: 4.1.2 contains only fixes compared to 4.1.1, without new features.