Skip to content

Docker and AWS Batch

Docker (for customers On-Premise or your local PC)

⚠ī¸ If docker is not installed, contact SYS.

Download Image

ℹī¸ You only need to do this the first time. If you have already pulled this image version on your machine, you can skip this step.

  1. Authenticate to the repository and pull the official images:

🔐 You can find the credentials on Confluence.

export AWS_ACCESS_KEY_ID=EXAMPLE
export AWS_SECRET_ACCESS_KEY=EXAMPLE
aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 288817675132.dkr.ecr.eu-west-1.amazonaws.com
  1. Download the ConnectX image:
docker pull 288817675132.dkr.ecr.eu-west-1.amazonaws.com/connectx:<VERSION>

Running the ETL

  1. Navigate to the folder containing the ETL.
  2. Run (⚠ī¸ use sudo on servers!):
docker run --rm -v $(pwd):/usr/src/app 288817675132.dkr.ecr.eu-west-1.amazonaws.com/connectx:<VERSION> python3 main.py -e dev

AWS Batch

Manual Execution via AWS Batch (from portal-cron)

  • Note: The correct command can be found in the client's ETL technical sheet.
aws batch submit-job --region eu-west-1 \
  --job-name <job-name> \
  --job-queue <job-queue> \
  --job-definition <job-definition> \
  --container-overrides 'command=["python3", "main.py", "-e", "<ENV>"]'

Modifying AWS Batch Execution Parameters

  • Default resources: ConnectX jobs normally run with 1 vCPU and 2 GB RAM.
  • When to increase: For large initial loads (T0) or heavy jobs, you may need more CPU, memory, or a longer timeout.

Example: Submitting a T0 job with 2 vCPU, 8 GB RAM, 8-hour timeout:

aws batch submit-job \
  --region eu-west-1 \
  --job-name <job-name>_t0 \
  --job-queue <job-queue> \
  --job-definition <job-definition> \
  --timeout '{"attemptDurationSeconds": 28800}' \
  --container-overrides '{
    "command": ["python3", "main.py", "-e", "prod"],
    "resourceRequirements": [
      {"type": "VCPU", "value": "2"},     # Number of CPUs for this job
      {"type": "MEMORY", "value": "8192"} # Memory in MB
    ]
  }'

Parameter explanations:

Parameter Meaning Default How to change
VCPU Number of CPUs the job uses 1 Increase if job is heavy
MEMORY RAM in MB 2048 Increase if job needs more memory
attemptDurationSeconds Maximum time job can run 3600 Increase for long-running jobs (like T0)
command Command to run inside the container - Change if you need a different Python script

⚠ī¸ Important:

  • Memory and timeout can be adjusted independently, depending on your job needs. Remove the unneeded parts of the command.
  • To increase vCPUs beyond the default, contact SYS.

AWS Batch Console

Access AWS Console

  1. Navigate to https://impresoftengage.awsapps.com/start
  2. Form the menu, choose the AWS account "Portal".
  3. Click on "DeveloperDefaultAccess".
  1. Search "AWS Batch" in the top bar and select the service.
  2. In the left menu, select "Jobs" (Italian: "Processi").
  3. Disable "Advanced search".
  4. Choose the "Job queue" corresponding to the client.
  5. Selecting a job shows additional info. Click "Log stream name" to view ConnectX output.