Skip to content

Docker and AWS Batch

Download Image (Local)

ℹ️ Only if you do not have the ConnectX image locally.

  1. Authenticate to the repository and pull the official images:
export AWS_ACCESS_KEY_ID=AKIAUGPW77N6BYPK6R4R
export AWS_SECRET_ACCESS_KEY=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
aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 288817675132.dkr.ecr.eu-west-1.amazonaws.com
  1. Download the ConnectX image:
docker pull 288817675132.dkr.ecr.eu-west-1.amazonaws.com/connectx:<VERSION>

Running the ETL (Local or On-Premise)

  1. Navigate to the folder containing the ETL.
  2. Run (⚠️ use sudo on servers!):
docker run --rm -v $(pwd):/usr/src/app 288817675132.dkr.ecr.eu-west-1.amazonaws.com/connectx:<VERSION> python3 main.py -e dev

Manual Execution via AWS Batch (AWS Portal-Cron)

  • Note: The correct command can be found in the client's ETL technical sheet.
aws batch submit-job --region eu-west-1 \
  --job-name <job-name> \
  --job-queue <job-queue> \
  --job-definition <job-definition> \
  --container-overrides 'command=["python3", "main.py", "-e", "<ENV>"]'

Modifying AWS Batch Execution Parameters

  • ConnectX jobs usually have 1 vCPU and 2 GB RAM. For large initial loads (T0), higher resources may be needed.
  • Example of submitting a job with 2 vCPU, 8192 MB RAM, and 28800 seconds timeout:
aws batch submit-job --region eu-west-1 \
  --job-name <job-name>_t0 \
  --job-queue <job-queue> \
  --job-definition <job-definition> \
  --timeout '{"attemptDurationSeconds": 28800}' \
  --container-overrides '{"command": ["python3", "main.py", "-e", "prod"], "resourceRequirements": [ {"value": "2", "type": "VCPU"}, {"value": "8192", "type": "MEMORY"}]}'

⚠️ To increase vCPUs, contact SYS.

AWS Batch Console

Prerequisites

  • Install the Chrome extension: AWS Extend Switch Roles
  • Pin the extension for visibility.
  • Click the extension → Configuration, add the following, and Save:
[Primary Account]
aws_account_id = 884493469407
role_name = OsyDevelopersRole
color = a3a3a3
region = eu-west-1

Access AWS Console

  1. Open a new tab.
  2. Click the Apps menu.
  3. Scroll down and select "Osy_Amazon_Connect" or "IE_Amazon_Connect".

AWS Console

  1. Click the extension again and select "Primary Account".
  2. Search "AWS Batch" in the top bar and select the service.
  3. In the left menu, select "Jobs" (Italian: "Processi"). Disable "Advanced search".
  4. Choose the "Job queue" corresponding to the client.
  5. Selecting a job shows additional info. Click "Log stream name" to view ConnectX output.