ETL Log Management

connectx.etllog

EtlJob

__init__

Represents an ETL job consisting of multiple steps.

Handles creation of logs, job start/end, and upload limits.

Parameters:

Name	Type	Description	Default
name	str	str, name of the ETL job.	required
steps	list	list of str, optional, names of steps to initialize logs for.	None
conn		object, optional, connection object for logging/job saving.	None

create_log

Create a log for a specific step or return existing log.

Parameters:

Name	Type	Description	Default
name	str	str, name of the log/step.	required

Returns:

Type	Description
	EtlLog, the log instance.

end

Mark the ETL job as ended, saving the last start date.

Returns:

Type	Description
	None

get

Retrieve a value from the job's data dictionary.

Parameters:

Name	Type	Description	Default
key		str, key to retrieve.	required
default		any, value to return if key is missing.	None

Returns:

Type	Description
	any, value stored in the job data.

set_upload_limit

Set an upload limit for all logs created under this job.

Parameters:

Name	Type	Description	Default
upload_limit	int | None	int or None, maximum number of records/files to upload.	None

Returns:

Type	Description
	EtlJob, self.

start

Start the ETL job and create a log for it.

Raises EtlJobNotEnabled if the job is disabled.

Returns:

Type	Description
	EtlLog, the log instance for the job.

EtlJobNotEnabled

Bases: Exception

Exception raised when an ETL job is disabled.

EtlLog

Represents a log entry for an ETL job step.

Allows tracking status, number of processed records, and errors.

Parameters:

Name	Type	Description	Default
name	str	str, name of the log step.	required
job		EtlJob or EtlLog, the parent job or log instance.	required
id	str	str, optional, ID of an existing log entry.	None

abort

Set the log status to 'abort', capturing exception info if present.

Parameters:

Name	Type	Description	Default
description	str	str, optional, description or error message.	''

Returns:

Type	Description
	EtlLog, self.

create_log

Create a new EtlLog instance as a sub-step.

Parameters:

Name	Type	Description	Default
name	str	str, name of the new log.	required

Returns:

Type	Description
	EtlLog, the newly created log instance.

error

Set the log status to 'error' and optionally provide files and row data.

Parameters:

Name	Type	Description	Default
files	list	list, optional, files associated with this step.	None
rows	list	list, optional, rows associated with this step.	None

Returns:

Type	Description
	EtlLog, self.

get

Get a value from the log data.

Parameters:

Name	Type	Description	Default
key		str, the key to retrieve.	required
default		any, value to return if key is not found. Default is None.	None

Returns:

Type	Description
	any, value stored in the log for the given key.

set_upload_limit

Set the upload limit for this log instance.

Parameters:

Name	Type	Description	Default
upload_limit	int | None	int or None, maximum number of records/files to upload.	required

Returns:

Type	Description
	EtlLog, self.

start

Set the log status to 'progress' and optionally set total records.

Parameters:

Name	Type	Description	Default
num_records	int	int, optional, total number of records.	None

Returns:

Type	Description
	EtlLog, self.

success

Set the log status to 'success' and optionally provide files and row data.

Parameters:

Name	Type	Description	Default
files	list	list, optional, files associated with this step.	None
rows	list	list, optional, rows associated with this step.	None

Returns:

Type	Description
	EtlLog, self.

update

Update the log entry with the current status and statistics.

Parameters:

Name	Type	Description	Default
status	EtlLogStatus	str, status of the log ('queued', 'progress', 'success', 'error', 'abort').	None
num_records	int	int, total number of records.	None
processed_records	int	int, number of processed records.	None
success_records	int	int, number of successful records.	None
fail_records	int	int, number of failed records.	None
description	str	str, optional, description or error message.	None
start_date	str	str, optional, start timestamp.	None
end_date	str	str, optional, end timestamp.	None
files	list	list, optional, list of file paths associated with the log.	None
rows	list	list, optional, list of row data associated with the log.	None

Returns:

Type	Description
	EtlLog, self.

Task

Represents a single executable task with arguments and keyword arguments.

This class wraps a callable target and allows it to be executed later via the run method. It also introspects the target's signature to track parameter types.

__init__

Initializes a Task instance.

Parameters:

Name	Type	Description	Default
target	callable	The callable function or method to execute.	required
args	tuple	Positional arguments to pass to the target when executed.	()
kwargs	dict	Keyword arguments to pass to the target when executed.	{}

run

Executes the target callable with the stored arguments and any additional arguments provided.

Combines the arguments passed at Task creation with arguments passed to this method. Catches exceptions and logs them as critical.

Parameters:

Name	Type	Description	Default
args	tuple	Additional positional arguments for the target.	()
kwargs	dict	Additional keyword arguments for the target.	{}

Returns:

Type	Description
Any	The return value of the target callable, or the exception instance if an error occurred.