ETL Job Trigger Integration Guide

Overview

This guide shows you how to trigger Databricks ETL jobs in the S0PMOBAI system by publishing messages to an Azure Storage Queue. When your job completes, send a simple JSON message to our queue, and we'll automatically trigger the corresponding Databricks ETL job.

Use Case: Your source system (S0TPRXXX, S0AIDXXX, etc.) completes a data export job and needs to trigger downstream processing in Databricks.

Quick Start

1. Get access to our Azure Storage Queue
2. Send a JSON message when your job completes
3. We automatically trigger the mapped Databricks job

Prerequisites

Before you can integrate, you'll need:

• Azure Service Principal with permissions to write to our queue
• Python environment (if using our code examples)
• Job Mapping registered in our system (see Job Mapping Registration)

Azure Storage Queue Details

Queue Information by Environment

Environment	Storage Account Names	Queue Name
DEV	cutdstorpmobchat01 & cucdstorpmobchat01	job-completion-audit
TST	cuttstorpmobchat01 & cuctstorpmobchat01	job-completion-audit
STG	cutsstorpmobchat01 & cucsstorpmobchat01	job-completion-audit
PRD	cutpstorpmobchat01 & cucpstorpmobchat01	job-completion-audit

Queue URLs

Format: https://{storage-account-name}.queue.core.windows.net/{queue-name}

Example: https://cutdstorpmobchat01.queue.core.windows.net/job-completion-audit

Access Requirements

Required Azure RBAC Role

Your Service Principal needs the Storage Queue Data Message Sender role on the storage account.

To request access, submit a CCOE request.

Message Format

JSON Schema

Send messages in this format:

{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "sourceSystem": "S0TPRXXX",
  "jobName": "tpr-data-export",
  "timestamp": "2025-12-13T20:30:00Z"
}

Field Descriptions

Field	Type	Required	Description
id	string (UUID)	Yes	Unique identifier for this message. Generate a new GUID for each message.
sourceSystem	string	Yes	Your system identifier (e.g., "S0TPRXXX", "S0AIDXXX"). Must match your registered system.
jobName	string	Yes	Name of the job that completed. Must match a registered job mapping.
timestamp	string (ISO 8601)	No	When the job completed. Defaults to message receipt time if omitted.

Validation Rules

• id: Must be a valid UUID/GUID
• sourceSystem: Maximum 100 characters, must be registered in our system
• jobName: Maximum 200 characters, must be registered for your source system
• timestamp: Must be valid ISO 8601 format (e.g., 2025-12-13T20:30:00Z)

Python DBWS Code Example

Install Required Packages

pip install azure-identity azure-storage-queue

from azure.identity import ClientSecretCredential
from azure.storage.queue import QueueClient
import json
import uuid
from datetime import datetime, timezone

# Retrieve secrets from Databricks Secret Scope
TENANT_ID = dbutils.secrets.get(scope="azure-kv", key="tenant-id")
CLIENT_ID = dbutils.secrets.get(scope="azure-kv", key="sp-client-id")
CLIENT_SECRET = dbutils.secrets.get(scope="azure-kv", key="sp-client-secret")

STORAGE_ACCOUNT_NAME = "cutdstorpmobchat01"  # Change per environment
QUEUE_NAME = "job-completion-audit"

# Authenticate and create client
credential = ClientSecretCredential(
    tenant_id=TENANT_ID,
    client_id=CLIENT_ID,
    client_secret=CLIENT_SECRET
)

queue_client = QueueClient(
    account_url=f"https://{STORAGE_ACCOUNT_NAME}.queue.core.windows.net",
    queue_name=QUEUE_NAME,
    credential=credential
)

# Send notification
def notify_job_complete(source_system: str, job_name: str):
    """Send job completion notification to S0PMOBAI queue."""
    message = {
        "id": str(uuid.uuid4()),
        "sourceSystem": source_system,
        "jobName": job_name,
        "timestamp": datetime.now(timezone.utc).isoformat()
    }
    
    queue_client.send_message(json.dumps(message))
    print(f"✓ Notification sent for {source_system}/{job_name}")

# Usage at the end of your notebook
notify_job_complete("S0TPRXXX", "tpr-data-export")

Job Mapping Registration

Before you can send messages, your job mapping must be registered in our system.

What is a Job Mapping?

A job mapping connects your source system job to a specific Databricks job in our platform.

Example:

• Your Job: S0TPRXXX → tpr-data-export
• Triggers: Databricks Job → s0pmobai-tpr-data-sql-import

How to Register a New Mapping

Step 1: Gather Required Information

• Your Source System Name (e.g., "S0TPRXXX")
• Your Source Job Name (e.g., "tpr-data-export")

Step 2: Submit Request

Email mobileappchatsupport@publix.com with:

Subject: ETL Job Mapping Registration Request

Source System: S0TPRXXX
Source Job Name: tpr-data-export
Brief Overview of Job: [Brief explanation]

Step 3: We'll Process Your Request

Our team will:

1. Register the mapping in our database
2. Test in DEV environment
3. Confirm when you can start sending messages

Step 4: Start Sending Messages

Once confirmed, send messages using the registered sourceSystem and jobName values.