2.0.12

aic_utils

aic_utils is a Python toolkit for managing JD Power AIC (Analytics Insights Console) pipelines and synchronizing them with GitLab under a strict, audit-friendly Git flow. It also bundles a few supporting utilities (dataset placeholder conversion, Slack logging, repository bootstrapping).

• PyPI: https://pypi.org/project/aic-utils/
• Source: https://github.com/dylandoyle11/aic_utils
• Current version (source tree): 2.0.2
• Author: Dylan Doyle

⚠️ The PyPI release may lag this repository. This README documents the current source tree. If a method described here is not present in the version you pip install-ed, install from source (pip install git+https://github.com/dylandoyle11/aic_utils) or pin to the version that matches.

---

Table of Contents

1. Installation
2. Package Overview
3. AIC — AIC REST API wrapper
4. GitLabManager — Git ↔ AIC sync engine
5. BranchWorkspaceMapper — branch ↔ workspace mapping
6. GitLabRepositoryInitializer — bootstrap a new repo
7. DatasetConverter — placeholder rewriting
8. SlackLogger — Slack-backed Python logger
9. Day-to-day workflows
10. Best practices & troubleshooting

---

1. Installation

pip install aic_utils

Or pin to the latest source:

pip install git+https://github.com/dylandoyle11/aic_utils.git

Runtime dependencies (from setup.py):

• requests >= 2.25.0
• PyYAML >= 5.4.0
• pandas >= 1.3.0
• slack_sdk >= 3.0.0

Python: 3.8+

The GitLabManager._send_email helper imports pyspark_utils lazily; that package is optional and only required when email_recipients is configured.

---

2. Package Overview

from aic_utils import (
    AIC,                           # AIC REST wrapper
    GitLabManager,                 # GitLab sync / Git-flow enforcement
    GitLabRepositoryInitializer,   # One-time repo bootstrap
    DatasetConverter,              # Rewrite SQL/JSON placeholders → dataset:table refs
    SlackLogger,                   # Pipe Python logging / print() to Slack
)

The package is organized so that one AIC instance + one GitLabManager instance together represent one workspace ↔ one branch at a time. Each environment (Prod / QA / Dev / sandbox) uses its own pair.

High-level architecture

┌──────────────┐        ┌────────────────────┐        ┌──────────────┐
│ AIC workspace│  ←──── │  AIC (REST wrapper)│ ────→  │ pipeline JSON│
│  (cloud)     │        └─────────┬──────────┘        │  in memory   │
└──────────────┘                  │                   └──────┬───────┘
                                  ▼                          ▼
                       ┌────────────────────┐       ┌──────────────────┐
                       │   GitLabManager    │ ────→ │  GitLab repo     │
                       │  (push / deploy /  │       │  config/*.json   │
                       │   branch / MR)     │       │  code/<pipe>/*   │
                       └─────────┬──────────┘       └──────────────────┘
                                 │
                                 ▼
                       ┌────────────────────┐
                       │ BranchWorkspace-   │  ← branch_to_workspace.yaml
                       │ Mapper             │
                       └────────────────────┘

---

3. AIC — AIC REST API wrapper

aic_utils/aic.py — class AIC

A thin REST client around the AIC core API (/apis/core/v1). Resolves project + workspace, lists pipelines, fetches/upserts pipeline configs, manages datasets and backups.

3.1 Constructor

AIC(api_key: str,
    project: str,
    workspace: str,
    pipelines: list[str] = [],
    qa: bool = False)

Resolved attributes

3.2 Discovery

3.3 Pipeline upsert (CRUD)

config shape:

{
  'name': '<pipeline title>',
  'jobConfig': { 'stages': [...], 'variables': [...], 'sourceStage': {...}, 'sinkStage': {...}, ... },
  'id': '<existing $id, or None for create>',
}

3.4 Workspace artifacts

3.5 Under the hood

• Concurrency: pop_pipeline_config uses ThreadPoolExecutor.
• Error handling: HTTP errors print status + body and continue where reasonable.
• Logging: print() for visibility (you can redirect to a logger via SlackLogger).

---

4. GitLabManager — Git ↔ AIC sync engine

aic_utils/gitlabmanager.py — class GitLabManager

Wraps the GitLab v4 REST API and binds it to an AIC instance. Enforces a strict 1:1 branch ↔ workspace contract, drives push/deploy passes, manages feature/release/hotfix Git flow, and can bootstrap a brand-new repo.

4.1 Branch role configuration

Class-level constants control which branches play which Git-flow role:

GitLabManager.branch_roles = {
    "production":  "main",
    "staging":     "staging",
    "development": "dev",
}

If your repo uses different names (e.g. develop instead of dev), set this before instantiating any managers:

GitLabManager.branch_roles = {"production": "main", "staging": "qa", "development": "develop"}

The defaults — main / staging / dev — are what GitLabRepositoryInitializer will scaffold.

4.2 Constructor

GitLabManager(
    aic_instance,                                   # AIC
    gitlab_token,                                   # str
    gitlab_namespace,                               # e.g. "pin/pin-analytics"
    repo_folder,                                    # e.g. "pin-fusion-2.0"
    gitlab_base_url: str = "https://git.autodatacorp.org/api/v4",
    use_hash_comparison: bool = True,
    email_recipients: list = None,
    email_sender: str = None,
    mapping_file: str = "branch_to_workspace.yaml",
)

What __init__ does, in order

1. Builds GitLab auth headers (Private-Token).
2. Resolves the project ({namespace}/{repo_folder}); raises a clear RuntimeError on 404.
3. Caches the repo's default_branch.
4. Loads the branch_to_workspace.yaml from the default branch into a BranchWorkspaceMapper.
5. Picks the working branch by reverse-mapping aic.workspace. If no mapping exists, slugs the workspace name (PIN FUSION 2.0 → pin-fusion-2-0).
6. Calls ensure_branch_exists on that branch (off default_branch).
7. Records _allowed_push = {slug, mapped_branch} plus _hotfix_prefixes = ['hotfix/'].
8. Calls check_token_expiration() (warns if ≤ 30 days).
9. Locks the instance — see §4.3.

4.3 Attribute lock (important)

After construction, __setattr__ raises if you try to mutate an attribute. The only sanctioned way to change _branch is via:

• create_feature_branch(name)
• create_release_branch(name)
• create_hotfix_branch(name)

…all of which use the internal _set_branch() helper. mgr.branch is exposed as a read-only property.

Practical implication: don’t reuse a GitLabManager across workspaces. Build a new one per environment.

4.4 Repository & branch utilities

4.5 File operations

4.6 Pushing AIC → Git

mgr.push_to_git()

• Production guard: if the current branch equals default_branch, prompts (y/n) before continuing.
• Branch guard: branch must be in _allowed_push or start with hotfix/.
• Pass 1: writes config/{name}.json for every entry in aic.pipeline_configs.
• Pass 2: for each successful pipeline, calls _extract_and_push_code to:
  • dump PYSPARK task scripts to code/{pipeline}/{task_id}.py,
  • find triple-quoted SQL blocks inside those scripts (heuristic: first word ∈ select|insert|update|delete|with|create|drop|alter|merge) and write each as code/{pipeline}/{task_id}_{n}.sql,
  • dump SQL task query/script/sql text to code/{pipeline}/{task_id}.sql.
• Email: if email_recipients is set, sends an HTML push summary via pyspark_utils.send_email.

4.7 Deploying Git → AIC

mgr.deploy_pipelines()

• Reverse-maps the current branch to a workspace and refuses to deploy if it doesn’t match aic.workspace.
• For each pipeline in aic.pipeline_configs:
  • Fetches config/{name}.json from the current branch.
  • Compares MD5 of normalized JSON (when use_hash_comparison=True); skips unchanged.
  • Calls aic.write_config_to_pipeline(...) for changed pipelines.
• Emails an HTML deploy summary if configured.

The earlier force_sync=True flag is no longer present — current behavior is hash-compare or always-write controlled by use_hash_comparison at construction time.

4.8 Git-flow branch creation

All three slugify the input (replace anything outside [A-Za-z0-9\-_/] with -) and switch the manager's branch on success.

4.9 Merge requests

mgr.create_merge_request(
    title:           str,
    description:     str,
    target_branch:   str | None = None,   # auto-detected if omitted
    assignee_ids:    list | None = None,
)

• Refuses unless the current branch starts with feature/, release/, or hotfix/.
• Auto-target rules when target_branch is omitted:
  • feature/* → staging
  • release/*, hotfix/*, or staging → main
  • else → main
• After opening the MR, switches the manager back to the original branch so the workflow stays predictable. The MR iid is logged.

Note: the auto-target string literals are hardcoded to "main" / "staging" — if you customize branch_roles, pass target_branch explicitly.

4.10 Repo bootstrap & wipe (classmethods)

GitLabManager.initialize_repository(
    gitlab_token, gitlab_namespace, repo_folder,
    gitlab_base_url="https://git.autodatacorp.org/api/v4",
)

Builds a minimally-initialized manager via _setup_for_initialization and hands off to GitLabRepositoryInitializer (see §6). Returns True/False.

GitLabManager.wipe_repository(
    gitlab_token, gitlab_namespace, repo_folder,
    gitlab_base_url="https://git.autodatacorp.org/api/v4",
    keep_branches=None,    # default ['main']
)

Destructive. Prompts (y/n) first, then:

1. Lists every branch.
2. For each kept branch, deletes every blob via per-file DELETE.
3. Deletes every other branch (skips main/master defensively, removes branch protections first).

4.11 Email helpers

_send_email(subject, body, cc=None, bcc=None, attachments=None, testing=True) _format_push_summary(branch, results) → (subject, html_body) _format_deploy_summary(branch, results) → (subject, html_body)

results is a list of (pipeline_name, status, message) tuples, rendered as an HTML table.

---

5. BranchWorkspaceMapper — branch ↔ workspace mapping

Defined in gitlabmanager.py. Constructed automatically by GitLabManager.__init__ from branch_to_workspace.yaml on the default branch.

YAML format:

mapping:
  main:      PIN FUSION 2.0
  staging:   PIN FUSION 2.0 QA
  dev:       PIN FUSION 2.0 DEV
  intg/*:    PIN FUSION 2.0 INTG     # wildcard prefix

API:

lookup looks up the mapping keys literally — its built-in feature/* fallback is keyed off develop (not dev). If you renamed the dev branch, add an explicit mapping for the dev branch and rely on exact/prefix rules rather than the built-in.

---

6. GitLabRepositoryInitializer — bootstrap a new repo

aic_utils/gitlab_init.py — class GitLabRepositoryInitializer

Usually invoked indirectly via GitLabManager.initialize_repository(...). It performs a one-time setup of a freshly-created empty GitLab project.

initialize_repository() runs, in order:

1. _enforce_main_branch() — if the default branch isn’t branch_roles["production"] (default main), creates it from the existing default, sets it as default, protects it (push/merge level 30 = maintainer), and deletes the original.
2. _create_initial_branches() — creates staging and dev (or whatever branch_roles specifies) off the default branch.
3. _create_initial_files() — pushes onto the default branch:
   • branch_to_workspace.yaml with placeholder workspace names,
   • SECURITY.md documenting JDP scanners and remediation timelines,
   • config/.gitkeep,
   • code/.gitkeep.
4. _setup_security_scanning() — writes a .gitlab-ci.yml that includes the corporate templates for the matching group (alg / pin / ucg → otherwise corp) and wires SAST, dependency scanning, secret detection, code quality, plus a manual deploy_pipelines job.
5. _enable_cicd_features() — PUT /projects/{id} to enable builds/MRs/security & compliance and disable wiki/snippets/registry.
6. _setup_branch_protection() — protects production, staging, and development (push/merge level 30).
7. _generate_readme() — derives project + company names from the namespace and writes a tailored README.md (the same one you’re reading describes how to use it).

All steps are idempotent enough to re-run safely (existing branches, files, and protections return 409 and are logged as warnings).

---

7. DatasetConverter — placeholder rewriting

aic_utils/dataset_converter.py — class DatasetConverter

Purpose: rewrite bare table names in pipeline configs into AIC dataset-aware placeholders (${DATASET.TABLE}), then push the rewritten config to a new interactive AIC branch named CONVERTED_<YYYYMMDDHHMMSS>. Intended for one-shot migration of legacy pipelines that hard-coded table names.

7.1 Constructor

DatasetConverter(api_key, project, workspace,
                 target_workspace=None, qa=False)

• Resolves project and workspace exactly like AIC.
• If target_workspace is supplied, the table catalog (used as the lookup dictionary) is built from that workspace; otherwise from the source workspace.
• Loads self.tables as a pandas.DataFrame of {dataset_name, dataset_id, table_name} (parallel fetch via ThreadPoolExecutor).

7.2 Methods

---

8. SlackLogger — Slack-backed Python logger

aic_utils/slack_logger.py — class SlackLogger

Bridges Python’s logging (and optionally print) to a Slack channel via slack_sdk. Includes a re-entrancy guard so messages emitted while sending don’t loop.

8.1 API

SlackLogger(token: str, channel: str)
    .send_message(text)                            # post a single message
    .SlackLoggerHandler                            # nested logging.Handler

SlackLogger.redirect_print_to_logger(logger)       # monkey-patches builtins.print
SlackLogger.create_logger(slack_token,
                          slack_channel='C07DYFK5SE8',
                          redirect_print=True)     # returns a configured logging.Logger

create_logger:

1. Instantiates SlackLogger.
2. Builds a SlackLoggerHandler wired to it.
3. Returns a logging.Logger named 'SlackLogger' at INFO level with format %(asctime)s - %(levelname)s - %(message)s.
4. If redirect_print=True, re-points builtins.print at logger.info.

The default channel ID 'C07DYFK5SE8' is JDP-internal — pass your own.

8.2 Example

from aic_utils import SlackLogger

logger = SlackLogger.create_logger(
    slack_token="xoxb-…",
    slack_channel="C0123456789",
    redirect_print=True,   # any subsequent print() lands in Slack
)
logger.info("Pipeline run started")

---

9. Day-to-day workflows

The branch role names below assume the defaults production="main", staging="staging", development="dev". Substitute as needed.

9.1 First-time repository setup

from aic_utils import GitLabManager

GitLabManager.initialize_repository(
    gitlab_token="<GIT_TOKEN>",
    gitlab_namespace="pin/pin-analytics",
    repo_folder="pin-fusion-2.0",
)
# Then edit branch_to_workspace.yaml with real workspace names.

9.2 Build the manager pair

from aic_utils import AIC, GitLabManager

aic_prod = AIC(
    api_key="<API_KEY>",
    project="dylan.doyle@jdpa.com",
    workspace="PIN FUSION 2.0",
    pipelines=["*"],          # or a curated list
)

mgr = GitLabManager(
    aic_instance=aic_prod,
    gitlab_token="<GIT_TOKEN>",
    gitlab_namespace="pin/pin-analytics",
    repo_folder="pin-fusion-2.0",
)

9.3 Sync AIC → Git on the dev environment

aic_dev = AIC(api_key=..., project=..., workspace="PIN FUSION 2.0 DEV", pipelines=[...])
mgr_dev = GitLabManager(aic_instance=aic_dev, ...)
mgr_dev.push_to_git()

Restrictions:

• push_to_git on main requires interactive y confirmation.
• Push is rejected if the current branch isn’t in _allowed_push (slug + mapped branch) and isn't a hotfix/*.

9.4 Promote dev → staging via a feature MR

mgr_dev.create_feature_branch("PINOPS-1831")          # cuts feature/PINOPS-1831 from dev
mgr_dev.create_merge_request(
    title="Demographics module",
    description="QA push",
    target_branch="staging",                           # pass explicitly if you customized roles
)

# After the MR is merged, deploy on the QA workspace:
aic_qa = AIC(api_key=..., workspace="PIN FUSION 2.0 QA", pipelines=[...])
mgr_qa = GitLabManager(aic_instance=aic_qa, ...)
mgr_qa.deploy_pipelines()

9.5 Promote staging → prod via a release MR

mgr_qa.push_to_git()                                  # capture latest QA state
mgr_qa.create_release_branch("v2.1.0")                # cuts release/v2.1.0 from staging,
                                                      # prunes pipelines not in aic.pipeline_configs
mgr_qa.create_merge_request(
    title="Release v2.1.0",
    description="QA-approved",
    target_branch="main",
)

# After merge:
mgr_prod = GitLabManager(aic_instance=aic_prod, ...)
mgr_prod.deploy_pipelines()

9.6 Production hotfix

The order is inverted from feature/release: the fix is applied in AIC prod first, then captured in Git.

# 1. Apply the fix manually in AIC prod workspace.

# 2. Build a fresh prod AIC + manager (so it sees the new config).
aic_prod = AIC(api_key=..., workspace="PIN FUSION 2.0", pipelines=["BROKEN_PIPELINE"])
mgr_prod = GitLabManager(aic_instance=aic_prod, ...)

# 3. Cut hotfix off main and push the AIC state into the hotfix branch.
mgr_prod.create_hotfix_branch("PINOPS-1831")
mgr_prod.push_to_git()

# 4. Open MR back into main for the audit trail.
mgr_prod.create_merge_request(
    title="Hotfix PINOPS-1831",
    description="Emergency fix for streaming timeout",
    target_branch="main",
)

9.7 Convert legacy placeholders

from aic_utils import DatasetConverter

dc = DatasetConverter(
    api_key="<API_KEY>",
    project="dylan.doyle@jdpa.com",
    workspace="LEGACY",
    target_workspace="PIN FUSION 2.0",   # use this workspace's catalog as the lookup
)
dc.convert(["LEGACY_PIPELINE_A", "LEGACY_PIPELINE_B"])
# Each pipeline now has a CONVERTED_<timestamp> AIC branch with rewritten placeholders.

9.8 Stream logs/print to Slack

from aic_utils import SlackLogger
SlackLogger.create_logger("<SLACK_TOKEN>", "<CHANNEL_ID>", redirect_print=True)
print("from now on, this lands in Slack")

---

10. Best practices & troubleshooting

Operating principles

1. One AIC + one GitLabManager per environment. The manager locks itself after init.
2. Promotions go through MRs. main / staging / dev should be protected; only feature/*, release/*, hotfix/* cut from them.
3. Hash compare is on by default. Set use_hash_comparison=False if you want every deploy to write regardless of diff.
4. Keep branch_to_workspace.yaml truthful. The mapper reads it from the default branch only.
5. branch_roles is class-level. Setting it after construction does not retroactively rebuild allowed-push sets.

Common errors

Contact

• GitHub: https://github.com/dylandoyle11/aic_utils
• Maintainer: Dylan Doyle