ADFMentor

ADFMentor

A Python package for evaluating Azure Data Factory (ADF) submissions with comprehensive AI-driven grading.

Provides accurate, fair assessment of ADF implementations including architecture, design quality, error handling, and best practices.

Features

• AI-Driven Grading: Comprehensive evaluation covering architecture, design, error handling, parameterization, and best practices
• Full Artifact Context: Sends full sanitized ADF JSON artifacts to the AI grader instead of only a compressed summary
• v1.0.0 Major Release: Complete API redesign with new evaluate_adf() method and professional reporting
• ADF JSON validation for ARM templates and repo-export style files
• Safe ZIP extraction with size and file-count limits
• Recursive discovery of ADF JSON artifacts
• Clear, actionable feedback with specific recommendations
• Secure API key management via environment variables

Installation

pip install ADFMentor

Quick Start

from ADFMentor import ADFMentor

mentor = ADFMentor(api_key="your-api-key", model_name="gemini-2.0-flash-exp")

result = mentor.evaluate_adf(
    submission_path="path/to/submission.zip",
    question="Grade this ADF assignment"
)

print(f"Score: {result['score']}/100")
print(f"Feedback:\n{result['feedback']}")
print(f"Metadata:\n{result['metadata_report']}")

custom_rubric = """
Evaluate focus on: error handling, parameterization, 
security practices, and performance optimization.
Score 0-100 with detailed feedback.
"""

result_custom = mentor.evaluate_adf(
    submission_path="path/to/submission.zip",
    prompt=custom_rubric,
    question="Assess ADF quality"
)

result_fast = mentor.evaluate_adf(
    submission_path="path/to/submission.zip",
    use_rule_based=True
)

Environment Setup

1. Get a free Google API key: https://makersuite.google.com/app/apikey
2. Copy .env.example to .env:

   cp .env.example .env
   

3. Edit .env and add your API key:

   GEMINI_API_KEY=your_actual_key_here
   

4. Important: Never commit .env to git (already protected in .gitignore)

Test files automatically load from .env.

Supported Formats

• .json - ARM templates or repo-export JSON files
• .zip - ADF export packages (may include .txt notes)
• folder path - ADF project directory
• .txt - Optional written explanations (AI grading only)

AI Grading Evaluation

The system uses Gemini AI (default: gemini-2.0-flash-exp) for comprehensive evaluation including:

• Full ADF JSON documents (sanitized for obvious credential-like secrets)

• Metadata summary for faster navigation and cross-checking

• Optional companion .txt notes included alongside the JSON artifacts

• Architecture & Design: Pipeline organization, activity flow, component relationships

• Error Handling: Exception handling, retry logic, resilience patterns

• Parameterization: Runtime flexibility, hardcoded values, reusability

• Best Practices: Naming conventions, schema definitions, performance, security

• Completeness: All required components, connections, configurations

• Code Quality: JSON structure, documentation, clarity

Returns: Score (0-100) + detailed feedback with specific improvements.

Security Validation

Safe ZIP extraction with:

• 200 file limit
• 50MB total size limit
• 10MB per-file limit
• Path traversal protection

Custom Rubrics

Define custom prompts for specialized evaluations:

from ADFMentor import ADFMentor
import os

mentor = ADFMentor(api_key=os.getenv("GEMINI_API_KEY"))

custom_prompts = {
    "pipeline": """Evaluate pipeline logic correctness (40 pts), activity selection (30 pts), 
                   performance optimization (20 pts), and completeness (10 pts). 
                   Provide concise feedback.""",
    
    "architecture": """Evaluate architecture design (40 pts), error handling (30 pts), 
                       security practices (20 pts), and documentation (10 pts). 
                       Provide concise feedback."""
}

result = mentor.evaluate_adf(
    submission_path="submission.zip",
    prompt=custom_prompts["pipeline"],
    question="Evaluate the pipeline"
)

Default AI grading works without custom prompts (recommended).

ADFMentor/
  __init__.py
ADFMentorInternal/
  core.py
  models/
    gemini.py
  utils/
    adf_validator.py
    adf_processor.py
    adf_evaluator.py

API

ADFMentor

• ADFMentor(api_key, model_name="gemini-2.0-flash-exp")
• evaluate_adf(submission_path, prompt=None, question=None)

Returns:

{
  "score": 0,
  "feedback": "...",
  "metadata_report": "..."
}

Notes

• For .txt submissions, provide prompt to enable AI grading
• For .zip/folder submissions, .txt files are treated as optional companion notes

License

MIT