Data Engineering Operations Toolkit

A Python CLI toolkit that consolidates the small, repetitive operational checks a data platform team runs regularly — resource validation, config checks, deployment health — into one consistent tool.

Operational checks that live as one-off scripts scattered across a team tend to drift, break silently, or simply not get run under time pressure. The goal was a single, tested CLI that made the right check the easy check.

• Commands must be safe to run against production without making unintended changes (read-only by default).
• Must be runnable both locally and inside CI without additional setup beyond credentials.
• Output must be structured enough to parse in automation, not just human-readable text.

• Designed the CLI command structure and argument interface.
• Implemented AWS resource-validation and configuration-validation checks using the AWS SDK for Python.
• Wrote the unit-test suite and the GitHub Actions workflow that runs it on every change.

A single Python package exposes a CLI entry point with subcommands for resource validation, configuration validation, and deployment health checks. Each check is implemented as an independent, testable function that returns a structured result, which the CLI layer formats as human-readable output or structured JSON depending on the invocation flags.

• Used the AWS SDK for Python (boto3) for read-only resource checks, wrapped with clear, actionable error messages.
• Built configuration validation around explicit schema checks rather than ad hoc parsing.
• Added structured logging so CLI output can be consumed by both humans and automation.

• Make every command read-only by default.

  A toolkit meant to be run frequently and without ceremony needs to be safe to run against production without a second thought.

• Support structured JSON output alongside human-readable output.

  The same checks needed to run both interactively and as a step inside CI, so the output had to be machine-parseable without a separate code path.

• Keeping checks generic enough to be reusable across pipelines.

  Parameterized each check (resource identifiers, expected configuration values) instead of hard-coding assumptions about a specific pipeline.

A tested, documented CLI that turns ad hoc operational scripts into a consistent set of commands that are safe to run locally, in CI, or during an incident.

• Unit tests covering each check's logic with Pytest.
• GitHub Actions workflow running the test suite on every pull request.
• Manual verification of CLI output formatting in both human-readable and JSON modes.

• Structured logging for every command invocation, including inputs and results.

• Read-only AWS permissions required for all validation commands.
• No credentials or secrets are logged or persisted by the toolkit.

Verified results have not been published for this project yet.

• A CLI that's genuinely fast to run gets adopted; anything that adds friction gets skipped, no matter how useful it is.
• Structured output from day one made it trivial to later wire the same checks into CI without rewriting them.

• Add a `--fix` mode for checks that have an obvious, safe remediation.
• Expand configuration validation to cover additional pipeline config formats.

• Python
• AWS SDK for Python
• Pytest
• Docker
• GitHub Actions