AWS Data Infrastructure Automation

A reusable Terraform module set for provisioning the AWS infrastructure behind a data pipeline — storage, compute, IAM, and notifications — with environment-aware configuration and CI validation.

Hand-provisioned cloud resources are hard to reproduce, easy to over-permission, and difficult to audit. The goal was infrastructure that is fully defined as code, safe to apply repeatedly, and structured so that least-privilege access is the default rather than an afterthought.

• Must support multiple environments (e.g., dev/staging/prod) from the same module set without duplicating configuration.
• IAM policies must be scoped to specific resources rather than using wildcard permissions.
• All changes must pass automated validation before being applied.

• Authored the Terraform module structure and environment variable strategy.
• Wrote least-privilege IAM policies for each Lambda function and service role.
• Set up GitHub Actions workflows for `terraform fmt`, `validate`, and `plan` on every pull request.

Terraform modules provision an encrypted S3 bucket for pipeline data, Lambda functions for lightweight processing, EventBridge rules to trigger them, SNS topics for operational notifications, and Secrets Manager entries for credentials. KMS keys encrypt data at rest, and CloudWatch captures logs and metrics for every component. Environment-specific values (naming, sizing, alerting endpoints) are isolated in per-environment `.tfvars` files.

• Structured the project as composable Terraform modules (storage, compute, notifications, IAM) rather than one large configuration.
• Used per-environment variable files so the same module code deploys consistently across environments.
• Wrote IAM policies scoped to specific resource ARNs and actions instead of broad managed policies.

• Split infrastructure into small, composable modules.

  Smaller modules are easier to review, test in isolation, and reuse across other pipelines without copy-pasting configuration.

• Validate every change in CI before allowing an apply.

  Running `terraform fmt`, `validate`, and `plan` in GitHub Actions catches configuration errors and unintended resource changes before they reach an environment.

• Balancing least-privilege IAM with practical maintainability.

  Built resource-scoped policy templates per module so tightening permissions didn't require hand-writing a new policy for every Lambda function.

A modular Terraform project that any environment can deploy consistently, with least-privilege IAM by default and automated validation gating every change before it reaches an environment.

• `terraform validate` and `terraform fmt -check` run automatically in GitHub Actions.
• `terraform plan` reviewed on every pull request before merge.
• Manual verification of IAM policies against the principle of least privilege.

• CloudWatch log groups for every Lambda function.
• SNS notifications for pipeline and deployment events.

• KMS encryption for data at rest across all provisioned storage.
• IAM policies scoped to specific resource ARNs rather than wildcard permissions.
• Credentials stored in Secrets Manager, never in Terraform variables or state files in plaintext.

Verified results have not been published for this project yet.

• Small, composable modules pay off quickly once the same infrastructure needs to be reused in a second environment.
• CI-gated `terraform plan` review catches unintended resource replacements before they become incidents.

• Add automated policy linting (e.g., checking for wildcard IAM actions) as a CI step.
• Add drift detection to periodically compare deployed state against configuration.

• AWS
• Terraform
• IAM
• Lambda
• EventBridge
• KMS
• GitHub Actions