Skip to content

Latest commit

 

History

History
725 lines (564 loc) · 32.7 KB

TUTORIAL.md

File metadata and controls

725 lines (564 loc) · 32.7 KB

AWS & Terraform — A Hands-On Tutorial

A practical lesson that uses this project as a live example. By the end you will understand how Terraform orchestrates AWS services, what each service does, and how the pieces fit together.


Table of Contents

  1. The Big Picture
  2. What is Terraform?
  3. AWS Core Services — What They Are
  4. Terraform — How It Works
  5. Execution Flow: Real Estate Scraper
  6. Execution Flow: YouTube Summarizer
  7. Terraform Modules — Organizing Your Code
  8. IAM — The Permission System
  9. The Deployment Pipeline (What deploy.ps1 Does)
  10. How Terraform Talks to AWS (API Calls Under the Hood)
  11. State File — Terraform's Memory
  12. Key Patterns and Best Practices
  13. Cheat Sheet

1. The Big Picture

This project deploys two independent automations on AWS using a single Terraform codebase:

YOUR PC                                       AWS (eu-west-1, Ireland)
───────                                       ────────────────────────

  terraform/                                  ┌──── Module 1 ─────────────────────┐
    main.tf        ── terraform apply ──>     │  EventBridge ──> Lambda (Python)   │
    modules/                                  │                    │               │
      real_estate_scraper/                    │            scrape website          │
      youtube_summarizer/                     │                    │               │
                                              │          DynamoDB (compare)        │
  lambda/                                     │                    │               │
    real_estate_scraper/                      │           new? ──> SNS ──> email   │
    youtube_summarizer/                       └───────────────────────────────────┘
                                              ┌──── Module 2 ─────────────────────┐
                                              │  Lambda (Python) <── you invoke   │
                                              │      │                            │
                                              │   SSM (read API key)              │
                                              │      │                            │
                                              │   Gemini API (analyze video)      │
                                              │      │                            │
                                              │   SES ──> email with summary      │
                                              └───────────────────────────────────┘

After terraform apply, your PC can be turned off. Everything runs on Amazon's servers.


2. What is Terraform?

Terraform is an Infrastructure as Code (IaC) tool: instead of creating AWS resources manually from the web console (clicking buttons), you describe them in text files (.tf). Terraform reads those files and creates, modifies, or destroys the resources for you.

Why use it?

  • Repeatability — you can destroy everything and recreate it from scratch in minutes
  • Version control — .tf files go into Git, so you have a full history of every infrastructure change
  • Collaboration — the team can propose changes via Pull Requests, just like application code
  • No forgotten resources — terraform destroy removes everything it created, nothing is left behind costing money

Provider

The Provider is the plugin that lets Terraform communicate with a cloud provider (AWS, Azure, GCP, etc.). It's the "bridge" between your .tf files and the cloud's APIs.

provider "aws" {
  region = "eu-west-1"   # Which data center to create resources in
}

When you run terraform init, Terraform downloads the specified provider automatically.

Resource

A Resource is an infrastructure component that Terraform creates and manages. Each resource has a type (what to create) and a logical name (how you refer to it in your code).

resource "aws_dynamodb_table" "listings" {
  #        ^^^^^^^^^^^^^^^^^   ^^^^^^^^
  #        resource type       logical name (for internal references)
  name = "scraper-listings"   # actual name on AWS
}

Terraform understands dependencies between resources: if the Lambda references the DynamoDB table's ARN, Terraform will create the table first and then the Lambda.


3. AWS Core Services — What They Are

Think of AWS as a massive toolbox. Each service is a specialized tool. Here are the ones we use:

Lambda — Serverless Code Execution

What it is: You upload a .zip with your Python code. AWS runs it when triggered, then shuts it down. No servers to manage.

Analogy: A vending machine. You put in a coin (trigger), it gives you a product (result), then it sits idle. You don't maintain the machine — AWS does.

Key properties we configure in Terraform:

Property Meaning Our value
runtime Which language/version python3.11
handler Entry point: filename.function_name scraper.lambda_handler
timeout Max execution time (seconds) 30s (scraper), 120s (summarizer)
memory_size RAM in MB (also determines CPU) 256 MB
environment Env vars passed to the code DB name, API key paths, etc.

How the code receives config values:

Terraform                              Python code
─────────                              ───────────
environment {                          import os
  variables = {
    DYNAMODB_TABLE = "..."   ─────>    table = os.environ["DYNAMODB_TABLE"]
    SNS_TOPIC_ARN  = "..."   ─────>    arn = os.environ["SNS_TOPIC_ARN"]
  }
}

Terraform injects the values at deploy time. The Python code reads them at runtime via os.environ. This decouples code from infrastructure.


DynamoDB — NoSQL Database

What it is: A key-value database fully managed by AWS. You don't install anything — you just create a table and read/write data.

Analogy: A spreadsheet in the cloud, but one that scales automatically and never crashes.

In this project: Stores the listing IDs the scraper has already seen, so it can detect new ones.

Key concept — Partition Key: Every DynamoDB table needs a primary key. Ours is listing_id (String). Each row is uniquely identified by this key.

┌─────────────┬────────────────────────────┬──────────────────────────┐
│ listing_id  │ title                      │ found_at                 │
├─────────────┼────────────────────────────┼──────────────────────────┤
│ "124229"    │ "Villa in Iseo"            │ "2026-02-24T10:30:00Z"   │
│ "124456"    │ "Apartment lake view"      │ "2026-02-24T11:00:00Z"   │
└─────────────┴────────────────────────────┴──────────────────────────┘

SNS — Simple Notification Service

What it is: A pub/sub messaging system. A "Topic" is a channel. Publishers send messages, subscribers receive them.

In this project: The scraper Lambda publishes to an SNS topic. Your email is subscribed to that topic. Result: you get an email when new listings appear.

Lambda ──publish──> SNS Topic ──forward──> your@email.com
                                       ──> another@email.com  (if subscribed)
                                       ──> a webhook          (if subscribed)

Important: After deployment, AWS sends a confirmation email. You MUST click the link, otherwise SNS will not deliver messages to you.


SES — Simple Email Service

What it is: AWS's email sending service. Unlike SNS (which uses its own email mechanism), SES gives you full control over sender address, HTML formatting, etc.

In this project: The YouTube Summarizer uses SES to send the AI-generated summary to your inbox.

Sandbox mode (default): When your AWS account is new, SES is in "sandbox" — you can only send to verified email addresses. You must verify both sender and recipient in the AWS Console under SES > Identities.

# Verify an email address for SES
aws ses verify-email-identity --email-address "your@email.com"
# Then click the verification link in your inbox

SSM Parameter Store — Secrets Management

What it is: A key-value store for configuration and secrets. Parameters can be encrypted with SecureString type.

In this project: We store the Gemini and OpenAI API keys here instead of hardcoding them in code or .tf files.

How it works:

1. Terraform creates the parameter (with a dummy placeholder)
2. You manually set the real value via CLI (aws ssm put-parameter)
3. At runtime, Lambda reads the value via boto3 (ssm.get_parameter)
4. lifecycle { ignore_changes = [value] } tells Terraform to NOT overwrite
   the real key on the next "terraform apply"

This pattern is called "Terraform creates the container, human fills it".


EventBridge — Scheduler / Event Bus

What it is: A serverless scheduler (like cron on Linux, but managed by AWS). You define a rule ("every 1 hour") and a target ("invoke this Lambda").

In this project: Triggers the real estate scraper Lambda on a schedule.

Three resources needed:

  1. aws_cloudwatch_event_rule — the schedule itself
  2. aws_cloudwatch_event_target — what to invoke (the Lambda ARN)
  3. aws_lambda_permission — explicit authorization for EventBridge to invoke the Lambda

Why the permission? AWS follows least privilege — every service must have explicit permission to interact with another.


CloudWatch — Logs & Monitoring

What it is: AWS's logging and monitoring service. Every Lambda automatically writes logs to CloudWatch.

In this project: We create a Log Group with Terraform so we control:

  • How long to keep logs (retention_in_days = 14)
  • That terraform destroy cleans up logs too (if AWS creates it automatically, it persists after destroy)

Viewing logs:

aws logs tail /aws/lambda/FUNCTION_NAME --since 1h    # last hour
aws logs tail /aws/lambda/FUNCTION_NAME --follow       # real-time stream

IAM — Identity and Access Management

This is so important it gets its own section: Section 8.


4. Terraform — How It Works

The Lifecycle: init → plan → apply → destroy

terraform init          Download provider plugins (run once)
      │
      ▼
terraform plan          Read .tf files, compare with state, show what will change
      │                 (NOTHING is created — this is a dry run)
      ▼
terraform apply         Actually create/modify/destroy resources on AWS
      │                 (asks "yes" for confirmation)
      ▼
terraform destroy       Delete everything Terraform manages
                        (the reverse of apply)

What Happens During terraform apply

  1. Terraform reads all .tf files in the current directory
  2. Builds a dependency graph of all resources
  3. Compares the desired state (your .tf files) with the actual state (terraform.tfstate)
  4. Calculates a plan: what to create, modify, or destroy
  5. Executes the plan by making API calls to AWS
  6. Updates terraform.tfstate with the new reality

Resource References — How Terraform Wires Things Together

In .tf files, resources can reference each other. Terraform uses this to:

  • Determine the order of creation (dependency resolution)
  • Pass values between resources (ARN, name, ID)
# sns.tf creates a topic
resource "aws_sns_topic" "notifications" {
  name = "my-topic"
}

# lambda.tf references the topic's ARN
environment {
  variables = {
    SNS_TOPIC_ARN = aws_sns_topic.notifications.arn
    #               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    #               TYPE.LOGICAL_NAME.ATTRIBUTE
  }
}

Terraform sees this reference and knows: "I must create the SNS topic FIRST, then the Lambda, because the Lambda needs the topic's ARN."

Variables — Making Config Flexible

variables.tf          Declares variables (name, type, default, description)
terraform.tfvars      Sets the actual values (git-ignored, contains your data)
*.tf files            Use variables with var.variable_name

Example flow:

variables.tf:           variable "notification_email" { type = string }
terraform.tfvars:       notification_email = "you@example.com"
sns.tf:                 endpoint = var.notification_email

5. Execution Flow: Real Estate Scraper

Here is exactly what happens, step by step, when the scraper runs:

  TRIGGER                               AWS SERVICES INVOLVED
  ───────                               ─────────────────────

  ┌─ EventBridge fires (every hour) ─┐
  │  "rate(1 hour)" schedule matched │
  └──────────────┬───────────────────┘
                 │
                 ▼
  ┌─ Lambda Invoked ─────────────────────────────────────────────┐
  │                                                              │
  │  1. scraper.lambda_handler(event, context) is called         │
  │                                                              │
  │  2. fetch_listings(TARGET_URL)                               │
  │     │                                                        │
  │     ├─ HTTP GET to latorrecase.it        (requests library)  │
  │     ├─ Parse HTML with BeautifulSoup                         │
  │     └─ Return list of {listing_id, title, url}               │
  │                                                              │
  │  3. get_known_ids()                                          │
  │     │                                                        │
  │     └─ DynamoDB Scan ─────────> read all listing_id values   │
  │                                                              │
  │  4. Compare: new = found - known                             │
  │                                                              │
  │  5. If new listings exist:                                   │
  │     │                                                        │
  │     ├─ save_new_listings()                                   │
  │     │  └─ DynamoDB BatchWriteItem ──> write new rows         │
  │     │                                                        │
  │     └─ send_notification()                                   │
  │        └─ SNS Publish ──────────────> email to subscriber    │
  │                                                              │
  │  6. Return { statusCode: 200, body: "..." }                  │
  └──────────────────────────────────────────────────────────────┘

AWS API calls made by the Lambda at runtime:

Call Service Action Why
1 DynamoDB Scan Read all known listing IDs
2 DynamoDB BatchWriteItem Save new listings (if any)
3 SNS Publish Send email notification (if new listings)

Each call requires an IAM permission — that's why the Lambda's IAM role has a policy granting dynamodb:Scan, dynamodb:PutItem, dynamodb:BatchWriteItem, and sns:Publish.


6. Execution Flow: YouTube Summarizer

This Lambda is invoked manually (no EventBridge schedule). You trigger it by passing a YouTube URL.

  TRIGGER                               AWS SERVICES INVOLVED
  ───────                               ─────────────────────

  ┌─ You invoke the Lambda ──────────┐
  │  payload: {"youtube_url": "..."}  │
  └──────────────┬───────────────────┘
                 │
                 ▼
  ┌─ Lambda Invoked ─────────────────────────────────────────────┐
  │                                                              │
  │  1. Parse the youtube_url from the event payload             │
  │                                                              │
  │  2. Read Gemini API key from SSM Parameter Store             │
  │     └─ SSM GetParameter (with decryption)                    │
  │                                                              │
  │  3. PHASE 1 — Try Google Gemini (native video analysis):     │
  │     │                                                        │
  │     ├─ Build a Part.from_uri(youtube_url, "video/mp4")       │
  │     ├─ Call Gemini API: generate_content(video + prompt)     │
  │     └─ Gemini watches the video and generates a summary      │
  │                                                              │
  │  4. If Gemini fails → PHASE 2 — OpenAI Fallback:            │
  │     │                                                        │
  │     ├─ Read OpenAI API key from SSM Parameter Store          │
  │     ├─ Fetch transcript using youtube_transcript_api         │
  │     │  (this calls YouTube's internal API, no key needed)    │
  │     ├─ Send transcript text to GPT-4o for summarization      │
  │     └─ GPT-4o returns a structured summary                   │
  │                                                              │
  │  5. Send the summary via SES email                           │
  │     └─ SES SendEmail(from, to, subject, body)                │
  │                                                              │
  │  6. Return { statusCode: 200, body: {provider, model, ...} } │
  └──────────────────────────────────────────────────────────────┘

AWS API calls made by the Lambda at runtime:

Call Service Action Why
1 SSM GetParameter Read Gemini API key (encrypted)
2 SSM GetParameter Read OpenAI API key (if fallback needed)
3 SES SendEmail Send the summary to your inbox

External API calls (not AWS):

Call Service Why
1 Google Gemini API Analyze the YouTube video and generate summary
2 YouTube Transcript API Extract the video's text transcript (fallback only)
3 OpenAI API Summarize the transcript text (fallback only)

7. Terraform Modules — Organizing Your Code

The Problem

Without modules, all resources live in one flat directory. With 20+ resources across two projects, the .tf files become a mess.

The Solution: Modules

A module is a folder of .tf files that acts as a reusable, self-contained unit. The root main.tf calls each module and passes variables to it:

terraform/
├── main.tf                    ← Root: calls the modules
├── variables.tf               ← Root-level variables
├── outputs.tf                 ← Root-level outputs
└── modules/
    ├── real_estate_scraper/   ← Module 1 (its own .tf files)
    │   ├── lambda.tf
    │   ├── dynamodb.tf
    │   ├── sns.tf
    │   ├── eventbridge.tf
    │   ├── variables.tf       ← Module's own input variables
    │   └── outputs.tf         ← Module's own outputs
    └── youtube_summarizer/    ← Module 2 (its own .tf files)
        ├── main.tf
        ├── iam.tf
        └── variables.tf

Root main.tf calling a module:

module "youtube_summarizer" {
  source = "./modules/youtube_summarizer"    # path to the module folder

  project_name        = var.project_name     # passing variables IN
  ses_sender_email    = var.youtube_ses_sender_email
  ses_recipient_email = var.youtube_ses_recipient_email
}

Key rules:

  • Each module has its own variables.tf — it only sees what you pass in
  • Each module has its own outputs.tf — the root accesses them as module.MODULE_NAME.output_name
  • Resources in different modules are isolated — they can't reference each other directly
  • terraform state list shows module paths like module.youtube_summarizer.aws_lambda_function.youtube_summarizer

8. IAM — The Permission System

IAM is the most important (and often most confusing) part of AWS. Here's how it works in our project.

The Concept

Every AWS service needs explicit permission to interact with other services. Without it, everything is denied by default.

Lambda function ──wants to──> Write to DynamoDB
       │
       └── Has an IAM Role attached
              │
              └── Role has a Policy that says:
                    "Allow dynamodb:PutItem on table X"

Three Layers

┌─────────────────────────────────────────────────────────┐
│ IAM ROLE                                                │
│                                                         │
│   "I am a badge. Whoever wears me gets the permissions  │
│    listed in my policies."                              │
│                                                         │
│   Trust Policy:                                         │
│     "Lambda service can wear this badge"                │
│     (sts:AssumeRole by lambda.amazonaws.com)            │
│                                                         │
│   ┌───────────────────────────────────────────────┐     │
│   │ POLICY 1: AWSLambdaBasicExecutionRole         │     │
│   │   (AWS managed — pre-built by Amazon)         │     │
│   │   Allows: logs:CreateLogGroup                 │     │
│   │           logs:CreateLogStream                │     │
│   │           logs:PutLogEvents                   │     │
│   └───────────────────────────────────────────────┘     │
│                                                         │
│   ┌───────────────────────────────────────────────┐     │
│   │ POLICY 2: Custom Policy (we wrote this)       │     │
│   │   Allows: dynamodb:Scan on TABLE_ARN          │     │
│   │           dynamodb:PutItem on TABLE_ARN       │     │
│   │           sns:Publish on TOPIC_ARN            │     │
│   └───────────────────────────────────────────────┘     │
│                                                         │
└─────────────────────────────────────────────────────────┘

Comparison: Scraper vs Summarizer Permissions

Real Estate Scraper YouTube Summarizer
CloudWatch Logs AWSLambdaBasicExecutionRole AWSLambdaBasicExecutionRole
DynamoDB Scan, PutItem, BatchWriteItem — (not used)
SNS Publish — (not used)
SSM — (not used) GetParameter (Gemini + OpenAI keys)
SES — (not used) SendEmail, SendRawEmail

Each Lambda gets only the permissions it needs — this is the Principle of Least Privilege.


9. The Deployment Pipeline (What deploy.ps1 Does)

The deploy.ps1 script automates the full build and deploy process. Here's what happens step by step:

deploy.ps1
│
├─ [1/3] Package Real Estate Scraper Lambda
│    ├─ Remove old lambda/real_estate_scraper/package/ folder
│    ├─ pip install -r requirements.txt -t package/
│    │   (downloads requests, beautifulsoup4 into the folder)
│    │   (--platform manylinux2014_x86_64 = Linux binaries for Lambda)
│    └─ Copy scraper.py into package/
│
├─ [2/3] Package YouTube Summarizer Lambda
│    ├─ Remove old lambda/youtube_summarizer/package/ folder
│    ├─ pip install -r requirements.txt -t package/
│    │   (downloads google-genai, openai, youtube-transcript-api)
│    └─ Copy lambda_function.py into package/
│
└─ [3/3] Run Terraform
     ├─ terraform init        (download providers if needed)
     ├─ terraform validate    (syntax check)
     └─ terraform apply       (deploy to AWS)
          │
          ├─ data.archive_file creates .zip from each package/ folder
          ├─ Uploads .zip to Lambda
          ├─ Creates/updates all other resources (DynamoDB, SNS, SES, etc.)
          └─ Prints outputs (function names, ARNs)

Why --platform manylinux2014_x86_64? Lambda runs on Amazon Linux (x86_64). If you pip install on Windows, some packages install Windows-specific binaries that won't work on Lambda. This flag forces pip to download Linux-compatible packages.


10. How Terraform Talks to AWS (API Calls Under the Hood)

When you run terraform apply, Terraform doesn't use any magic — it makes HTTP API calls to AWS, just like the AWS CLI does.

Example: creating a DynamoDB table

Your .tf file:

resource "aws_dynamodb_table" "listings" {
  name         = "scraper-listings"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "listing_id"
  attribute {
    name = "listing_id"
    type = "S"
  }
}

What Terraform actually does:

POST https://dynamodb.eu-west-1.amazonaws.com/
Headers: Authorization: AWS4-HMAC-SHA256 ... (your credentials)
Body: {
  "TableName": "scraper-listings",
  "BillingMode": "PAY_PER_REQUEST",
  "KeySchema": [{"AttributeName": "listing_id", "KeyType": "HASH"}],
  "AttributeDefinitions": [{"AttributeName": "listing_id", "AttributeType": "S"}]
}

It's the same API call you'd make with aws dynamodb create-table from the CLI. Terraform just automates it and tracks the result in its state file.

Authentication: Terraform uses the same credentials as aws configure — the Access Key ID and Secret Access Key stored in ~/.aws/credentials.


11. State File — Terraform's Memory

What Is It?

terraform.tfstate is a JSON file that records:

  • Every resource Terraform created
  • Its real AWS ID, ARN, and all properties
  • The mapping between your logical names (in .tf files) and real resource IDs

Why It Matters

Without state, Terraform wouldn't know if a resource already exists. It would try to create a duplicate and fail. The state file is how Terraform answers: "Does this already exist? Has it changed?"

Example Entry

{
  "type": "aws_dynamodb_table",
  "name": "listings",
  "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
  "instances": [{
    "attributes": {
      "name": "scraper-listings",
      "arn": "arn:aws:dynamodb:eu-west-1:123456789:table/scraper-listings",
      "billing_mode": "PAY_PER_REQUEST",
      "hash_key": "listing_id"
    }
  }]
}

Golden Rules

Rule Why
Never commit .tfstate to Git It contains sensitive data (ARNs, IDs, sometimes secrets)
Never edit .tfstate by hand Corrupting it can orphan resources on AWS
In teams, use a remote backend S3 + DynamoDB backend lets team members share state safely
Always run terraform plan before apply To see what Terraform THINKS needs to change

12. Key Patterns and Best Practices

Pattern 1: Environment Variables Bridge

Terraform injects config values into Lambda via environment variables. The Python code reads them with os.environ. This means:

  • The code has zero hardcoded AWS resource names
  • You can change the DynamoDB table name in Terraform without touching Python code
  • The same code works in dev, staging, and production

Pattern 2: SSM for Secrets

API keys are stored in SSM Parameter Store (encrypted). Terraform creates the parameter with a dummy value and lifecycle { ignore_changes = [value] }. You then set the real value manually. This way:

  • Secrets never appear in .tf files or Git
  • Terraform doesn't overwrite your real key on every apply
  • The Lambda reads the key at runtime, not at deploy time

Pattern 3: Least Privilege IAM

Each Lambda gets a custom policy with only the permissions it needs. The scraper can write to DynamoDB and publish to SNS — nothing else. The summarizer can read SSM and send SES emails — nothing else.

Pattern 4: depends_on for Ordering

Some dependencies are implicit (Terraform figures them out from references). Others are explicit:

depends_on = [
  aws_iam_role_policy_attachment.lambda_basic_execution,
  aws_iam_role_policy_attachment.lambda_custom,
]

This tells Terraform: "Don't create the Lambda until the IAM policies are attached." Without this, the Lambda might be created before its permissions are ready, causing a brief window where it can't write logs.

Pattern 5: Tags for Organization

Every resource is tagged with Project and ManagedBy:

tags = {
  Project   = var.project_name
  ManagedBy = "terraform"
}

This helps in the AWS Console: you can filter by tag to see only resources from this project, and ManagedBy = "terraform" warns you not to edit them manually.


13. Cheat Sheet

Terraform Commands

Command What it does
terraform init Download providers, initialize working directory
terraform plan Dry run: show what will be created/changed/destroyed
terraform apply Execute the plan (creates real resources)
terraform destroy Delete all managed resources
terraform output Show output values (names, ARNs)
terraform state list List all resources in the state file
terraform state show RESOURCE Show details of a specific resource
terraform fmt Auto-format your .tf files
terraform validate Check .tf syntax without connecting to AWS

AWS CLI Commands (Most Useful)

Command What it does
aws sts get-caller-identity Who am I? (verify credentials)
aws lambda invoke ... Trigger a Lambda manually
aws logs tail LOG_GROUP --follow Stream Lambda logs in real time
aws dynamodb scan --table-name TABLE Read all items from a DynamoDB table
aws ssm put-parameter --name X --value Y --type SecureString Store a secret
aws ses verify-email-identity --email-address X Verify an email for SES
aws events describe-rule --name RULE Check an EventBridge schedule
aws events disable-rule --name RULE Temporarily pause a schedule

Key File Locations

File Purpose
terraform/main.tf Entry point: provider + module calls
terraform/variables.tf All configurable variables
terraform/terraform.tfvars Your actual values (git-ignored)
terraform/terraform.tfstate Terraform's state (git-ignored, sensitive)
terraform/modules/*/ Individual module definitions
lambda/*/lambda_function.py The actual Python code that runs on AWS
lambda/*/requirements.txt Python dependencies for each Lambda
deploy.ps1 One-click build and deploy script
help.ps1 Interactive command reference