A practical lesson that uses this project as a live example. By the end you will understand how Terraform orchestrates AWS services, what each service does, and how the pieces fit together.
- The Big Picture
- What is Terraform?
- AWS Core Services — What They Are
- Terraform — How It Works
- Execution Flow: Real Estate Scraper
- Execution Flow: YouTube Summarizer
- Terraform Modules — Organizing Your Code
- IAM — The Permission System
- The Deployment Pipeline (What
deploy.ps1Does) - How Terraform Talks to AWS (API Calls Under the Hood)
- State File — Terraform's Memory
- Key Patterns and Best Practices
- Cheat Sheet
This project deploys two independent automations on AWS using a single Terraform codebase:
YOUR PC AWS (eu-west-1, Ireland)
─────── ────────────────────────
terraform/ ┌──── Module 1 ─────────────────────┐
main.tf ── terraform apply ──> │ EventBridge ──> Lambda (Python) │
modules/ │ │ │
real_estate_scraper/ │ scrape website │
youtube_summarizer/ │ │ │
│ DynamoDB (compare) │
lambda/ │ │ │
real_estate_scraper/ │ new? ──> SNS ──> email │
youtube_summarizer/ └───────────────────────────────────┘
┌──── Module 2 ─────────────────────┐
│ Lambda (Python) <── you invoke │
│ │ │
│ SSM (read API key) │
│ │ │
│ Gemini API (analyze video) │
│ │ │
│ SES ──> email with summary │
└───────────────────────────────────┘
After terraform apply, your PC can be turned off. Everything runs on Amazon's servers.
Terraform is an Infrastructure as Code (IaC) tool: instead of creating AWS resources manually from the web console (clicking buttons), you describe them in text files (.tf). Terraform reads those files and creates, modifies, or destroys the resources for you.
Why use it?
- Repeatability — you can destroy everything and recreate it from scratch in minutes
- Version control —
.tffiles go into Git, so you have a full history of every infrastructure change - Collaboration — the team can propose changes via Pull Requests, just like application code
- No forgotten resources —
terraform destroyremoves everything it created, nothing is left behind costing money
The Provider is the plugin that lets Terraform communicate with a cloud provider (AWS, Azure, GCP, etc.). It's the "bridge" between your .tf files and the cloud's APIs.
provider "aws" {
region = "eu-west-1" # Which data center to create resources in
}When you run terraform init, Terraform downloads the specified provider automatically.
A Resource is an infrastructure component that Terraform creates and manages. Each resource has a type (what to create) and a logical name (how you refer to it in your code).
resource "aws_dynamodb_table" "listings" {
# ^^^^^^^^^^^^^^^^^ ^^^^^^^^
# resource type logical name (for internal references)
name = "scraper-listings" # actual name on AWS
}Terraform understands dependencies between resources: if the Lambda references the DynamoDB table's ARN, Terraform will create the table first and then the Lambda.
Think of AWS as a massive toolbox. Each service is a specialized tool. Here are the ones we use:
What it is: You upload a .zip with your Python code. AWS runs it when triggered, then shuts it down. No servers to manage.
Analogy: A vending machine. You put in a coin (trigger), it gives you a product (result), then it sits idle. You don't maintain the machine — AWS does.
Key properties we configure in Terraform:
| Property | Meaning | Our value |
|---|---|---|
runtime |
Which language/version | python3.11 |
handler |
Entry point: filename.function_name |
scraper.lambda_handler |
timeout |
Max execution time (seconds) | 30s (scraper), 120s (summarizer) |
memory_size |
RAM in MB (also determines CPU) | 256 MB |
environment |
Env vars passed to the code | DB name, API key paths, etc. |
How the code receives config values:
Terraform Python code
───────── ───────────
environment { import os
variables = {
DYNAMODB_TABLE = "..." ─────> table = os.environ["DYNAMODB_TABLE"]
SNS_TOPIC_ARN = "..." ─────> arn = os.environ["SNS_TOPIC_ARN"]
}
}
Terraform injects the values at deploy time. The Python code reads them at runtime via os.environ. This decouples code from infrastructure.
What it is: A key-value database fully managed by AWS. You don't install anything — you just create a table and read/write data.
Analogy: A spreadsheet in the cloud, but one that scales automatically and never crashes.
In this project: Stores the listing IDs the scraper has already seen, so it can detect new ones.
Key concept — Partition Key:
Every DynamoDB table needs a primary key. Ours is listing_id (String). Each row is uniquely identified by this key.
┌─────────────┬────────────────────────────┬──────────────────────────┐
│ listing_id │ title │ found_at │
├─────────────┼────────────────────────────┼──────────────────────────┤
│ "124229" │ "Villa in Iseo" │ "2026-02-24T10:30:00Z" │
│ "124456" │ "Apartment lake view" │ "2026-02-24T11:00:00Z" │
└─────────────┴────────────────────────────┴──────────────────────────┘
What it is: A pub/sub messaging system. A "Topic" is a channel. Publishers send messages, subscribers receive them.
In this project: The scraper Lambda publishes to an SNS topic. Your email is subscribed to that topic. Result: you get an email when new listings appear.
Lambda ──publish──> SNS Topic ──forward──> your@email.com
──> another@email.com (if subscribed)
──> a webhook (if subscribed)
Important: After deployment, AWS sends a confirmation email. You MUST click the link, otherwise SNS will not deliver messages to you.
What it is: AWS's email sending service. Unlike SNS (which uses its own email mechanism), SES gives you full control over sender address, HTML formatting, etc.
In this project: The YouTube Summarizer uses SES to send the AI-generated summary to your inbox.
Sandbox mode (default): When your AWS account is new, SES is in "sandbox" — you can only send to verified email addresses. You must verify both sender and recipient in the AWS Console under SES > Identities.
# Verify an email address for SES
aws ses verify-email-identity --email-address "your@email.com"
# Then click the verification link in your inboxWhat it is: A key-value store for configuration and secrets. Parameters can be encrypted with SecureString type.
In this project: We store the Gemini and OpenAI API keys here instead of hardcoding them in code or .tf files.
How it works:
1. Terraform creates the parameter (with a dummy placeholder)
2. You manually set the real value via CLI (aws ssm put-parameter)
3. At runtime, Lambda reads the value via boto3 (ssm.get_parameter)
4. lifecycle { ignore_changes = [value] } tells Terraform to NOT overwrite
the real key on the next "terraform apply"
This pattern is called "Terraform creates the container, human fills it".
What it is: A serverless scheduler (like cron on Linux, but managed by AWS). You define a rule ("every 1 hour") and a target ("invoke this Lambda").
In this project: Triggers the real estate scraper Lambda on a schedule.
Three resources needed:
aws_cloudwatch_event_rule— the schedule itselfaws_cloudwatch_event_target— what to invoke (the Lambda ARN)aws_lambda_permission— explicit authorization for EventBridge to invoke the Lambda
Why the permission? AWS follows least privilege — every service must have explicit permission to interact with another.
What it is: AWS's logging and monitoring service. Every Lambda automatically writes logs to CloudWatch.
In this project: We create a Log Group with Terraform so we control:
- How long to keep logs (
retention_in_days = 14) - That
terraform destroycleans up logs too (if AWS creates it automatically, it persists after destroy)
Viewing logs:
aws logs tail /aws/lambda/FUNCTION_NAME --since 1h # last hour
aws logs tail /aws/lambda/FUNCTION_NAME --follow # real-time streamThis is so important it gets its own section: Section 8.
terraform init Download provider plugins (run once)
│
▼
terraform plan Read .tf files, compare with state, show what will change
│ (NOTHING is created — this is a dry run)
▼
terraform apply Actually create/modify/destroy resources on AWS
│ (asks "yes" for confirmation)
▼
terraform destroy Delete everything Terraform manages
(the reverse of apply)
- Terraform reads all
.tffiles in the current directory - Builds a dependency graph of all resources
- Compares the desired state (your
.tffiles) with the actual state (terraform.tfstate) - Calculates a plan: what to create, modify, or destroy
- Executes the plan by making API calls to AWS
- Updates
terraform.tfstatewith the new reality
In .tf files, resources can reference each other. Terraform uses this to:
- Determine the order of creation (dependency resolution)
- Pass values between resources (ARN, name, ID)
# sns.tf creates a topic
resource "aws_sns_topic" "notifications" {
name = "my-topic"
}
# lambda.tf references the topic's ARN
environment {
variables = {
SNS_TOPIC_ARN = aws_sns_topic.notifications.arn
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# TYPE.LOGICAL_NAME.ATTRIBUTE
}
}Terraform sees this reference and knows: "I must create the SNS topic FIRST, then the Lambda, because the Lambda needs the topic's ARN."
variables.tf Declares variables (name, type, default, description)
terraform.tfvars Sets the actual values (git-ignored, contains your data)
*.tf files Use variables with var.variable_name
Example flow:
variables.tf: variable "notification_email" { type = string }
terraform.tfvars: notification_email = "you@example.com"
sns.tf: endpoint = var.notification_email
Here is exactly what happens, step by step, when the scraper runs:
TRIGGER AWS SERVICES INVOLVED
─────── ─────────────────────
┌─ EventBridge fires (every hour) ─┐
│ "rate(1 hour)" schedule matched │
└──────────────┬───────────────────┘
│
▼
┌─ Lambda Invoked ─────────────────────────────────────────────┐
│ │
│ 1. scraper.lambda_handler(event, context) is called │
│ │
│ 2. fetch_listings(TARGET_URL) │
│ │ │
│ ├─ HTTP GET to latorrecase.it (requests library) │
│ ├─ Parse HTML with BeautifulSoup │
│ └─ Return list of {listing_id, title, url} │
│ │
│ 3. get_known_ids() │
│ │ │
│ └─ DynamoDB Scan ─────────> read all listing_id values │
│ │
│ 4. Compare: new = found - known │
│ │
│ 5. If new listings exist: │
│ │ │
│ ├─ save_new_listings() │
│ │ └─ DynamoDB BatchWriteItem ──> write new rows │
│ │ │
│ └─ send_notification() │
│ └─ SNS Publish ──────────────> email to subscriber │
│ │
│ 6. Return { statusCode: 200, body: "..." } │
└──────────────────────────────────────────────────────────────┘
AWS API calls made by the Lambda at runtime:
| Call | Service | Action | Why |
|---|---|---|---|
| 1 | DynamoDB | Scan |
Read all known listing IDs |
| 2 | DynamoDB | BatchWriteItem |
Save new listings (if any) |
| 3 | SNS | Publish |
Send email notification (if new listings) |
Each call requires an IAM permission — that's why the Lambda's IAM role has a policy granting dynamodb:Scan, dynamodb:PutItem, dynamodb:BatchWriteItem, and sns:Publish.
This Lambda is invoked manually (no EventBridge schedule). You trigger it by passing a YouTube URL.
TRIGGER AWS SERVICES INVOLVED
─────── ─────────────────────
┌─ You invoke the Lambda ──────────┐
│ payload: {"youtube_url": "..."} │
└──────────────┬───────────────────┘
│
▼
┌─ Lambda Invoked ─────────────────────────────────────────────┐
│ │
│ 1. Parse the youtube_url from the event payload │
│ │
│ 2. Read Gemini API key from SSM Parameter Store │
│ └─ SSM GetParameter (with decryption) │
│ │
│ 3. PHASE 1 — Try Google Gemini (native video analysis): │
│ │ │
│ ├─ Build a Part.from_uri(youtube_url, "video/mp4") │
│ ├─ Call Gemini API: generate_content(video + prompt) │
│ └─ Gemini watches the video and generates a summary │
│ │
│ 4. If Gemini fails → PHASE 2 — OpenAI Fallback: │
│ │ │
│ ├─ Read OpenAI API key from SSM Parameter Store │
│ ├─ Fetch transcript using youtube_transcript_api │
│ │ (this calls YouTube's internal API, no key needed) │
│ ├─ Send transcript text to GPT-4o for summarization │
│ └─ GPT-4o returns a structured summary │
│ │
│ 5. Send the summary via SES email │
│ └─ SES SendEmail(from, to, subject, body) │
│ │
│ 6. Return { statusCode: 200, body: {provider, model, ...} } │
└──────────────────────────────────────────────────────────────┘
AWS API calls made by the Lambda at runtime:
| Call | Service | Action | Why |
|---|---|---|---|
| 1 | SSM | GetParameter |
Read Gemini API key (encrypted) |
| 2 | SSM | GetParameter |
Read OpenAI API key (if fallback needed) |
| 3 | SES | SendEmail |
Send the summary to your inbox |
External API calls (not AWS):
| Call | Service | Why |
|---|---|---|
| 1 | Google Gemini API | Analyze the YouTube video and generate summary |
| 2 | YouTube Transcript API | Extract the video's text transcript (fallback only) |
| 3 | OpenAI API | Summarize the transcript text (fallback only) |
Without modules, all resources live in one flat directory. With 20+ resources across two projects, the .tf files become a mess.
A module is a folder of .tf files that acts as a reusable, self-contained unit. The root main.tf calls each module and passes variables to it:
terraform/
├── main.tf ← Root: calls the modules
├── variables.tf ← Root-level variables
├── outputs.tf ← Root-level outputs
└── modules/
├── real_estate_scraper/ ← Module 1 (its own .tf files)
│ ├── lambda.tf
│ ├── dynamodb.tf
│ ├── sns.tf
│ ├── eventbridge.tf
│ ├── variables.tf ← Module's own input variables
│ └── outputs.tf ← Module's own outputs
└── youtube_summarizer/ ← Module 2 (its own .tf files)
├── main.tf
├── iam.tf
└── variables.tf
Root main.tf calling a module:
module "youtube_summarizer" {
source = "./modules/youtube_summarizer" # path to the module folder
project_name = var.project_name # passing variables IN
ses_sender_email = var.youtube_ses_sender_email
ses_recipient_email = var.youtube_ses_recipient_email
}Key rules:
- Each module has its own
variables.tf— it only sees what you pass in - Each module has its own
outputs.tf— the root accesses them asmodule.MODULE_NAME.output_name - Resources in different modules are isolated — they can't reference each other directly
terraform state listshows module paths likemodule.youtube_summarizer.aws_lambda_function.youtube_summarizer
IAM is the most important (and often most confusing) part of AWS. Here's how it works in our project.
Every AWS service needs explicit permission to interact with other services. Without it, everything is denied by default.
Lambda function ──wants to──> Write to DynamoDB
│
└── Has an IAM Role attached
│
└── Role has a Policy that says:
"Allow dynamodb:PutItem on table X"
┌─────────────────────────────────────────────────────────┐
│ IAM ROLE │
│ │
│ "I am a badge. Whoever wears me gets the permissions │
│ listed in my policies." │
│ │
│ Trust Policy: │
│ "Lambda service can wear this badge" │
│ (sts:AssumeRole by lambda.amazonaws.com) │
│ │
│ ┌───────────────────────────────────────────────┐ │
│ │ POLICY 1: AWSLambdaBasicExecutionRole │ │
│ │ (AWS managed — pre-built by Amazon) │ │
│ │ Allows: logs:CreateLogGroup │ │
│ │ logs:CreateLogStream │ │
│ │ logs:PutLogEvents │ │
│ └───────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────┐ │
│ │ POLICY 2: Custom Policy (we wrote this) │ │
│ │ Allows: dynamodb:Scan on TABLE_ARN │ │
│ │ dynamodb:PutItem on TABLE_ARN │ │
│ │ sns:Publish on TOPIC_ARN │ │
│ └───────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────┘
| Real Estate Scraper | YouTube Summarizer | |
|---|---|---|
| CloudWatch Logs | AWSLambdaBasicExecutionRole | AWSLambdaBasicExecutionRole |
| DynamoDB | Scan, PutItem, BatchWriteItem | — (not used) |
| SNS | Publish | — (not used) |
| SSM | — (not used) | GetParameter (Gemini + OpenAI keys) |
| SES | — (not used) | SendEmail, SendRawEmail |
Each Lambda gets only the permissions it needs — this is the Principle of Least Privilege.
The deploy.ps1 script automates the full build and deploy process. Here's what happens step by step:
deploy.ps1
│
├─ [1/3] Package Real Estate Scraper Lambda
│ ├─ Remove old lambda/real_estate_scraper/package/ folder
│ ├─ pip install -r requirements.txt -t package/
│ │ (downloads requests, beautifulsoup4 into the folder)
│ │ (--platform manylinux2014_x86_64 = Linux binaries for Lambda)
│ └─ Copy scraper.py into package/
│
├─ [2/3] Package YouTube Summarizer Lambda
│ ├─ Remove old lambda/youtube_summarizer/package/ folder
│ ├─ pip install -r requirements.txt -t package/
│ │ (downloads google-genai, openai, youtube-transcript-api)
│ └─ Copy lambda_function.py into package/
│
└─ [3/3] Run Terraform
├─ terraform init (download providers if needed)
├─ terraform validate (syntax check)
└─ terraform apply (deploy to AWS)
│
├─ data.archive_file creates .zip from each package/ folder
├─ Uploads .zip to Lambda
├─ Creates/updates all other resources (DynamoDB, SNS, SES, etc.)
└─ Prints outputs (function names, ARNs)
Why --platform manylinux2014_x86_64? Lambda runs on Amazon Linux (x86_64). If you pip install on Windows, some packages install Windows-specific binaries that won't work on Lambda. This flag forces pip to download Linux-compatible packages.
When you run terraform apply, Terraform doesn't use any magic — it makes HTTP API calls to AWS, just like the AWS CLI does.
Example: creating a DynamoDB table
Your .tf file:
resource "aws_dynamodb_table" "listings" {
name = "scraper-listings"
billing_mode = "PAY_PER_REQUEST"
hash_key = "listing_id"
attribute {
name = "listing_id"
type = "S"
}
}What Terraform actually does:
POST https://dynamodb.eu-west-1.amazonaws.com/
Headers: Authorization: AWS4-HMAC-SHA256 ... (your credentials)
Body: {
"TableName": "scraper-listings",
"BillingMode": "PAY_PER_REQUEST",
"KeySchema": [{"AttributeName": "listing_id", "KeyType": "HASH"}],
"AttributeDefinitions": [{"AttributeName": "listing_id", "AttributeType": "S"}]
}
It's the same API call you'd make with aws dynamodb create-table from the CLI. Terraform just automates it and tracks the result in its state file.
Authentication: Terraform uses the same credentials as aws configure — the Access Key ID and Secret Access Key stored in ~/.aws/credentials.
terraform.tfstate is a JSON file that records:
- Every resource Terraform created
- Its real AWS ID, ARN, and all properties
- The mapping between your logical names (in
.tffiles) and real resource IDs
Without state, Terraform wouldn't know if a resource already exists. It would try to create a duplicate and fail. The state file is how Terraform answers: "Does this already exist? Has it changed?"
{
"type": "aws_dynamodb_table",
"name": "listings",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [{
"attributes": {
"name": "scraper-listings",
"arn": "arn:aws:dynamodb:eu-west-1:123456789:table/scraper-listings",
"billing_mode": "PAY_PER_REQUEST",
"hash_key": "listing_id"
}
}]
}| Rule | Why |
|---|---|
Never commit .tfstate to Git |
It contains sensitive data (ARNs, IDs, sometimes secrets) |
Never edit .tfstate by hand |
Corrupting it can orphan resources on AWS |
| In teams, use a remote backend | S3 + DynamoDB backend lets team members share state safely |
Always run terraform plan before apply |
To see what Terraform THINKS needs to change |
Terraform injects config values into Lambda via environment variables. The Python code reads them with os.environ. This means:
- The code has zero hardcoded AWS resource names
- You can change the DynamoDB table name in Terraform without touching Python code
- The same code works in dev, staging, and production
API keys are stored in SSM Parameter Store (encrypted). Terraform creates the parameter with a dummy value and lifecycle { ignore_changes = [value] }. You then set the real value manually. This way:
- Secrets never appear in
.tffiles or Git - Terraform doesn't overwrite your real key on every
apply - The Lambda reads the key at runtime, not at deploy time
Each Lambda gets a custom policy with only the permissions it needs. The scraper can write to DynamoDB and publish to SNS — nothing else. The summarizer can read SSM and send SES emails — nothing else.
Some dependencies are implicit (Terraform figures them out from references). Others are explicit:
depends_on = [
aws_iam_role_policy_attachment.lambda_basic_execution,
aws_iam_role_policy_attachment.lambda_custom,
]This tells Terraform: "Don't create the Lambda until the IAM policies are attached." Without this, the Lambda might be created before its permissions are ready, causing a brief window where it can't write logs.
Every resource is tagged with Project and ManagedBy:
tags = {
Project = var.project_name
ManagedBy = "terraform"
}This helps in the AWS Console: you can filter by tag to see only resources from this project, and ManagedBy = "terraform" warns you not to edit them manually.
| Command | What it does |
|---|---|
terraform init |
Download providers, initialize working directory |
terraform plan |
Dry run: show what will be created/changed/destroyed |
terraform apply |
Execute the plan (creates real resources) |
terraform destroy |
Delete all managed resources |
terraform output |
Show output values (names, ARNs) |
terraform state list |
List all resources in the state file |
terraform state show RESOURCE |
Show details of a specific resource |
terraform fmt |
Auto-format your .tf files |
terraform validate |
Check .tf syntax without connecting to AWS |
| Command | What it does |
|---|---|
aws sts get-caller-identity |
Who am I? (verify credentials) |
aws lambda invoke ... |
Trigger a Lambda manually |
aws logs tail LOG_GROUP --follow |
Stream Lambda logs in real time |
aws dynamodb scan --table-name TABLE |
Read all items from a DynamoDB table |
aws ssm put-parameter --name X --value Y --type SecureString |
Store a secret |
aws ses verify-email-identity --email-address X |
Verify an email for SES |
aws events describe-rule --name RULE |
Check an EventBridge schedule |
aws events disable-rule --name RULE |
Temporarily pause a schedule |
| File | Purpose |
|---|---|
terraform/main.tf |
Entry point: provider + module calls |
terraform/variables.tf |
All configurable variables |
terraform/terraform.tfvars |
Your actual values (git-ignored) |
terraform/terraform.tfstate |
Terraform's state (git-ignored, sensitive) |
terraform/modules/*/ |
Individual module definitions |
lambda/*/lambda_function.py |
The actual Python code that runs on AWS |
lambda/*/requirements.txt |
Python dependencies for each Lambda |
deploy.ps1 |
One-click build and deploy script |
help.ps1 |
Interactive command reference |