SERP (Search Engine Results Page)
Getting started
- rename .env.example to .env and fill in the required environment variables
- run
npm installto install dependencies - run
npm run devto start the development server
AWS Setup
architecture-beta
service browser(internet)[SERP frontend app in browser]
group cdn[AWS Account]
service cft(logos:aws-cloudfront)[CDN] in cdn
group globalComponents [AWS Account global components]
group unifiedSearch [AWS Account Search backend]
service tGateway1(logos:aws-transit-gateway) [Transit Gateway] in vpc1
service tGateway2(logos:aws-transit-gateway) [Transit Gateway] in vpc2
group vpc1[VPC not set up yet] in globalComponents
service serpBackend(logos:aws-lambda)[SERP backend in Lambda] in vpc1
group vpc2[VPC without internet] in unifiedSearch
service searchBackend(logos:aws-fargate)[Search API in Fargate] in vpc2
browser:R -- L:cft
cft:R -- L:serpBackend
serpBackend:R -- L:tGateway1
tGateway1:R -- L:tGateway2
tGateway2:R -- L:searchBackendLambda in VPC
To enable Lambda with a frontend app running in the custom managed VPC of your AWS account, it must be configured in a certain way. AWS docs quote:
"To access resources in a VPC in your account, you can add a VPC configuration to a function. This restricts the function to resources within that VPC, unless the VPC has internet access."
https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc-internet.html
The following steps must be taken to enable Lambda to access the Internet within VPC: (assuming the following prerequisite is already prepared: you have a VPC with private subnets and public subnets in one or more availablity zones of a region, route tables. The route tables are each "associated" with respective subnets)
1. Check Lambda policy
2. Create an Internet gateway
3. Attach the Internet gateway to VPC
4. Create NAT gateways
5. Route private subnets to NAT
6. Route public subnets to IGW
7. Deploy Lambda to private subnets only
8. Remove Lambda from public subnets
9. Configure Security groups
Target resource map should loook as follows:
flowchart TD
subgraph Regionn1["Region eu-central-1"]
subgraph VPC["VPC"]
subgraph PublicSubnetAZa["Subnet public AZ-a"]
NATA["NAT Gateway AZ-a (connectivity type 'public' + ElasticIPs)"]
RouteTablePublicAZa["Route Table AZ-a (0.0.0.0/0 -> IGW)"]
end
subgraph PrivateSubnetAZa["Subnet private AZ-a"]
LambdaAZa["Lambda in AZ-a"]
RouteTablePrivateAZa["Route Table AZ-a (0.0.0.0/0 -> NAT Gateway a)"]
end
subgraph PublicSubnetAZb["Subnet public AZ-b"]
NATB["NAT Gateway AZ-b (connectivity type 'public' + ElasticIPs)"]
RouteTablePublicAZb["Route Table AZ-b (0.0.0.0/0 -> IGW)"]
end
subgraph PrivateSubnetAZb["Subnet private AZ-b"]
LambdaAZb["Lambda in AZ-b"]
RouteTablePrivateAZb["Route Table AZ-b (0.0.0.0/0 -> NAT Gateway b)"]
end
subgraph PublicSubnetAZc["Subnet public AZ-c"]
NATC["NAT Gateway AZ-c (connectivity type 'public' + ElasticIPs)"]
RouteTablePublicAZc["Route Table AZ-c (0.0.0.0/0 -> IGW)"]
end
subgraph PrivateSubnetAZc["Subnet private AZ-c"]
LambdaAZc["Lambda in AZ-c"]
RouteTablePrivateAZc["Route Table AZ-c (0.0.0.0/0 -> NAT Gateway c)"]
end
IGW["Internet Gateway (IGW)"]
end
end
Internet[Internet]
%% Flow AZ-a
LambdaAZa --> RouteTablePrivateAZa
RouteTablePrivateAZa --> NATA
RouteTablePublicAZa --> IGW
%% Flow AZ-b
LambdaAZb --> RouteTablePrivateAZb
RouteTablePrivateAZb --> NATB
RouteTablePublicAZb --> IGW
%% Flow AZ-c
LambdaAZc --> RouteTablePrivateAZc
RouteTablePrivateAZc --> NATC
RouteTablePublicAZc --> IGW
%% Flow Internet
IGW --> InternetCheck Lambda policy
Lambda needs the policy "AWSLambdaENIManagementAccess" to be added Lambda policy contains the following:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeSubnets",
"ec2:DetachNetworkInterface",
"ec2:AssignPrivateIpAddresses",
"ec2:UnassignPrivateIpAddresses"
],
"Resource": ["*"]
}
]
}
or include it in your custom policy.
https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html#configuration-vpc-permissions
Create Internet gateway
Create an Internet gateway (IGW) in yor VPC. (VPC -> Internet gateways)
https://docs.aws.amazon.com/vpc/latest/userguide/working-with-igw.html#create-igw
Attach the Internet gateway to VPC
https://docs.aws.amazon.com/vpc/latest/userguide/working-with-igw.html#attach-igw
Create NAT gateways
Create NAT gateways to connect each of the private subnets with it, the connectivity type must be "public", assign public elastic IPs ( ENI=elastic network interfaces will be created automatically)
[!WARNING]
It is important that the internet gateway is created and connected to the VPC before starting to create NAT gateways. Otherwise the NAT is created but with the status “failed”, the same status occurs if the added elasticIP is already in use by another NAT.
Route private subnets to NAT
Route each private subnet to the NAT gateways in VPC, add a route with 0.0.0.0/0 as destination and NAT geteway, you created before, as a target.
https://docs.aws.amazon.com/vpc/latest/userguide/vpc-example-private-subnets-nat.html
Route public subnets to IGW
Route each public subnet to IGW, add a route table with 0.0.0.0/0 as destination and IGW as a target.
Deploy Lambda to private subnets
Connect Lambda to private subnets only (otherwise it will not work if Lambda is also directly linked to public subnets)
(the how-to with CDK for Remix and filtering of private subnets follows...)
Remove Lambda from public subnets
Check if Lambda is connected to all available subnets of VPC, remove public subnets links
(follows...)
Configure Security groups
Security groups with inbound and outbound rules for the internet traffic must be created