为什么要从WebServer01 ping超时到DBServer？

Question

我已经使用下面的Terraform代码创建了自定义VPC。

我已经在下面应用了Terraform模块，其中：

1. 添加公共和私有子网
2. 配置IGW
3. 添加NAT网关
4. 为Web服务器和数据库服务器添加SG

应用此功能后，我无法从公共ec2实例向私有ec2实例ping / ssh。

不确定是否缺少什么。

# Custom VPC
resource "aws_vpc" "MyVPC" {
  cidr_block       = "10.0.0.0/16"
  instance_tenancy = "default" # For Prod use "dedicated"

  tags = {
    Name = "MyVPC"
  }
}

# Creates "Main Route Table", "NACL" & "default Security Group"

# Create Public Subnet, Associate with our VPC, Auto assign Public IP
resource "aws_subnet" "PublicSubNet" {
  vpc_id                  = aws_vpc.MyVPC.id # Our VPC
  availability_zone       = "eu-west-2a"     # AZ within London, 1 Subnet = 1 AZ
  cidr_block              = "10.0.1.0/24"    #  Check using this later > "${cidrsubnet(data.aws_vpc.MyVPC.cidr_block, 4, 1)}"
  map_public_ip_on_launch = "true"           # Auto assign Public IP for Public Subnet
  tags = {
    Name = "PublicSubNet"
  }
}

# Create Private Subnet, Associate with our VPC
resource "aws_subnet" "PrivateSubNet" {
  vpc_id            = aws_vpc.MyVPC.id # Our VPC
  availability_zone = "eu-west-2b"     # AZ within London region, 1 Subnet = 1 AZ
  cidr_block        = "10.0.2.0/24"    #  Check using this later > "${cidrsubnet(data.aws_vpc.MyVPC.cidr_block, 4, 1)}"
  tags = {
    Name = "PrivateSubNet"
  }
}

# Only 1 IGW per VPC
resource "aws_internet_gateway" "MyIGW" {
  vpc_id = aws_vpc.MyVPC.id
  tags = {
    Name = "MyIGW"
  }
}

# New Public route table, so we can keep "default main" route table as Private. Route out to MyIGW
resource "aws_route_table" "MyPublicRouteTable" {
  vpc_id = aws_vpc.MyVPC.id # Our VPC

  route {                    # Route out IPV4
    cidr_block = "0.0.0.0/0" # IPV4 Route Out for all
    # ipv6_cidr_block = "::/0"        The parameter destinationCidrBlock cannot be used with the parameter destinationIpv6CidrBlock # IPV6 Route Out for all
    gateway_id = aws_internet_gateway.MyIGW.id # Target : Internet Gateway created earlier
  }
  route {                                           # Route out IPV6
    ipv6_cidr_block = "::/0"                        # IPV6 Route Out for all
    gateway_id      = aws_internet_gateway.MyIGW.id # Target : Internet Gateway created earlier
  }
  tags = {
    Name = "MyPublicRouteTable"
  }
}

# Associate "PublicSubNet" with the public route table created above, removes it from default main route table
resource "aws_route_table_association" "PublicSubNetnPublicRouteTable" {
  subnet_id      = aws_subnet.PublicSubNet.id
  route_table_id = aws_route_table.MyPublicRouteTable.id
}

# Create new security group "WebDMZ" for WebServer
resource "aws_security_group" "WebDMZ" {
  name        = "WebDMZ"
  description = "Allows SSH & HTTP requests"
  vpc_id      = aws_vpc.MyVPC.id # Our VPC : SGs cannot span VPC

  ingress {
    description = "Allows SSH requests for VPC: IPV4"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]   # SSH restricted to my laptop public IP <My PUBLIC IP>/32
  }
  ingress {
    description = "Allows HTTP requests for VPC: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]   # You can use Load Balancer
  }
  ingress {
    description      = "Allows HTTP requests for VPC: IPV6"
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    ipv6_cidr_blocks = ["::/0"]
  }
  egress {
    description = "Allows SSH requests for VPC: IPV4"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]   # SSH restricted to my laptop public IP <My PUBLIC IP>/32
  }
  egress {
    description = "Allows HTTP requests for VPC: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    description      = "Allows HTTP requests for VPC: IPV6"
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    ipv6_cidr_blocks = ["::/0"]
  }
}

# Create new EC2 instance (WebServer01) in Public Subnet
# Get ami id from Console
resource "aws_instance" "WebServer01" {
  ami           = "ami-01a6e31ac994bbc09"
  instance_type = "t2.micro"
  subnet_id     = aws_subnet.PublicSubNet.id
  key_name = "MyEC2KeyPair"   # To connect using key pair
  security_groups = [aws_security_group.WebDMZ.id]    # Assign WebDMZ security group created above
  # vpc_security_group_ids = [aws_security_group.WebDMZ.id]
  tags = {
    Name = "WebServer01"
  }
}

# Create new security group "MyDBSG" for WebServer
resource "aws_security_group" "MyDBSG" {
  name        = "MyDBSG"
  description = "Allows Public WebServer to Communicate with Private DB Server"
  vpc_id      = aws_vpc.MyVPC.id # Our VPC : SGs cannot span VPC

  ingress {
    description = "Allows ICMP requests: IPV4" # For ping,communication, error reporting etc
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["10.0.1.0/24"]    # Public Subnet CIDR block, can be "WebDMZ" security group id too as below
    security_groups = [aws_security_group.WebDMZ.id]        # Tried this as above was not working, but still doesn't work
  }
  ingress {
    description      = "Allows SSH requests: IPV4" # You can SSH from WebServer01 to DBServer, using DBServer private ip address and copying private key to WebServer
    from_port        = 22                          # ssh ec2-user@Private Ip Address -i MyPvKey.pem     Private Key pasted in MyPvKey.pem
    to_port          = 22                          # Not a good practise to use store private key on WebServer, instead use Bastion Host (Hardened Image, Secure) to connect to Private DB
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  ingress {
    description = "Allows HTTP requests: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  ingress {
    description      = "Allows HTTPS requests : IPV4"
    from_port        = 443
    to_port          = 443
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  ingress {
    description      = "Allows MySQL/Aurora requests"
    from_port        = 3306
    to_port          = 3306
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  egress {
    description = "Allows ICMP requests: IPV4" # For ping,communication, error reporting etc
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["10.0.1.0/24"] # Public Subnet CIDR block, can be "WebDMZ" security group id too
  }
  egress {
    description      = "Allows SSH requests: IPV4" # You can SSH from WebServer01 to DBServer, using DBServer private ip address and copying private key to WebServer
    from_port        = 22                          # ssh ec2-user@Private Ip Address -i MyPvtKey.pem     Private Key pasted in MyPvKey.pem chmod 400 MyPvtKey.pem
    to_port          = 22                          # Not a good practise to use store private key on WebServer, instead use Bastion Host (Hardened Image, Secure) to connect to Private DB
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  egress {
    description = "Allows HTTP requests: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  egress {
    description      = "Allows HTTPS requests : IPV4"
    from_port        = 443
    to_port          = 443
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  egress {
    description      = "Allows MySQL/Aurora requests"
    from_port        = 3306
    to_port          = 3306
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
}


# Create new EC2 instance (DBServer) in Private Subnet, Associate "MyDBSG" Security Group
resource "aws_instance" "DBServer" {
  ami           = "ami-01a6e31ac994bbc09"
  instance_type = "t2.micro"
  subnet_id     = aws_subnet.PrivateSubNet.id
  key_name = "MyEC2KeyPair"   # To connect using key pair
  security_groups = [aws_security_group.MyDBSG.id] # THIS WAS GIVING ERROR WHEN ASSOCIATING
  # vpc_security_group_ids = [aws_security_group.MyDBSG.id]
  tags = {
    Name = "DBServer"
  }
}

# Elastic IP required for NAT Gateway
resource "aws_eip" "nateip" {
  vpc = true
  tags = {
    Name = "NATEIP"
  }
}

# DBServer in private subnet cannot access internet, so add "NAT Gateway" in Public Subnet
# Add Target as NAT Gateway in default main route table. So there is route out to Internet.
# Now you can do yum update on DBServer

resource "aws_nat_gateway" "NATGW" {         # Create NAT Gateway in each AZ so in case of failure it can use other
  allocation_id = aws_eip.nateip.id          # Elastic IP allocation
  subnet_id     = aws_subnet.PublicSubNet.id # Public Subnet

  tags = {
    Name = "NATGW"
  }
}

# Main Route Table add NATGW as Target

resource "aws_default_route_table" "DefaultRouteTable" {
  default_route_table_id = aws_vpc.MyVPC.default_route_table_id

  route {
    cidr_block     = "0.0.0.0/0"              # IPV4 Route Out for all
    nat_gateway_id = aws_nat_gateway.NATGW.id # Target : NAT Gateway created above
  }

  tags = {
    Name = "DefaultRouteTable"
  }
}

为什么要从WebServer01 ping超时到DBServer？

Answer 1

为此，

DBServer上的安全组需要允许出口到

DBServer的安全组或包含它的CIDR。

aws_instance.DBServer使用

aws_security_group.MyDBSG

。aws_instance.WebServer01使用

aws_security_group.WebDMZ

。aws_security_group.WebDMZ]上的出口规则如下：

  egress {
    description = "Allows SSH requests for VPC: IPV4"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]   # SSH restricted to my laptop public IP <My PUBLIC IP>/32
  }
  egress {
    description = "Allows HTTP requests for VPC: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    description      = "Allows HTTP requests for VPC: IPV6"
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    ipv6_cidr_blocks = ["::/0"]
  }

那些出口规则的意思是：

允许SSH到所有IPv4主机
允许HTTP到所有IPv4和IPv6主机
ICMP未列出，因此ICMP回显请求将在它离开
aws_security_group.WebDMZ
之前被删除。对于ENI 
aws_instance.WebServer01

。，这应该在VPC FlowLog中显示为REJECT。将此egress规则添加到

aws_security_group.WebDMZ

应该可以解决：

  egress {
    description      = "Allows ICMP requests: IPV4" # For ping,communication, error reporting etc
    from_port        = -1
    to_port          = -1
    protocol         = "icmp"
    cidr_blocks      = ["10.0.2.0/24"]
  }

DBServer可能不响应ICMP，因此进行此更改后，您仍然可能会看到超时。引用VPC FlowLog将有助于确定差异。如果您在VPC FlowLog中看到ICMP流的ACCEPT，则说明问题是DBServer不响应ICMP。

aws_security_group.WebDMZ]中的任何内容都不能阻止SSH，因此，问题一定出在其他地方。
aws_security_group.MyDBSG]上的入口规则如下。
  ingress {
    description = "Allows ICMP requests: IPV4" # For ping,communication, error reporting etc
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["10.0.1.0/24"]    # Public Subnet CIDR block, can be "WebDMZ" security group id too as below
    security_groups = [aws_security_group.WebDMZ.id]        # Tried this as above was not working, but still doesn't work
  }
  ingress {
    description      = "Allows SSH requests: IPV4" # You can SSH from WebServer01 to DBServer, using DBServer private ip address and copying private key to WebServer
    from_port        = 22                          # ssh ec2-user@Private Ip Address -i MyPvKey.pem     Private Key pasted in MyPvKey.pem
    to_port          = 22                          # Not a good practise to use store private key on WebServer, instead use Bastion Host (Hardened Image, Secure) to connect to Private DB
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
那些出口规则的意思是：
允许来自公共子网（10.0.1.0/24）的所有ICMP
允许从公共子网（10.0.1.0/24）使用SSH

SSH应该正常工作。 DBServer可能不接受SSH连接。

假设您的DBServer是10.0.2.123，那么如果未运行SSH，则从运行ssh -v 10.0.2.123的WebServer01中看起来像这样：

debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to 10.0.2.123 [10.0.2.123] port 22.
debug1: connect to address 10.0.2.123 port 22: Operation timed out
ssh: connect to host 10.0.2.123 port 22: Operation timed out

参考VPC FlowLog将有助于确定差异。如果您在VPC FlowLog中看到SSH流的ACCEPT，则问题是DBServer不接受SSH连接。

旁边：风格推荐
您应该使用aws_security_group_rule资源，而不是嵌入式出口和入口规则。

关于安全组和安全组规则的说明：Terraform当前既提供独立的安全组规则资源（单个入口或出口规则），也提供具有嵌入式定义的入口和出口规则的安全组资源。目前，您不能将安全组与嵌入式规则一起使用，并且不能与任何安全组规则资源一起使用。这样做会导致规则设置冲突并覆盖规则。

使用内嵌规则使用旧式

aws_security_group

：

resource "aws_security_group" "WebDMZ" {
  name        = "WebDMZ"
  description = "Allows SSH & HTTP requests"
  vpc_id      = aws_vpc.MyVPC.id

  ingress {
    description = "Allows HTTP requests for VPC: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]   # You can use Load Balancer
  }

  egress {
    description = "Allows SSH requests for VPC: IPV4"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

并用此现代风格的

aws_security_group

替换，每个规则的

aws_security_group_rule

资源：

resource "aws_security_group" "WebDMZ" {
  name        = "WebDMZ"
  description = "Allows SSH & HTTP requests"
  vpc_id      = aws_vpc.MyVPC.id
}

resource "aws_security_group_rule" "WebDMZ_HTTP_in" {
  security_group_id = aws_security_group.WebDMZ.id

  type        = "ingress"
  description = "Allows HTTP requests for VPC: IPV4"
  from_port   = 80
  to_port     = 80
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "WebDMZ_SSH_out" {
  security_group_id = aws_security_group.WebDMZ.id

  type        = "egress"
  description = "Allows SSH requests for VPC: IPV4"
  from_port   = 22
  to_port     = 22
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}