*23 Jan 2019* One of the most annoying parts of setting up [Elastic Search](https://eu-west-2.console.aws.amazon.com/es) on AWS is the way access is configured for that service. From what i've gathered you have one of two options to protect it: 1. Make sure it's running internally only (i.e. not internet facing and only available to the other services on the same VPC) 2. Completely open to the internet and eventually relying (if you can make it work) on the [Cognito](https://eu-west-2.console.aws.amazon.com/cognito) authentication service Typically you would, as we did, go for the path of least resistance and greater security, which is to make it accessible only to the internal VPC. But that means you now cannot access [Kibana](https://www.elastic.co/products/kibana) to interact with ES. The way we went around this at [Drover](https://www.joindrover.com) was to setup a container running an Nginx proxy that provides basic auth protected access to kibana (using SSL). Here's how we did it: ## 1. The container ### The docker container definition (Dockerfile) ```bash FROM nginx:1.15 RUN apt-get update -qq && apt-get install -y --allow-unauthenticated openssh-server nano curl # PROXY PORT EXPOSE 80:80 # SSH EXPOSE 22:22 # SSH access RUN mkdir /root/.ssh RUN touch /root/.ssh/authorized_keys RUN chmod 700 /root/.ssh RUN chmod 644 /root/.ssh/authorized_keys RUN mkdir /var/run/sshd # SSH public keys RUN echo "ssh-rsa <public_key> void@mypc" >> /root/.ssh/authorized_keys RUN mkdir /auth RUN chmod 701 /auth COPY /kibana.htpasswd /auth RUN chmod 644 /auth/kibana.htpasswd # TODO: Rest of envs RUN chown -R www-data /auth RUN rm /etc/nginx/conf.d/default.conf COPY /nginx.conf /etc/nginx/conf.d/kibanas.conf COPY /start.sh / RUN chmod +x /start.sh CMD /start.sh ``` Notes: - Nginx listens on port 80, but we also expose port 22 so we can ssh into the container for troubleshooting - The appropriate authorized keys are set into the container for passwordless login - The password file is copied into the appropriate location so it can be later read by nginx - The nginx config file is also copied to the correct location - Docker will run the start script when booting ### The start script (**start.sh**) ```bash #!/bin/bash service ssh start nginx ``` ### The nginx config file (**nginx.conf**) ```ruby server { listen 80; server_name kibanaz.myapp.com; auth_basic "Restricted"; auth_basic_user_file /auth/kibana.htpasswd; # redirect / location = / { rewrite ^ /_plugin/kibana/ redirect; } location / { proxy_http_version 1.1; proxy_set_header Authorization ""; # Don't pass auth to kibana proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; proxy_pass https://<your_url_to_es>/; } } ``` Notes: - Make sure you have a DNS entry (either [route53](https://console.aws.amazon.com/route53/home) or another) that points to this server name - The **\<your_url_to_es\>** should be the Elastic Search url you can find in the ES AWS service for the particular instance you want access to - This single proxy could potentially serve more than one Kibana/ES access, just add more server definitions ### The password file (kibana.htpasswd) ```bash <username_you_want>:<your_passwd_encrypted_password> ``` Notes: - These files can be generated with **htpasswd** (there's even an [online generator, here](http://www.htaccesstools.com/htpasswd-generator/)) ## 2. Terraforming makes it easier If you read one of my previous posts, you know that [terraform is an excellent tool](/terraform-infrastructure-as-code) for setting up infrastructure. So that's what we use to setup the container in AWS ECS. I will skip the cluster setup and instead focus on the task-definitions and services. Setting up resources using terraform is something that you can easily learn about in [their documentation](https://www.terraform.io/docs/). ### Task definition ```ruby data "aws_ecs_task_definition" "kibana" { task_definition = "${aws_ecs_task_definition.kibana.family}" } resource "aws_ecs_task_definition" "kibana" { family = "kibana" task_role_arn = "arn:aws:iam::<your_amazon_account_id>:role/ecsTaskExecutionRole" execution_role_arn = "arn:aws:iam::<your_amazon_account_id>:role/ecsTaskExecutionRole" network_mode = "awsvpc" placement_constraints = [], requires_compatibilities = ["FARGATE"] cpu = 256 memory = 2048 container_definitions = <<DEFINITION [ { "name": "KibanaContainer", "image": "<your_amazon_account_id>.dkr.ecr.eu-west-2.amazonaws.com/kibana-proxy:latest", "cpu": 0, "portMappings": [ { "containerPort": 80, "hostPort": 80, "protocol": "tcp" } ], "essential": true, "mountPoints": [], "volumesFrom": [], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/KibanaProxy", "awslogs-region": "eu-west-2", "awslogs-stream-prefix": "ecs" } } } ] DEFINITION } ``` Notes: - This definition makes certain assumptions, as follows - There is already an initial empty task definition named Kibana in your fargate account - **\<your_amazon_account_id\>** should be filled with whatever is your account id - You have built, tagged and pushed the previously defined docker image into amazon's container repository and the name and tag of the image are **kibana-proxy:latest** - This is deploying to the **eu-west-2** region (**London**), change it to closer to home if you need ### The service ```ruby resource "aws_security_group" "kibana_sg" { name = "kibana_sg" description = "Kibana proxy security group" vpc_id = "${var.vpc_id}" ingress { from_port = 80 to_port = 80 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] ipv6_cidr_blocks = ["::/0"] } ingress { from_port = 22 to_port = 22 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] ipv6_cidr_blocks = ["::/0"] } # allow all outbound traffic egress { from_port = "0" to_port = "0" protocol = "-1" cidr_blocks = ["0.0.0.0/0"] ipv6_cidr_blocks = ["::/0"] } tags { Name = "kibana_sg" } } resource "aws_ecs_service" "kibana" { name = "kibana" cluster = "${module.support_cluster.cluster_id}" task_definition = "${aws_ecs_task_definition.kibana.family}:${max("${aws_ecs_task_definition.kibana.revision}", "${data.aws_ecs_task_definition.kibana.revision}")}" desired_count = "${var.replicas}" launch_type = "FARGATE" load_balancer { target_group_arn = "${aws_alb_target_group.kibana-target-group.arn}" container_port = 80 container_name = "KibanaContainer" } network_configuration { subnets = "${var.subnet_ids}" assign_public_ip = true security_groups = ["${aws_security_group.kibana_sg.id}"] } } ``` Notes: - **vpc_id** is a variable being passed into this module, you can hard code it with the VPC you have setup for your cluster - **cluster_id** is the name of the cluster that you create to house this service (for instance Kibana, or support) - The service will try to use the latest task definition available (hence the previous need to have already a task definition with that name, otherwise it will fail the first time it runs) - **replicas** is the number of containers you want to be running (used by the load balancer) - **subnet_ids** is an array of subnets that you create for your cluster (typically one public for the load balancers and one private for the containers) - **target_group_arn** is setup from the load balancers definition (up next) ### Load balancing ```ruby resource "aws_alb" "kibana-load-balancer" { name = "kibana-lb" security_groups = ["<public_facing_security_group_id>"] subnets = "${var.subnet_ids}" tags { Name = "kibana-load-balancer" } } resource "aws_alb_target_group" "kibana-target-group" { name = "kibana-tg" port = "80" protocol = "HTTP" vpc_id = "${var.vpc_id}" target_type = "ip" health_check { healthy_threshold = "5" unhealthy_threshold = "2" interval = "60" matcher = "200,302,301,401" path = "/_plugin/kibana/" port = "traffic-port" protocol = "HTTP" timeout = "15" } tags { Name = "kibana-target-group" } } resource "aws_alb_listener" "kibana-lb-listener" { load_balancer_arn = "${aws_alb.kibana-load-balancer.arn}" port = "443" protocol = "HTTPS" ssl_policy = "ELBSecurityPolicy-2016-08" certificate_arn = "arn:aws:acm:eu-west-2:<your_amazon_account_id>:certificate/<certificate_key_for_lb_domain>" default_action { target_group_arn = "${aws_alb_target_group.kibana-target-group.arn}" type = "forward" } } resource "aws_alb_listener" "kibana-lb-redirect-listener" { load_balancer_arn = "${aws_alb.kibana-load-balancer.arn}" port = "80" protocol = "HTTP" default_action { type = "redirect" redirect { port = "443" protocol = "HTTPS" status_code = "HTTP_301" } } } ``` Notes: - **public_facing_security_group_id** is the security group id for the load balancer, which should be open to the outside (internet) - **vpc_id** is the public vpc - **certificate_key_for_lb_domain** - basically the whole **certificate_arn** is supposed to be the arn of the certificate you generate for the domain. Say you have the domain **myapp.com** in route53. You should generate a certificate for the load balancer to use that contains the myapp.com domain so that SSL works properly. Make sure it's consistent with the **nginx.conf** definition as well ### Bonus - auto DNS ```ruby resource "aws_route53_record" "dns-kibana" { zone_id = "<myapp.com_zone_id>" name = "kibanaz.myapp.com" type = "CNAME" ttl = "300" records = ["${aws_alb.kibana-load-balancer.dns_name}"] } ``` Notes: - This resource creates a DNS entry for the load balancer in the provided zone id. The load balancer module defined previously will feed into the record being setup here so that it all ties together. ## The conclusion When you have it all setup, you can just fire up terraform apply and it should take care of setting up everything for you, assuming you already pushed the docker image into your account's docker repository. Once everything is running **kibanaz.myapp.com** should direct you to the kibana interface, asking for the Basic Auth username/password you setup before allowing you inside. There is some glue work to do to make this all work, but i know you can do it. Just reference the [documentation on terraform](https://www.terraform.io/docs/) and it should be super easy. Happy Kibaning. [[tags/amazon|amazon]] [[tags/aws|aws]] [[tags/cloud|cloud]] [[tags/elastic_search|elastic_search]] [[tags/kibana|kibana]] [[tags/software_development|software_development]]