Running Containers Exercise

Installing Prometheus to our cluster

make helm_update

helm repo add stable https://charts.helm.sh/stable && \
helm repo update
"stable" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈

make prom

Expand me...

Deploying Fluentbit for our logs

“FireLens gives you a simplified interface to filter logs at source, add useful metadata and send logs to almost any destination. You can now stream logs directly to Amazon CloudWatch, Amazon Kinesis Data Firehose destinations such as Amazon Elasticsearch, Amazon S3, Amazon Kinesis Data Streams and partner tools. Using Amazon ECS task definition parameters, you can select destinations and optionally define filters for additional control and FireLens will ingest logs to target destinations.”

Update the EKS node group with permissions to push to the log group

make deploy-fluent-iam

Then Deploy fluent bit make deploy-fluent-bit

Expand me...

We now have deployed the Fluentbit plugin, we should see logs in our CloudWatch log Groups for our application.

Deploy Falco

make deploy-falco

Expand me...

You can see in Cloud Watch Falco Logs

https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/falco/log-events/alerts

Now we can deploy the demo app to raise alerts in Cloud watch

kubectl apply -f falco-demo/nodejs-bad-rest-api/falco-demo.yml

Get the Loadbalancer EXTERNAL-IP for the falco-demo

kubectl get svc falco-demo

NAME         TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)        AGE
falco-demo   LoadBalancer   10.100.249.117   adb9d277014bd4bdd9eef8e94f000f1b-1234566.us-west-2.elb.amazonaws.com   80:32527/TCP   4h9m

Access urls under /api/exec/<cmd> to run arbitrary commands.

Run the following commands to execute arbitrary commands like ’ls’, ‘pwd’, etc:

$ curl http://$LOAD_BALANCER_API/api/exec/ls

demo.yml
node_modules
package.json
README.md
server.js

$ curl http://$LOAD_BALANCER_API:8181/api/exec/pwd

Try to run bash via /api/exec/bash, falco sends alert.

If you try to run bash via /api/exec/bash, falco will generate an alert:

$ curl http://$LOAD_BALANCER_API:8181/api/exec/bash

falco          | 22:26:53.536628076: Warning Shell spawned in a container other than entrypoint (user=root container_id=6f339b8aeb0a container_name=express_server shell=bash parent=sh cmdline=bash )

Navigate to CloudWatch Logs