Infrastructure

This is a quick note on how to set up a Spark cluster in standalone mode. This is useful if you want to setup a cluster for your own development purposes or if you just want to do it for fun—for more serious use cases, Spark clusters should be setup on top of YARN or Kubernetes. You will need to create a few VMs: one VM for the cluster manager and then one or more VMs where the executors will be running....

There are many resources out there which—rightly—focus on teaching Puppet using simple architectures, e.g. a single node, masterless setup. However, I found less info on how to set up Puppet with an Agent-Master architecture. This short article gives a very high-level overview of the main steps. It is not a self-sufficient tutorial by any means—but I am hoping it will still be useful for those trying to understand the big picture....