We all like to make our lives easier in technology with tooling (well most of us do), be that CI/CD/monitoring/scans/git repos, however, how can we deploy these functions in a way that makes our lives easier? If you ask this same question to a group of architects/engineer you will get a long list of answers, my answer is…Bosh!

The reason for Bosh is covered in this blog post.

At this stage I do not want to be prescriptive of which tools are needed, just that I need something to make my deployment of these tools easier. OK, thats a bit miss-leading, I ultimately need to stand up Concourse for some CI/CD pipeline fun and already know that Bosh is a good fit due to a previous role :-)

Where do I start? Based on experience, I know that many options exist for this. Things like bosh-init, Bosh Bootloader (BBL), BUCC all exist to help with this. They each have their pros and cons, however, my preferred method at the moment is bosh-deployment which will allow me to use any IaaS, it has useful options for things like no Internet/or proxy access to the Internet and is also pretty generic rather than focussing on deploying a specific tool (like concourse).

vSphere Deployment

To start off with I want to work in an on-prem environment using vSphere. So my first step would be to get a jumpbox (in my case I picked an Ubuntu image) up and running with the following things installed;

I will also assume that the vSphere environment is up and running and you have the required topology, credentials, firewalls already defined.

Now I can just follow the bosh-deployment instructions and if it no longer quite works, refer to the official docs.

Firstly create a working directory for the bosh deployment

$ mkdir o-bosh && cd o-bosh

Then pull down the bosh-deployment repo

$ git clone https://github.com/cloudfoundry/bosh-deployment

The repo contains a good baseline manifest/template on how to get bosh up and running and will just need to pass a few parameters. Remember what I said about making my life easier? I am a firm believer of not always reinventing the wheel and I look to reuse things where applicable.

The create-env option is used. It will take the manifest we have and implement the desired configuration. A key file is created that bosh create-env needs to keep track of resources created for future alterations. We also need to use something called a manifest operation to tell bosh to use the vSphere CPI and also define what variables need to be passed in. What does this look like? The -h flag can guide us;

$ bosh create-env -h
Usage:
  bosh [OPTIONS] create-env [create-env-OPTIONS] PATH

Application Options:
  -v, --version                  Show CLI version
      --config=                  Config file path (default: ~/.bosh/config) [$BOSH_CONFIG]
  -e, --environment=             Director environment name or URL [$BOSH_ENVIRONMENT]
      --ca-cert=                 Director CA certificate path or value [$BOSH_CA_CERT]
      --sha2                     Use sha256 checksums. Requires recent director and stemcells.
      --client=                  Override username or UAA client [$BOSH_CLIENT]
      --client-secret=           Override password or UAA client secret [$BOSH_CLIENT_SECRET]
  -d, --deployment=              Deployment name [$BOSH_DEPLOYMENT]
      --json                     Output as JSON
      --tty                      Force TTY-like output
      --no-color                 Toggle colorized output
  -n, --non-interactive          Don't ask for user input

Help Options:
  -h, --help                     Show this help message

[create-env command options]
      -v, --var=VAR=VALUE        Set variable
          --var-file=VAR=PATH    Set variable to file contents
      -l, --vars-file=PATH       Load variables from a YAML file
          --vars-env=PREFIX      Load variables from environment variables (e.g.: 'MY' to load MY_var=value)
          --vars-store=PATH      Load/save variables from/to a YAML file
      -o, --ops-file=PATH        Load manifest operations from a YAML file
          --state=PATH           State file path

[create-env command arguments]
  PATH:                          Path to a manifest file

Succeeded

Now we have the background, lets just get something running (the below is an example and you should reflect the settings for your own environment);

bosh create-env bosh-deployment/bosh.yml \
    --state state.json \
    --vars-store ./creds.yml \
    -o ~/bosh-deployment/vsphere/cpi.yml \
    -v director_name=o-bosh \
    -v internal_cidr=10.193.177.0/24 \
    -v internal_gw=10.193.177.1 \
    -v internal_ip=10.193.177.201 \
    -v network_name="VM Network" \
    -v vcenter_dc=Datacenter \
    -v vcenter_ds=LUN01 \
    -v vcenter_ip=10.193.177.11 \
    -v vcenter_user="administrator@vsphere.local" \
    -v vcenter_password=GkTzepNWtbtw! \
    -v vcenter_templates=o-bosh-templates \
    -v vcenter_vms=o-bosh-vms \
    -v vcenter_disks=o-bosh-disks \
    -v vcenter_cluster=Cluster \
    -v vcenter_rp=RP01

Commit the state thats created and the manifest files to git (or some other safe location that multiple people can access).

What happens if I want to see what the deployment manifest will look like without making the deployment? Well, bosh int is your best friend.

bosh int ~/bosh-deployment/bosh.yml \ ................

Logging in

After the deployment has finished you should be able to login to your director;

bosh alias-env o-bosh -e 10.193.177.201 --ca-cert <(bosh int creds.yml --path /director_ssl/ca)

$ bosh -e o-bosh env

Alternatively you can set export BOSH_ENVIRONMENT=o-bosh once instead of using –environment flag for each command. You could always make use of direnv.

Internet Restrictions

In the majority of enterprise customers I have deployed bosh direct, Internet access has been a bit of an issue. This means a couple of things;

Default DNS servers are Google
Bosh ‘stuff’ is on the Internet

To get around the Google DNS issue you can simply use an operation file to alter the default settings

    -o ~/bosh-deployment/misc/dns.yml -v internal_dns=[10.193.177.2]

The default mainfest trys to pull files (stellcells, releases etc.) from the Internet. The bosh release location can be found in bosh.yml;

releases:
- name: bosh
  version: "264.7.0"
  url: https://s3.amazonaws.com/bosh-compiled-release-tarballs/bosh-264.7.0-ubuntu-trusty-3468.21-20180125-031202-579399892-20180125031207.tgz?versionId=gnR.RVKDRH_Vk63jxsGBOhcHOZ9p8qCJ
  sha1: 3850c68124bf5bce3cfb1433cce52e2d67741d94

To solve this I have created a few operations files (PR) to overide these settings here.

To make use of these you need to include the opertions files in your deployment and also specify the lcations of the files.

    -o ~/bosh-deployment/misc/no-internet-access/vsphere-cpi.yml \
    -o ~/bosh-deployment/misc/no-internet-access/stemcell.yml \
    -o ~/bosh-deployment/local-bosh-release.yml \
    
    -v local_bosh_release="~/bosh/bosh-264.6.0.tgz" \
    -v local_vsphere_cpi="~/bosh/bosh-vsphere-cpi" \
    -v local_stemcell="~/bosh/bosh-stemcell-3468.13-vsphere-esxi-ubuntu-trusty-go_agent.tgz" \

Other operation options

If you browse the GitHub repo you will find a number of operations files that can be used to add UAA/Credhub/Syslog etc. functionality. They are mostly in the /, /misc or /experimental directories.

The operations file concept is pretty cool because it means that you can use a base line deployment template and then add/remplace functionality as required.

Tidier way to Deploy

I have found that to make deployments more reusable, the above command can be seperated into two discrete components;

Deployment Script
Variable file

This means that you can use the same deployment script for all deployments and just point to the variables file for the specific environment you are working on.

Deployement Script

#!/bin/bash

if [ -z "$1" ]
then
  echo "Please supply an enviroment"
  echo "example is deploy-bosh.sh development"
  exit 2
fi

WORKDIR=/pcf/workspace/deployments/${1}
BOSHDIR=/pcf/workspace/bosh-deployment
PATCHDIR=/pcf/workspace/deployments/patches

bosh create-env ${BOSHDIR}/bosh.yml \
    --state ${WORKDIR}/bosh-state.json \
    --vars-file=${WORKDIR}/params-bosh.yml \
    --vars-store=${WORKDIR}/bosh-creds.yml \
    -o ${BOSHDIR}/misc/dns.yml \
    -o ${BOSHDIR}/vsphere/cpi.yml \
    -o ${BOSHDIR}/local-bosh-release-tarball.yml \
    -o ${BOSHDIR}/misc/no-internet-access/vsphere-cpi.yml \
    -o ${BOSHDIR}/misc/no-internet-access/stemcell.yml

Variables file

The variables file is just plain old Yaml.

director_name: o-bosh 
internal_cidr: 10.193.177.0/24 
internal_gw: 10.193.177.1 
internal_ip: 10.193.177.201 
network_name: "VM Network" 
vcenter_dc: Datacenter 
vcenter_ds: LUN01 
vcenter_ip: 10.193.177.11 
vcenter_user: "administrator@vsphere.local" 
vcenter_password: GkTzepNWtbtw! 
vcenter_templates: o-bosh-templates 
vcenter_vms: o-bosh-vms 
vcenter_disks: o-bosh-disks 
vcenter_cluster: Cluster 
vcenter_rp: RP01

Each environment should have its own file and these should be stored in a git repo or backed up file system (you may want to use git-crypt if using a git repo).

You now have a great place to deploy other services to make your life easy, once you upload a cloud-config that is.

As a "Platform Operator" I want a stable platform to deploy tooling.