2018 SCaLE 16x Container Days
Container Days
March 8, 2018 - March 11, 2018
Container Days is a set of tracks and events co-located at SCALE. The event offers indepth technical presentations on containers, orchestrators and service meshes. Join the Los Angeles container community and dive into Kuberentes, Mesos, Istio and more.
Hands-On Intro to Kubernetes with kubectl
Audience: Developer
Topic: Container and Virtualization
Synopsis
Learn Kubernetes basics through a series of hands-on command-line interactions.
This interactive session involves using kubectl to learn about kubernetes APIs and the basic abstractions provided for modeling distributed solutions. By the end of this workshop you’ll be deploying, scaling, and automating container-based solutions using open source tools for distributed computing.
To follow along, bring a laptop with: http://bit.ly/scale-k8s#/laptop-setup
Presentation Link
Kubernetes Basics with kubectl at #SCaLE16X
Introduction to the U-Boot bootloader
Audience: Developer
Topic Embedded
Synopsis
U-Boot is the universal bootloader used on a vast majority of embedded systems, development kits, products and so on. This session is an introduction into the U-Boot bootloader, including a hands-on part, and covers practical topics like identifying that the board is running U-Boot, accessing and exploring the U-Boot shell, including advanced scripting techniques to make life easier, obtaining information about the current hardware, accessing busses and storage and finally booting the kernel. Furthermore, since every embedded project has it's unique set of requirements, U-Boot customization topics are briefly touched at the end of the session.
Running Containerized Workloads in an on-prem Datacenter
Steven Wong
Audience: Intermediate
Topic: Container and Virtualization
Synopsis
Over the past decade virtualization moved from upstart, to mainstream, to become the dominant form of application deployment in a datacenter. Containerization is much newer but is growing as a means to deploy applications in a more portable fashion, with better support for horizontal scale out to handle elastic demands.
This talk will cover options and best practices for using containers and open source platforms to deploy applications and backing services, in a way that will run in an on premises datacenter, but still be portable, largely unchanged, to public clouds.
We will survey and compare popular orchestrators (Apache Mesos, Cloud Foundry, DC/OS, Kubernetes , Docker Swarm) and address what is possible, enhancements underway now, and limitations that still exist.
The menu of options for on-premises deployment is almost too rich by some measures (number of choice of distributions, deployment tools, etc). Just as few organizations find it viable to start using Linux by building the kernel from scratch, from GitHub source, a practical production worthy deployment plan will likely start with a distribution, but will also need to consider day 2+ operations like patching, version upgrades, and node count growth.
We will sort of the options for deploying container orchestrators on bare metal (without a mandated virtualization layer) hardware, and over on prem virtualization in a broad, vendor neutral comparison of features and attributes.
Notes
There are many different flavors of orchestrators
PaaS vs CaaS vs IaaS
Platform as a Service
Container as a Service
Infrastructure as a Service
i.e. virtual machines
Principles of mapping workloads to dynamic resources all trace their heritage back to Google's Borg project
What's your ideal datacenter?
Deploy from open source / commercial distro
Bare metal / Iaas / CaaS / PaaS / SaaS
Many organizations will find they want "platform-opinionated"
Differences between public cloud vs on premise
Public cloud has a nice user interface
Dashboards are already built
On-prem Kubernetes, out of the box, will not have dashboards
Networking choices
Network considerations
App level flow control
i.e. a camera might be collecting data more quickly than the image processing app can handle; consider having the image processing app signal to the camera to drop frames instead of filling up system memory
Storage choices
"There is no such thing as a 'stateless' architecture. It's just someone else's problem. -Jonas Boner
For an on-prem datacenter, you are responsible for maintaining the "backing store"
Avoid database monoliths
Container advantages make sense for stateful
Container Storage Interface (CSI)
Typhoon (https://github.com/poseidon/typhoon)
Good for a home lab, legit Kubernetes cluster
Be wary of installers that download at deploy time
Why Johnny can't Container: Tips for Application On-boarding to OpenShift & Kubernetes
Audience: Developer
Topic: Container and Virtualization
Synopsis
Johnny Developer has taken 3 months to move his JBoss App to your OpenShift or Kubernetes cluster. WHY?! With a little understanding and a few tips tips and tricks, on-onbarding an app to Containers could take just a few hours.
We'll also discuss how and when to break a Monolith into a Microservice. Developing a process & practice around Application Migration. Key process features of Application On-boarding to help ensure Johnny Developer's success. And, how to scale these processes when you have hundreds or thousands of applications to migrate.
Notes
Build short, small, iterative parts
We call this Agile today
Know Your References
The Unix Philosphy - The McIlroy Definition
"Write programs that do one thing and do it well."
"Write programs to work together."
Nothing new in the industry, but something we've forgotten
The Suckless Philosophy
"Ultra-minimalist"
Minimalism produces higher quality, more performant, software
DJB - Daniel J. Bernstein
His focus is security
Minimize trusted code and design clean interfaces
"Minimizing privilege is not the same as minimizing the amount of trusted code, and does not move us any closer ..."
Felix von Leitner
Long-time Linux software developer
Writing Small and Fast Software - Presentation
"A good test whether your interface is too generic is: If you leave the names of the arguments away, is it still obvious what each argument does?"
The Twelve-Factor App
Help inform app developers on the apps they're developing
The DevOps Reading List
How to get developers to writing with containers in mind
2 Pronged Attack: Tactical & Strategic
Start with a checklist
Pick a base image which is closest to your application platform or framework
Use tools that make building your container easier (automation)
Build it up-front
Use s2i in OpenShift and OpenShift Origin
With s2i, standing up an app can be as easy as four lines
Even more powerful is a binary s2i deployment using JBoss EAP
Use a CI/CD tool like Jenkins
Use Templates for describing your app
Understand how to use your base image to build your app
It allows to make the mapping between VM to containers
Pull out (parameterize) all of the config from internals of your app
Why?
Build once, deploy anywhere and run in any container
How?
Have app startup/inialization based on external factors and information
Your code shouldn't need to be rebuilt for DEV or QA or PROD
Be comfortable with creating ENV vars, ConfigMaps, Secrets, Volume Mounts
Want the container to be as static as possible
Pull out hard-coded IPs, paths, credentials, etc
This is how you save testing time
Configure your logging to write everything to stdout
The model in Kubernetes is to write to stdout
As opposed to writing to an app specific logging file
Tactical
Strategic: Guidance for your Enterprise
You must measure!
What do you want to optimize?
Development cycle?
Deployment cycle?
Quality?
Standardization?
"T-shirt sizes"
Small/big app?
Failures?
Bugs?
Lead time between changes?
Use a checklist to standardize on-boarding activities
Take inventory of the current development/deployment lifecycle
Take inventory of your candidate on-boarding applications
Start with simple apps
Checklist should allow you to take inventory of your applications architecture ports, dependencies, TLS, startup, monitoring, HA, etc
Anything that makes your app peculiar
Prioritize based on value to the business
Don't be afraid sto start with simple apps
Create templates for your technology stacks
Create an expert on-boarding group
This usually leads to success
Want to have a group of 4-5 who work together and have elevated access to have visibility into other applications to help developers onboard their apps
Train the trainers
This team needs to understand the technology stack at a deep level
OpenShift
Kubernetes
Develop a way to disseminate knowledge
Templates, tutorials, training, wikis, etc
JBoss EAP, Tomcat/JWS Tips
Your apps must run in STandalone Mode; Domain Mode is not usable
Clustering is done via JGroups instead
Be sure your pom.xml does not create dor or source jars
JNDI connections should be externalized in standalone-openshift.xml
Property vars with '.' neet to be renamed with underscores
APP.NAME => APP_NAME
If you need to overwrite startup command line parameters, use the right ENV var
JBoss EAP - JAVA_APTS_APPEND
To use ENV vars in your standalone-openshift.xml config file
${env.MYAPP_VAR}
You Got Your VM In My Container
Audience: Beginner
Topic: Container and Virtualization
Synopsis
Of course we're used to running containers on top of virtual machines, but did you know you can also run virtual machines in containers, or as containers? And containers on VMs in containers on VMs? That may sound crazy, but there's plenty of reasons to intermix containers and VMs ... truly they are Two Great Tastes that Taste Great Together.
In this talk, we'll go over the current state of the art in mixing containers and VMs, including technologies like KubeVirt, ClearContainers, and OpenStack-on-Kubernetes. We'll go over some of the reasons you might have for both mixing and layering. And then we'll wrap up with a practical demonstration of running a Kubernetes database service using Kubevirt.
Notes
VM and Containers are natural enemies?
Containers-in-VMs
i.e. Docker on AWS
Run VMs-in-containers
Specifically to this talk, VMs-on-Kubernetes
Why would we want this?
VMs are awesome, but they require infra-like shared storage and HA management services
With multiple hosts, need orchestration
Example lab: Hyper-Converged oVirt + Gluster
oVirt for virtualization
Gluster for storage
Kubernetes purpose to run generic workloads
Easy to run stoarge, management, and authentication on Kubernetes
Container-native storage has continued to progress
Gluster CNS
Ceph (Rook)
Virtualization was tricky and brittle a few years ago
Progress has been made
KubeVirt
An Add0on that extends Kubernetes to support scheduling of VM workloads alongside container workloads
KubeVirt anatomy
Custom Resource Definitions
New virt-specific object types added to the kube apiserver
Manage like other kube objects, using kubectl
virt-controller
One per cluster
A kubernetes operator
Watches the apiserver for VM objects
Creates pods for new VMs to like in
virt-handler
One of these runs on every node
Takes over the VM pod once the virt-controler creates it
virt-launcher
Each VM gets a virt-launcher pod
This is the pod that the virt-controller creates initially, and the virt-handler takes over
Storage
Kubernetes takes care of the persistent storage
pv volumes
Ephermeral volumes
A copy-on-write volume
Registry disk volumes
Cloud-init nocloud volumes
Networking
Regular pod networking
Every kubernetes pod gets its own IP address
Expose ports with kube services
Virtualization security
Container Isolation
Namespaces
SECCOMP
SE-Linux
Bubblewrap
De-escalate priviledges
Clair & OpenSCAP
Major class of issues: Kernel Exploits
Containers share the same kernel as the host OS
Any hole that is exploited in one container can be used to access all other containers
Resource usage
Resource limits are more of a suggestion, rather than a hard limit
Containers can use more than the resource limit and the host OS will come along at some later time to limit it
kata containers
Goal is to combine the advantages of virtualization and containers
Minimal VM wrapper
Linus-based
Contains microkernel and not much else
Sub-second startup
Plugs into kubernetes with CRI-O
CRI: Container Runtime Interface
Kubelet > CRI shim
How Kata works
Kubernetes calls CRI
CRI calls CRI-O
CRI-O calls kata-runtime
The kata-runtime needs to be pre-installed on each node
kata-runtime starts QEMU mini-VM
mini-VM runs Kube pod
Startup Times are greatly reduced compared to starting a full VM
Enables Trusted/Untrusted
Can be used as an option
TrustedSandbox controls if it'll be run as a regular container or a kata-container
Limitations
Pre-alpha
Launched December 2016
Still debating fundamental architecture
It is a project to watch
Linux guest only
Doesn't have any features
Doesn't have access to system resources
No special devices
Can't support VM apps
Other Technologies in this space
Virtlet
Rkt VMs
RancherVM (RIP)
Service Broker
There was a talk on Friday)
Super Duper Hyper Convergence
Use kubernetes to manage VMs
Give VMs access to Kube resources
Add VM security to containers
Run VMs on containers on VMs!
Links
Container Orchestration Nightmares
Audience: Intermediate
Topic: Container and Virtualization
Synopsis
A lot of talks will tell you how to setup a system correctly. This talk is about what not to do with your Apache Mesos and DC/OS cluster.
We will share some of our favorite/scariest support stories covering typical system-setup, configuration, and application pitfalls for new (and not-so-new) Mesos and DC/OS operators. And, we will give some hints about how to debug those pitfalls if you do encounter them, resulting in fewer nightmares.
Notes
Common patterns that are not ideal for container orchestration systems
Avoid creating your own scripts, but use an existing container orchestration system
Addresses:
Container scheduling
Placement
Preplicatoin/scaling
Resurrection
Rescheduling
RUlling delployment
Upgrades
Downgrades
Collocation
Resource management
Memory
CPU
GPU
Volumes
Ports
IPs
Images/Artifacts
Service management
Labels
Groups/namespaces
Dependencies
Load balancing
Readiness checking
Assumption Applications
Apps should not rely on external assumptions
dockerentry.sh create default config
Implement a short test/retry loop for external connections
Twelve-factor app
Dependencies
Config
Backing services
Build, release, run
Processes
Concurrency
Monitoring
Monitoring to detect problems early
OOM
JVM garbage collection times
During garbage collection, your app will be unavailable
Metrics
Application
Container
Cluster
Want to correlate these metrics with the host OS
Consider percentiles over average
Averages/min/max may not be that informative
Health Checks
Specify health checks carefully
Different options
Mesos vs Marathon
Command vs HTTP
Impacts load=balancers and restarts
Readiness checks
Defines the first time an app is healthy and ready
This can guard against prematurely killing a container because it is still coming online
Immutable container images
Use tagged container images
i.e. specify the version of a package so it doesn't just pull from master
Keep tagged images immutable!
Private container registries
Private container registries should be running from within the cluster and not pulling from the internet
Repeatable container builds
Use repeatable builds for images
Including FROM clause
Keep images minimal
Multistage build
From scratch
UI Deployments
Want control what is running on your infrastructure
Force the developer to go through a CI/CD system so they can never push code onto the production system
Use (Marathon) endpoints for deployments!
Version (and track) your app definitions!
Backup State
Services state
Cluster state
CI/CD Deployments
Prevent developers from accessing the system directly
Developer just pushes changes to version control
The pipeline will then automatically take the changed code
Networking
Consider networking challenges
Connectivity
Load-balancing
Layer 4 vs 7
Consider performance penalty of layer 7
Service discovery
Isolation
Resource Constraints
One of things that are difficult to get right
Don't want to over-allocate and be wasteful
DOn't want to under-allocate and the application runs slowly
Memory constraints are hard limits
Consider overhead (e.g., Java)
Difficult to approximate
Monitor
Disk Usage
Docker and logs are great at filllng up disk space!
Images
Container
Cleanup docker instances and images!
Monitor available disk space!
Zookeeper/etcd Cluster Size
Even number node size is not good
Don't pick a node size of 4
Pick either 3 or 5
Zookeeper quorum (i.e., #Masters) should be odd!
(Typically) Production size of 5 is optimal
Adding more than 5 nodes slows down writes
NoSQL Datastores
Consider the semantics of your datastore!
ACID vs Base
Model your data and queries accordingly!
Removing Stateful Frameworks
Follow the uninstall instructions!
Reservations and Zookeeper state!
state.json
Container vs VMs
Be aware of different isolation semantics!
Container Definition...
What is the difference between container image and instance?
container runtime
!= container image
!= container instance
!= container runtime b
Write Once Run Anywhere
Not necessarily true
Java (<9) not groups aware
threads for GC
Memory
Java Meets Container
Development
Java app packaged in a container
Production
10 JVM container ona 32 core box
By default, each of the JMV containers will allocate 32 cores for each JRE, 32 threads for ForkJoinPool, etc
Linux Distributions
If possible, use officially tested distributions by the container orchestration you're using
Especially for DC/OS
Services on the same node...
Be careful when running services outside Mesos but on the same cluster
Adjust resources accordingly
Spreading Out Master Nodes
Be careful when distributing Master nodes across high latency links
Different AWS AZ is ok, different region probably not
Worker nodes typically don't have to worry about latency, but be careful with the master nodes
Agent Attributes
Set agent attributes when starting an agent
This will impact scheduling decisions
Do not change for running agents
Best to take down the agent, change the attributes, then restart
Software Upgrades
Follow upgrade instructions
Backup
Explicit control of versions
Cluster Upgrades
Check state before
Follow upgrade instructions
Automation
Remember to backup
Upgrade procedure (with DC/OS)
Before upgrading
Make sure cluster is healthy!
Perform backup
a. ZK
a. Replicated logs
A. other state
REview release notes
Generate install bundle
During
Master rolling upgrade
Agent rolling upgrade
Framework upgrades
Day 2 Operations
This is how to keep your operation running after it is up an running
Configuration Updates
scaling
reconfiguration
Binary upgrades
Cluster maintenance
Backup
Restore
Restart
Monitor progress of operations
Debug any runtime blockages
Metrics
Cluster utilization
Resource utilization
Bottlenecks
Production Checklist
Last updated