1. No category

OSU DevOps Bootcamp Documentation
Release 0.0.1
OSU OSL
OSU LUG
October 11, 2014
Contents
1
2
Contents:
1.1 Get Involved . . . . . . . . . . . . . . . . . . . . . . .
1.2 Policies . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Curriculum . . . . . . . . . . . . . . . . . . . . . . . .
1.4 BootCamp Guide . . . . . . . . . . . . . . . . . . . . .
1.5 Recommended Resources . . . . . . . . . . . . . . . .
1.6 FAQ . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.7 What is DevOps? . . . . . . . . . . . . . . . . . . . . .
1.8 Purpose of DevOps BootCamp . . . . . . . . . . . . . .
1.9 Vagrant . . . . . . . . . . . . . . . . . . . . . . . . . .
1.10 DevOps Bootcamp Introduction . . . . . . . . . . . . .
1.11 Lesson 1: The Very Basics . . . . . . . . . . . . . . . .
1.12 Lesson 2: Single System Fundamentals . . . . . . . . .
1.13 Lesson 2: Homework . . . . . . . . . . . . . . . . . . .
1.14 Lesson 3: Editors & Version Control . . . . . . . . . .
1.15 Lesson 3: Intro to Git . . . . . . . . . . . . . . . . . .
1.16 Lesson 3: GitHub . . . . . . . . . . . . . . . . . . . .
1.17 Lesson 4: Scripting, Troubleshooting, & Workflow . . .
1.18 Lesson 5: Web Applications . . . . . . . . . . . . . . .
1.19 Lesson 6: Boot and the Filesystem Hierarchy . . . . . .
1.20 Lesson 7: Databases . . . . . . . . . . . . . . . . . . .
1.21 Lesson 8: Security & Authentication . . . . . . . . . .
1.22 Lesson 8: Web application security . . . . . . . . . . .
1.23 Lesson 9: Networking . . . . . . . . . . . . . . . . . .
1.24 Lesson 10: Networking part 2 . . . . . . . . . . . . . .
1.25 Lesson 11: Devops & Configuration Management Intro
1.26 Lesson 12: Configuration Management . . . . . . . . .
1.27 Lesson 13: Contributing to Open Source . . . . . . . .
1.28 Lesson 14: Review . . . . . . . . . . . . . . . . . . . .
1.29 Lesson 16: Email . . . . . . . . . . . . . . . . . . . . .
Indices and tables
Bibliography
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
3
5
7
7
8
9
9
10
12
15
28
39
39
46
50
54
63
68
81
88
100
104
111
125
134
142
147
149
155
157
i
ii
OSU DevOps Bootcamp Documentation, Release 0.0.1
The OSL is kicking off DevOps BootCamp this year with the brand new DevOps DayCamp! DevOps DayCamp is
a dual-track day with one track to help inexperienced attendees get started with DevOps, as well as a second track
comprised of a hands-on hackathon with educational sessions throughout the day for the more advanced DevOps
crowd. Advanced sessions will be given by industry professionals and will include Ansible, Travis CI and Docker. For
more information, checkout http://devopsbootcamp.osuosl.org/daycamp/.
DevOps BootCamp is an OSU Open Source Lab and OSU Linux Users Group program that aims to boost our impact
and outreach while shrinking the skills gap for OSU students interested in DevOps.
The OSU Open Source Lab provides quality hosting for more than 160 open source projects. Along with delivering over 460 terabytes of open source data a month, the OSL employs and mentors students to be expert systems
administrators and software developers ready to enter the industry. The OSL encounters more demand for qualified
students than the employees graduating each year can supply. We’d like to give back to OSU and the wider open
source community by training interested students from all disciplines in the real-world skills necessary for success as
software developers and systems engineers. To this end, we’ll be offering an extracurricular training program with 2 to
3 sessions per month through the school year, in which current OSL students and software professionals will teach all
topics that are essential for success. Although hands-on labs will be offered later in the program, no initial familiarity
with software development or Linux will be assumed.
This program is based on the PSU Braindump, where for 21 years they have been turning PSU students into sysadmins
who have rapidly been flowing into the growing number of startups in Portland.
The DevOps BootCamp program will be dynamically shaping itself based on the feedback we get from interested
students.
Contents
1
OSU DevOps Bootcamp Documentation, Release 0.0.1
2
Contents
CHAPTER 1
Contents:
1.1 Get Involved
1.1.1 Mailing list
Join the mailing list for updates.
1.1.2 IRC
Join us on irc.freenode.net in #devopsbootcamp (students will be setting up an IRC network for the
program in a later lesson).
1.1.3 Website & Curriculum
If you’d like to help edit this site, email devopsbootcamp or ping anyone in #devopsbootcamp on Freenode with
your github username to get access to the web site repo. You’ll also want to learn the ReStructured Text markup
language to edit the site, if you don’t already know it.
1.1.4 Meetings
We typically meet weekly in KEC 1007 or 1005 depending on the room schedules. Please check back later for when
our first meeting in the Fall will be. Recurring meeting times following the first meeting will be decided based on the
availability of those who show up for the first meeting. Last year we met on Thursdays from 6-8pm.
1.2 Policies
1.2.1 “The Deal”
You get:
• Mentorship from students and professionals with advanced skills in software development and systems administration
• Professional connections in the software industry
3
OSU DevOps Bootcamp Documentation, Release 0.0.1
• A welcoming environment to start learning in if you’ve always wanted to learn about software development and
systems administration but have never been sure where to start
• An opportunity to fill in any gaps in your knowledge if you’re a self-taught coder or sysadmin
• The skills to build and deploy open source software or to contribute to existing projects
Why we’re offering this:
• The OSL gets a larger pool of candidates to recommend to companies interested in recruiting OSL students
• The OSL gets to work with a wider variety of students, which fits better with its new status as part of the school
of EECS
• The open source community gets more contributors to its projects
1.2.2 Attendance
We will schedule BootCamp meetings for a time that works for a majority of attendees. Each meeting’s curriculum
will build on what you learned the previous session, so if you miss a meeting you’re responsible for studying the basics
of its content on your own. To facilitate this, we will provide resources for independent study about each topic that we
cover.
BootCamp leaders will be available at times outside of the regular meetings to help answer any questions about the
training program’s content. We will not spend class time reviewing material for those who skip a lecture. If you attend
a lesson and don’t understand something from it, you’re welcome to ask in class, because others very likely have the
same concern. But if you want to make up a class you skipped, it’s disrespectful to those who attended to spend their
class time on questions which you could resolve on your own time.
1.2.3 Laptops
As the course progresses, you will need a laptop. We hope and recommend that you will decide to set up your laptop
to dual-boot to Linux as the course progresses, but but it’s not required. If you don’t own a laptop and are an OSU
student you can check out a laptop from the OSU Library for at least 24 hours at a time.
Realistically, as long as it’s new enough to boot from USB and connect to wireless networks, your laptop’s specifications don’t matter much. If it’s so new that its UEFI configuration prevents you from dual-booting with Linux, it
will be powerful enough to run virtual machines. If it’s old enough to be unable to run VMs but still has wireless
connectivity, we’ll teach you how to ssh to a remote server to perform more computationally intensive tasks.
If you are not an OSU student and do not have access to a working laptop, contact the DevOps BootCamp organizers
and we’ll see whether we can arrange to loan you something for meetings.
1.2.4 Target Audience
Our goal is to make the DevOps BootCamp program accessible to students and community members from all different
backgrounds. You have to want to learn, be willing to ask questions whenever you don’t understand something, and
be open to making time to play with the cool tools/toys which we’ll be teaching you about.
Fluency in English is strongly recommended, because it’s the language in which the course will be taught as well as
the language of almost all open source technical documentation and code comments.
4
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.3 Curriculum
The curriculum will be constantly evolving as we develop this course. The first unit will be an overview of what
students will need to know in order to understand the deeper technical concepts which will be covered later.
Here’s what we’re hoping to mentor and teach students about:
1. Linux Basics
2. Basic System Administration
3. Basic FOSS Development Methodologies
4. Base infrastructure services for any organization (DNS, Email, etc)
5. Building a mock infrastructure for a mock company from top to bottom
Note: Note that the curriculum for future weeks is constantly under development. Lesson titles are included to
provide an overview of what we’ll cover, but specific content is likely to get moved around between lessons as we
refine the curriculum. Slide content will be linked here as soon as it’s drafted, but only finalized 1-2 weeks before the
lessons when it’s presented.
1.3.1 Lesson 1: The Very Basics
Shell, virtualbox+vagrant, IRC.
• 6pm 11/7/2013, KEC1007. 34 people attended.
• Lesson 1 Video
• Lesson 1 Slides
1.3.2 Lesson 2: Single System Fundamentals
File permissions, users, groups, and package management
• 6pm 11/14/2013, KEC1001. 27 people showed up.
• Lesson 2 Video
• Lesson 2 Slides
1.3.3 Lesson 3: Editors & Git
Vim, Emacs, and Git for version control
• 6pm 11/21/2014, BEXL. 21 people showed up.
• Part 1, text editors video
• Part 2, version control video
• Lesson 3 Slides
1.3. Curriculum
5
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.3.4 Lesson 4: Scripting & Troubleshooting
Python and Bash. Overview of troubleshooting/debugging skills.
• 6pm 01/9/2014, KEC1007.
• Scripting and Troubleshooting Video
• Lesson 4 Slides
1.3.5 Lesson 5: Services & Deploying a Web App
• 6pm 1/16/2014, KEC1007
• Services & Web App video 1/2
• Services & Web App video 2/2
• Lesson 5 Slides
1.3.6 Lesson 6: Boot Process & Filesystem Hierarchy
• 6pm 1/30/3014 – 14 attendees; time conflicts with Intel and Reddit founder
• Boot & Filesystem Video
• Lesson 6 slides
1.3.7 Lesson 7: Databases
• 6pm 2/13/2014 (moved from 2/6 due to snow)
• Lesson 7 slides
1.3.8 Review of VM and App Setup
• 6pm 2/20/2014
• No video, hands-on... 14 people showed up
• Review Slides
1.3.9 Lesson 8: Security & Authentication
• 6pm 2/27/2014 – 22 people present
• Lesson 8 video
• Lesson 8 slides
1.3.10 Lesson 9: Networking overview
• 6pm 4/10/2014
• Lesson 9 video
• Lesson 9 slides
6
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.3.11 Lesson 10: DNS
• 6pm 5/1/2014
• Lesson 10 video
• Lesson 10 slides
1.3.12 Lesson 11: Automation and DevOps
• 6pm 5/8/2014
• Lesson 11 video
• Lesson 11 slides
1.3.13 Lesson 12: Configuration Management
• 6pm 5/15/2014 – 8 students, 10 people total
• Lesson 12 video
• Lesson 12 slides
1.3.14 Lesson 13: Open Source & Hands-On
• 6pm 5/22/2014
• Lesson 13 Video
• Lesson 13 slides
1.4 BootCamp Guide
This is the DevOps BootCamp Guide. It will cover the various topics during the sessions in more detail.
• Vagrant
1.4.1 Administrative Stuff
• link for finding rooms
1.5 Recommended Resources
1.5.1 Scripting
• The Advanced Bash Scripting Guide
• Learn Python The Hard Way
1.4. BootCamp Guide
7
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.5.2 Systems Administration Tutorials
• LinuxTraining
– This is the resource which the OSL uses to quickly train systems administrators. The whole book is
here.
• OpsSchool is working on a tutorial somewhat like DevOps BootCamp, but it’s currently incomplete. Consider
contributing to it!
• Linux Command Line Tips: Become a Master is a quick start guide to get you going with bash
1.5.3 Open Source
The following resources are used as textbooks in Carlos Jensen’s CS419 Open Source class:
• The Cathedral and the Bazaar (Eric S. Raymond)
• Producing Open Source Software (Karl Fogel)
• Open Advice (Lydia Pintscher)
1.5.4 Culture
• The Jargon File
• ESR’s guide to asking good questions
• Geek Feminism and Systers are worth looking into if you enjoy discussing diversity or IT’s lack thereof.
1.6 FAQ
Email questions to [email protected] if they aren’t answered here!
1.6.1 Is this a for-credit class?
No, although the curriculum is based on CS312, Linux System Administration, which hasn’t been offered for several
years.
1.6.2 What does it cost?
Financially, it’s free. However, to benefit from the hands-on learning opportunities that we’ll be offering, you should
plan to commit 5 or more hours per week to attending lectures and playing with the tools and technologies that the
lectures will discuss.
1.6.3 I don’t know about computers! Can I still join?
YES! If you already knew all this stuff, you’d be teaching rather than studying it. If you think you don’t know enough,
try anyway – in the worst case you’ll discover that your self-assessment was correct, and in the best case you’ll become
a successful open source contributor.
8
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.6.4 When will meetings be?
Weekday evenings, with a day of the week being chosen during the first meeting. We try to choose a day that works
best for most students but it’s always difficult to find something that works for everyone. For example, last year we
had meetings on Thursdays.
1.6.5 Is this only for students?
No, anyone from the community can participate. Parts of the earlier lessons will assume access to student resources,
but we can work with non-students on an individual basis to get them access to equivalent tools. Community members
can use the Visitor wifi network when on campus.
1.6.6 What’s the difference between BootCamp and regular LUG meetings?
Regular LUG meetings are usually targeted toward an intermediate to expert audience, and make no attempt to comprehensively teach a set of skills across multiple meetings. We might teach about version control one week, privacy
tools the next, and a new open source project by a local company the next.
DevOps BootCamp will provide continuity and comprehensive open source education that’s outside the scope of
regular LUG meetings. As you progress through Bootcamp, you’ll understand more and more of the technical content
of LUG talks.
1.6.7 I wasn’t able to participate during Fall term. Can I still join?
Yes! Please read through lessons 1-3 (linked from the curriculum page) and ask in IRC if you have trouble following
the instructions for them. Catch up on that content and you’ll be ready to join in Winter term.
1.6.8 I have a question?
Ask it on IRC if you have that set up, or email [email protected] or the mailing list.
1.7 What is DevOps?
DevOps (a portmanteau of software development and server operations) represents the integration between software
development and operations engineering that has become increasingly necessary with the emergence of cloud computing. In essense, the jobs are no longer mutually exclusive- developers need more operations-based knowledge in order
to understand how their application will be deployed, test locally and preemptively address potential security risks,
while systems administrators need to know developer-based materials in order to design infrastructure optimally to
fit an application’s needs without wasting capacity and to troubleshoot issues with the increasingly complex software
cycle. Additionally, site reliability engineering and many modern security roles require a background with a balance
between development and operations.
1.8 Purpose of DevOps BootCamp
1.8.1 From front page:
DevOps BootCamp is an OSU Open Source Lab and OSU Linux Users Group program aimed at boosting our impact
and outreach while shrinking the skills gap for OSU students interested in DevOps.
1.7. What is DevOps?
9
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.8.2 From FAQ:
DevOps BootCamp will provide continuity and comprehensive open source education that’s outside the scope of
regular LUG meetings.
1.8.3 “The Deal”
Students get:
• Mentorship from students and professionals with advanced skills in software development and systems administration
• Professional connections in the software industry
• A welcoming environment to start learning in, if you’ve always wanted to learn about software development and
systems administration but never been sure where to start
• An opportunity to fill in any gaps in your knowledge, if you’re a self-taught coder or sysadmin
• The skills to build and deploy open source software, or contribute to existing projects
Why we’re offering this:
• The OSL gets a larger pool of candidates to recommend to companies interested in recruiting OSL students
• The OSL gets to work with a wider variety of students, which fits better with its new status as part of the school
of EECS
• The open source community gets more contributors to its projects
1.8.4 Target Audience
Our goal is to make the DevOps BootCamp program accessible to students and community members from all different
backgrounds. You have to want to learn, and be willing to ask questions whenever you don’t understand something,
and be open to making time to play with the cool tools/toys which we’ll be teaching you about.
1.9 Vagrant
1.9.1 Trying Linux as a virtual machine with Vagrant
In order to create a stable learning environment, we’re going to have everyone use a Devop tool called Vagrant.
Vagrant is typically used to help assist both developers and system engineers in ensuring that their application and
system deployments work predictably. For our purposes we’re going to use it as an easy way for new people to get to
a Linux prompt quickly with no fear of breaking their system.
Vagrant basically interfaces with hypervisors such as VirtualBox with a unified command line interface. This makes
it portable between Mac, Windows and even Linux! Vagrant itself is just a collection of ruby scripts while Virtualbox
does all the virtual machine magic.
We will go in more depth with how to install Linux manually, but for now we’ve done all the hard work and have
create pre-made Linux images.
10
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.9.2 Installing Virtualbox
VirtualBox is a free and open source virtual machine manager that you can install on a variety of operating system
platforms such as Mac and Windows.
1. Go to the VirtualBox download page and download the latest copy of VirtualBox.
2. Go through the installer and use the default settings when prompted.
3. Once completed, see if you can open up the VirtualBox Manager
4. If it asks you to download the latest extensions, go ahead and do that.
1.9.3 Installing Vagrant
Vagrant is a free and open source tool used to build complete devops environments.
1. Go to the Vagrant download page and choose the newest version available.
2. Choose all the defaults during the install if it asks you any questions
1.9.4 Testing Vagrant
1. If you’re on a Mac, open up the Terminal. If you’re on windows, open up a DOS prompt (cmd.exe).
2. Type: vagrant status
3. If there are no errors then Vagrant was installed correctly!
1.9.5 Cloning the Vagrant Repo
We keep all of our vagrant configuration in a git version controlled repository. If you don’t have git installed, please
go download git and install it.
Once you have downloaded and installed git, go to a folder where you want to keep the repository and type the
following:
git clone https://github.com/DevOpsBootcamp/devopsbootcamp-vagrant.git
If installing git is too difficult, you can also download a zip file containing the repository. We will teach you more
about git later in the year, so don’t worry!
1.9.6 Using Vagrant
Now that we have the entire environment working, lets get to playing with Linux! Open the terminal and get to the
directory where the devopsbootcamp-vagrant repo is at. You can run the following commands:
# Start the VM
vagrant up
# SSH into the VM
vagrant ssh
# Stop and halt the VM
vagrant halt
1.9. Vagrant
11
OSU DevOps Bootcamp Documentation, Release 0.0.1
# Destroy and remove the VM
vagrant destroy
Also check out the Vagrant Documentation for more information. You can also always type -h to find out more
information about a command.
1.9.7 Troubleshooting
Depending on how your laptop or computer is confused in the BIOS, you may or may not run into issues getting
Vagrant and VirtualBox to work properly.
Booting the VM takes forever and never completes
The most common problem is when you type vagrant up the system just waits and waits and never finishes the
boot process. The most likely cause is because your system doesn’t have hardware virtualization enabled in the BIOS.
You can either enable the feature or you can disable the requirement in the Vagrantfile.
1. Enabling Hardware Virtualization
Reboot your machine and press the appropriate buttons to get into the BIOS or Setup screen. This is usually done by
hitting a combination of the escape or function keys.
Once you get into the BIOS, find a screen that has some options for the CPU. Each BIOS is different but you are
basically trying to find a feature called Hardware Virtualization. Once you find it, go ahead and enable it and reboot
your system.
2. Disabling HW Virtualization in Vagrant
Open the Vagrantfile with your favorite editor of choice. Find the line that says hwvirtex and remove the #
from the front of the line. Now run vagrant destroy -f and then vagrant up again. This should boot the
VM properly now.
1.10 DevOps Bootcamp Introduction
OSU OSL & OSU LUG
http://devopsbootcamp.osuosl.org
1.10.1 What’s DevOps?
Software Development + Operations Engineering
1.10.2 What’s BootCamp?
• New this year
• New this year
• Free education program
– Mentorship
– Apprenticeship
12
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
– Hands-on training
• NOT:
– OSU class for credit
– student job
1.10.3 Why are we doing this?
• OSL’s move to EECS
• Bridge the “Skills Gap”
• Demand from companies for students
• Build community
1.10.4 What you’ll do:
• Learn:
– Linux systems
– Networking
– Software development
– ...and the other skills you’ll need
• Build:
1.10. DevOps Bootcamp Introduction
13
OSU DevOps Bootcamp Documentation, Release 0.0.1
– Duplicate internet infrastructure
– Graduate to helping with OSL tasks
* work with real open source projects
* solve real problems
1.10.5 Can you do it?
• Probably!
• No background knowledge needed
• Time commitment
– Attend lectures
– Play with what we teach you
– You get out what you put in
• If in doubt. . . try anyway!
1.10.6 Career Paths:
• Hands-on learning & OSL reputation:
– Internships
– Industry connections
– Start career with mid-level/high-level experience
• Justin
– community experience builds skills, professional relationships
– Linux skills helped in grad school
* skills learned there helped land first sysadmin job
– having your own ‘playground’ to experiment with
1.10.7 OSL Hiring:
• Students need basic skills to join
– Systems engineers
– Developers
– Media
• Typically 2-3 years employment
• Alumni
– Google, Rackspace, Intel, Microsoft, Mozilla, Facebook
14
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.10.8 What next?
• Find our site (http://devopsbootcamp.osuosl.org)
• Fill out registration with times available
• Join mailing list
• Tell us about yourself
– Who are you
– What do you want to learn here
– Any background in devops?
1.11 Lesson 1: The Very Basics
1.11.1 Today’s Agenda
• Getting (to) Linux
• The Terminal & Shell
– Scripts, file paths, special characters
• Productivity tricks
– Getting help
1.11. Lesson 1: The Very Basics
15
OSU DevOps Bootcamp Documentation, Release 0.0.1
• IRC
– Vocabulary
– Get connected
– Etiquette
1.11.2 A note about notation
• Variables
– $varname
– <varname>
• Shell prompt
– $
– ‘literal stuff in backticks‘
• foo, bar, baz, username, etc.
1.11.3 How to get (to) Linux
• How many have it already installed?
• Install VM or dual-boot
• When stuck on Windows, use PuTTy:
• Students:
ssh <onidusername>@shell.onid.oregonstate.edu
• flip{1-3} are Engineering servers; less reliable
1.11.4 Trying Linux on a Virtual Machine
Virtual machines act as a full system on a physical machine
• Hypervisors:
– VirtualBox (free)
16
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.11. Lesson 1: The Very Basics
17
OSU DevOps Bootcamp Documentation, Release 0.0.1
– VMWare (mostly free)
– KVM (Linux only hosts)
– Parallels
• Public Cloud Virtual Machines
– Amazon EC2, Rackspace Cloud, Google Compute Engine, etc
• Easy way to test without breaking your machine!
1.11.5 Installing Linux on Virtualbox
Note: Try other distributions if you like to see what’s different. Debian is a great next step to try out.
1. Download and install: https://www.virtualbox.org/wiki/Downloads
2. Grab the latest minimal ISO: http://centos.osuosl.org/6/isos/x86_64/
3. Create VM
(a) New -> Name “CentOS” -> Default Ram -> Default Disk settings
(b) Settings -> Storage -> Empty -> CD/DVD Drive -> Select ISO
(c) Start -> press enter -> Skip media check
4. \o/
1.11.6 Vagrant & VirtualBox
Note: We’re using CentOS as our base image for now but will use Debian later. You can see the gui by uncommenting
the line in the Vagrantfile.
• Vagrant is a tool used with Virtualbox (and other) platforms
• Make a reproducible pre-installed Linux environment
• Download and install: http://www.vagrantup.com/
• Clone our repo, start and access the vm:
# clone
git clone https://github.com/DevOpsBootcamp/devopsbootcamp-vagrant.git
# start up
cd devopsbootcamp-vagrant
vagrant up
# access vm
vagrant ssh
1.11.7 Vagrant cheat sheet
Note: We’ll get into more detail later in how you can access ports on your VMs and other use cases.
18
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
# start
vagrant up
# stop
vagrant halt
# destroy (remove vm)
vagrant destroy
# ssh to the vm
vagrant ssh
Also check out the Vagrant Documentation for more information.
1.11.8 The Terminal
• Used to mean the keyboard+monitor
– Now that’s a crash cart
• Terminal emulator
• Shell: Use bash; others include csh, zsh, tsch
– ~/.bashrc
1.11.9 Basic Shell Commands
Note:
Explain architecture built in commands vs. external binaries
Demo commands Directory movement and file manipulation: Cd, pwd, ls, rm, mv, touch
User info id, whoami, w
1.11. Lesson 1: The Very Basics
19
OSU DevOps Bootcamp Documentation, Release 0.0.1
20
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
Pipes redirection (pipe.txt, redirect.txt)
Special variables $?, $$ (pid.sh), !!, !*, !$
• ls, cd, cat, echo
• invoke/call an installed program
• get help: man <program>
test@x230 ~ $ tree
.
-- Documents
|
-- Code
|
|
-- scripts
|
|
-- test.sh
|
-- School
|
-- Work
-- Pictures
-- manatee.gif
-- turtle.png
6 directories, 5 files
1.11.10 Invoking a script
Note: Permissions discussed later.
$ ls -l
$ chmod +x $filename
Arguments are extra information that you pass to a script or program when you call it. They tell it in more detail what
you want to do.
$ ls -a -l
$ ls -al
1.11. Lesson 1: The Very Basics
21
OSU DevOps Bootcamp Documentation, Release 0.0.1
Why pass arguments on the command line rather than having an interactive mode?
1.11.11 File Paths
• . means current directory
• .. means parent directory
• Tilde (~) means your homedir (/home/$username)
• / separates directories (not \)
• / is root directory, so ~ expands to /home/$username/
• current path appears in your prompt: I’m logged in as the user test on the machine named x230
test@x230 ~ $ ls
Documents Pictures
test@x230 ~ $ cd Documents/
test@x230 ~/Documents $ ls
Code School Work
test@x230 ~/Documents $
Note: root directory is not to be confused with a home directory for the root account
1.11.12 Special Characters
• escape with \ to use them literally
• # means a comment
• ; allows multiple commands per line
• !, ?, *, &&, &
• Regular expressions (we’ll learn more later)
22
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.11.13 Type less
• Reverse-i-search
– ctrl+r then type command
• aliases
– ~/.bashrc
• Tab completion
Automation > Typing > Mouse
1.11.14 Help, get me out of here!
• ctrl+c kills/quits
• ctrl+d sends EOF (end-of-file)
– also means logout
• :q gets you out of Vi derivatives and man pages
– esc - esc - :q if you changed modes
• read what’s on your screen; it’ll help you
1.11.15 Knowledge Check
1.11. Lesson 1: The Very Basics
23
OSU DevOps Bootcamp Documentation, Release 0.0.1
test@x230 ~ $ tree
.
-- Documents
|
-- Code
|
|
-- scripts
|
|
-- test.sh
|
-- School
|
-- Work
-- Pictures
-- manatee.gif
-- turtle.png
6 directories, 5 files
• What user am I logged in as?
• What command did I just run?
• What is my current directory when I run that command?
1.11.16 More about Man Pages
• the manual (rtfm):
$ man <program>
$ man man
• use /phrase to search for phrase in the document; n for next match
• else:
$ <program> --help
1.11.17 Documentation
Man pages, blogs you find by Googling, StackOverflow
• Contribute to community
– Correct it if it’s wrong
– Remind them what newbies don’t know
– Write your own
• For your future self as well
• Start now
1.11.18 Asking for help
It’s okay to ask.
1. What should be happening?
2. What’s actually happening?
3. Google it
24
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
4. Skim the manuals of each component
5. Identify a friend, mentor, or IRC channel who could help
6. When they’re not busy, give them a quick synopsis of points 1 and 2, mentioning what possibilities you’ve ruled
out by searching.
Contributions = expertise + time
Don’t waste experts’ time, but do build your expertise.
1.11.19 IRC
• Internet Relay Chat
• Very old (RFC 1459 May 1993)
• Works on everything (no GUI needed)
• The people you want to listen to are there
1.11.20 A Client
Note: Switche to a terminal and show example
Use irssi in screen
# This step is optional, but persistent IRC is cool
$ ssh <username>@<preferred shell host>
# start Screen
$ screen -S irc
# start your client
$ irssi
# after ending ssh session, to get back:
$ ssh <username>@<preferred shell host>
$ screen -dr IRC
1.11.21 Networks
/connect irc.freenode.net
/nick <myawesomenickname>
/msg nickserv register <password> <email>
/nick <myawesomenickname>
/msg nickserv identify <password>
1.11. Lesson 1: The Very Basics
25
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.11.22 Channels
/join #osu-lug
/join #devopsbootcamp
/list
• tells all channels on network
• Don’t do this on Freenode!
/topic tells you the current channel’s topic
/names tells you who’s here
1.11.23 Commands
• take action with /me does thing‘
• everything else starting with / is a command
/say $thing
/join, /part, /whois <nick>, /msg, /help <command>
Note that nothing shows up in the channel when you run a /whois command; it shows up either in your status buffer
or your conversation with the person.
12:04 -!- _test_ [[email protected]]
12:04 -!- ircname : Example User
12:04 -!- channels : #ExampleChannel
12:04 -!- server
: moorcock.freenode.net [TX, USA]
12:04 -!- hostname : c-50-137-46-63.hsd1.or.comcast.net 50.137.46.63
12:04 -!- idle
: 0 days 0 hours 2 mins 38 secs [signon: Wed Nov 6
12:00:30
2013]
12:04 -!- End of WHOIS
1.11.24 Useful tricks
• Tab-complete works on nicknames. use it.
• Highlight when people say your name
• Symbols are not part of names; they mark status in channel
• Logging (expect it); ‘/set autolog on‘
• chanserv and nickserv are good bots to know
– hamper is also a bot
1.11.25 Screen & Irssi Hints
• Paste with ctrl+shift+v
– PuTTY defaults to right-click to paste
• to get back, screen -dr IRC
• Can you use man screen to find out what the d and r flags mean?
26
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
SCREEN(1)
SCREEN(1)
NAME
screen - screen manager with VT100/ANSI terminal emulation
SYNOPSIS
screen [ -options ] [ cmd [ args ] ]
screen -r [[pid.]tty[.host]]
screen -r sessionowner/[[pid.]tty[.host]]
Manual page screen(1) line 1 (press h for help or q to quit)
1.11.26 Etiquette
• Lurk more
• Don’t ask to ask
– Lure help out of hiding with tasty details of problem
• Show that you’re worth helping
• Read the topic
– /topic
– Output only shows up in your channel, not to everyone else
• Pastebin code
• Choose your nick carefully
1.11.27 Terminology
• ping/pong
• flapping
• tail
• hat
• nick
• netsplit
• kick/ban/k-line
• common emotes
1.11. Lesson 1: The Very Basics
27
OSU DevOps Bootcamp Documentation, Release 0.0.1
– o/ AND \o high fives
– /me & means afk
1.11.28 Review
• What’s Linux?
• How do you open a terminal emulator?
– this varies between window managers
• I have the script test.py. How do I run it??
• How do you list all the files in the current directory?
• Give 2 ways to change directory to your home directory.
• How do you start an irc client?
– How often should you need to start your IRC client?
• How do you reconnect to a screen session?
• Give an example of something which you should not do in IRC
1.12 Lesson 2: Single System Fundamentals
Or, how to be a power user.
1.12.1 Housekeeping
10 minutes to make sure everyone’s Virtualbox instances are up and running
Your opinions:
• Want to meet during dead week vs. have last meeting of term next week?
• Start wk1 of winter term?
28
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.12.2 Today’s Topics
• What are files?
– Permissions/users
• What are user accounts?
– User management
– Root vs. normal
– Groups
• What are packages?
– How to use package managers
– Differences between Linux Distributions
– Other package managers
1.12.3 What are users?
• You, right now
$
$
$
$
whoami
who
w
id
#
#
#
#
your username
who is logged in?
who is here and what are they doing?
user ID, group ID, and groups you’re in
• Not just people: Apache, Mailman, ntp
1.12.4 Users have
• Username
• UID
• Group
• Usually (but not always) password
• Usually (but not always) home directory
1.12.5 Managing users
$
#
$
$
$
cat /etc/passwd
username:x:UID:GID:GECOS:homedir:shell
useradd $USER # vs adduser, the friendly Ubuntu version
userdel $USER
passwd
#
#
$
$
GECOS: full name, office number and building, office phone extension,
home phone number (General Electric Comprehensive Operating System)
chfn # change GECOS information; only works sometimes
finger # tells you someone’s GECOS info
1.12. Lesson 2: Single System Fundamentals
29
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.12.6 Passwords
• /etc/shadow, not /etc/passwd
test@x230 ~ $ ls -l /etc/ | grep shadow
-rw-r----- 1 root shadow
1503 Nov 12 17:37 shadow
$ sudo su $ cat /etc/shadow
daemon:*:15630:0:99999:7:::
bin:*:15630:0:99999:7:::
sys:*:15630:0:99999:7:::
mail:*:15630:0:99999:7:::
# name:hash:time last changed: min days between changes: max days
#
between changes:days to wait before expiry or disabling:day of
#
account expiry
$ chage # change when a user’s password expires
1.12.7 Root/Superuser
• UID 0
• sudo
1.12.8 Acting as another user
$
$
$
$
su $USER
su
sudo su sudo su $USER
#
#
#
#
become user, with
become root, with
use user password
become $USER with
THEIR password
root’s password
instead of root’s
your password
If someone has permissions errors:
• Check that they or their group owns the files
• Check that they have the flag +x to execute
1.12.9 What are groups?
• Manage permissions for groups of users
30
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.12. Lesson 2: Single System Fundamentals
31
OSU DevOps Bootcamp Documentation, Release 0.0.1
$
$
$
$
groupadd
usermod
groupmod
cat /etc/group
root:x:0:
daemon:x:1:
bin:x:2:
sys:x:3:
adm:x:4:
tty:x:5:
# group name:password or placeholder:GID:member,member,member
1.12.10 Hands-On: Users and Groups
Note: To give yourself sudo powers do the following:
1. Add your user to the wheel group using gpasswd.
2. As the root user, use visudo and uncomment this line:
%wheel
ALL=(ALL)
ALL
3. Save the file and now you should have sudo!
We’ll cover sudo in more depth at a later time.
• Create a user on your system for yourself, with your preferred username
• Give your user sudo powers
• Use use su to get into your user account
• Change your password
• Create a directory called bootcamp in your home directory
• Create a group called devops
1.12.11 What are files?
• Nearly everything
• Files have:
– Owner
– Permissions
– inode
– Size
– Filename
test@x230 ~ $ ls -il
total 8
2884381 drwxrwxr-x 5 test test 4096 Nov 6 11:46 Documents
2629156 -rw-rw-r-- 1 test test
0 Nov 13 14:09 file.txt
2884382 drwxrwxr-x 2 test test 4096 Nov 6 13:22 Pictures
32
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.12.12 File extensions
• .jpg, .txt, .doc
• Really more of a recommendation
– File contains information about its encoding
$ file $FILENAME # tells you about the filetype
test@x230 ~ $ file file.txt
file.txt: ASCII text
test@x230 ~ $ file squirrel.jpg
squirrel.jpg: JPEG image data, JFIF standard 1.01
1.12.13 ls -l
• First bit: type
• Next 3: user
• Next 3: group
• Next 3: world
• user & group
$ ls -l
drwxrwxr-x 5 test test 4096 Nov 6 11:46 Documents
-rw-rw-r-- 1 test test
0 Nov 13 14:09 file.txt
drwxrwxr-x 2 test test 4096 Nov 6 13:22 Pictures
chmod and octal permissions
+=====+========+=======+
| rwx | Binary | Octal |
+=====+========+=======+
| --- | 000
| 0
|
| --x | 001
| 1
|
| -w- | 010
| 2
|
| -wx | 011
| 3
|
| r-- | 100
| 4
|
| r-x | 101
| 5
|
| rw- | 110
| 6
|
| rwx | 111
| 7
|
+=====+========+=======+
• u, g, o for user, group, other
• -, +, = for remove, add, set
• r, w, x for read, write, execute
chown, chgrp
user & group
1.12. Lesson 2: Single System Fundamentals
33
OSU DevOps Bootcamp Documentation, Release 0.0.1
# Change the owner of myfile to "root".
$ chown root myfile
# Likewise, but also change its group to "staff".
$ chown root:staff myfile
# Change the owner of /mydir and subfiles to "root".
$ chown -hR root /mydir
# Make the group devops own the bootcamp dir
$ chgrp -R devops /home/$yourusername/bootcamp
1.12.14 Types of files
drwxrwxr-x
5 test
test
4096
Nov 6 11:46 Documents
-rw-rw-r-1 test
test
0
Nov 13 14:09 file.txt
drwxrwxr-x
2 test
test
4096
Nov 6 13:22 Pictures
---------------- ------- -------- ------------ ------------|
|
|
|
|
|
|
|
|
|
|
File Name
|
|
|
|
+--- Modification Time
|
|
|
+------------Size (in bytes)
|
|
+----------------------Group
|
+-------------------------------Owner
+---------------------------------------------File Permissions
- is a normal file
d is a directory
b is a block device
1.12.15 ACLs
• Access control lists
• Not recommended; hard to maintain
• Typically how other OSes manage permissions
• Support depends on OS and filesystem
1.12.16 Hands-On: Files and Permissions
$ touch foo # create empty file called foo
• As root, create a file in /home/$yourusername/bootcamp
• Who can do what to the file?
• Make the devops group own the file
• Make a file called allperms and give user, group, and world +rwx
• Make more files and practice changing their permissions
34
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.12.17 Package Management
Take care of installation and removal of software
Core Functionality:
• Install, Upgrade & uninstall packages easily
• Resolve package dependencies
• Install packages from a central repository
• Search for information on installed packages and files
• Pre-built binaries (usually)
Popular Linux Package Managers
• .deb / APT (used by Debian, Ubuntu)
• .rpm / YUM (used by RedHat, CentOS, Fedora)
1.12.18 RPM & yum (RedHat, CentOS, Fedora)
RPM
Binary file format which includes metadata about the package and the application binaries as well.
Yum
RPM package manager used to query a central repository and resolve RPM package dependencies.
Yum Commands (Redhat, CentOS, Fedora)
We’ll use the tree package as an example below.
# Searching for a package
$ yum search tree
# Information about a package
$ yum info tree
1.12. Lesson 2: Single System Fundamentals
35
OSU DevOps Bootcamp Documentation, Release 0.0.1
# Installing a package
$ yum install tree
# Upgrade all packages to a newer version
$ yum upgrade
# Uninstalling a package
$ yum remove tree
# Cleaning the RPM database
$ yum clean all
RPM Commands
Low level package management. No dependency checking or central repository.
# Install an RPM file
$ rpm -i tree-1.5.3-2.el6.x86_64.rpm
# Upgrade an RPM file
$ rpm -Uvh tree-1.5.3-3.el6.x86_64.rpm
# Uninstall an RPM package
$ rpm -e tree
# Querying the RPM database
$ rpm -qa tree
# Listing all files in an RPM package
$ rpm -ql tree
1.12.19 DPKG & Apt (Debian, Ubuntu)
Deb
Binary file format which includes metadata about the package and the application binaries as well.
DPKG
Low level package installer for the .deb file format. Does no package dependency resolution.
Apt
DPKG package manager used to query a central repository and resolve Deb package dependencies. Considered mostly a front-end to dpkg.
36
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
Apt Commands (Debian, Ubuntu)
Note: You can also use aptitude as a front-end to dpkg instead of apt-get.
# Update package cache database
$ apt-get update
# Searching for a package
$ apt-cache search tree
# Information about a package
$ apt-cache show tree
# Installing a package
$ apt-get install tree
# Upgrade all packages to a newer version
$ apt-get upgrade
$ apt-get dist-upgrade
# Uninstalling a package
$ apt-get remove tree
$ apt-get purge tree
Dpkg Commands
Low level package management. No dependency checking or central repository.
# Install or upgrade a DEB file
$ dpkg -i tree_1.6.0-1_amd64.deb
# Removing a DEB package
$ dpkg -r tree
# Purging a DEB package
$ dpkg -P tree
# Querying the DPKG database
$ dpkg-query -l tree
# Listing all files in a DEB package
$ dpkg-query -L tree
1.12.20 Language-specific Package Managers
• Languages sometimes have their own package management suite
• Can be useful for using newer versions of packages
• Examples
– pip (Python)
– rubygems (Ruby)
– CPAN (Perl)
1.12. Lesson 2: Single System Fundamentals
37
OSU DevOps Bootcamp Documentation, Release 0.0.1
– cabal (Haskell)
– npm (NodeJS)
– ... and so on forever ...
1.12.21 Other Package Managers
They each fill a specific niche and have their own pros and cons.
• Portage (Gentoo) – Source based package installer
• pacman (Arch Linux)
• ZYpp / Zypper (SUSE) – Yet another RPM package manager
1.12.22 Installing from source
• Download source tarball, run build scripts and install in a local directory.
• RPM/DEB packages do this for you
• Not for the faint of heart ... Not recommended!
• Using grep as an example
$
$
$
$
$
$
wget http://mirrors.kernel.org/gnu/grep/grep-2.15.tar.xz
tar -Jxvf grep-2.15.tar.xz
cd grep-2.15
./configure
make
make install
1.12.23 Hands-on: Package Management
• Install the git package
• Query the RPM/APT database for installed packages
• List the files in an installed package
• Remove the git package
1.12.24 Questions:
• read example output of ls -al
• read output of yum or aptitude search
• install a package on their VM/partition (Vim, Git)
– explain what dependencies it also installed
38
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.13 Lesson 2: Homework
1. Why is a salt used when storing encrypted passwords in /etc/shadow?
2. What portion of the MD5 hash ’$1$xxUwcovy$JfV9i7j9H/NFA3RBCrVHN.’ is the salt?
3. What UID is used for the root user?
4. Where is a user’s primary (“default”) group defined? Specifically which file and which “field”.
5. Add a user foobar to your system. Use useradd to add the user and to create their home directory containing
files from /etc/skel. Show the user’s entry in /etc/passwd as well as the full useradd command
needed.
6. Create a group bootcamp on your system. Show the command used (not editing files by hand).
7. Assume the user foobar belongs to multiple groups. Add foobar to the bootcamp group by using a system
command (not editing files by hand) without changing any of their other groups. Show the command used.
8. What chmod command (using octal mode) would you use to allow owner read and write access and group read
access (and no other permissions!) to a file foo? Using chmod without octal mode, how would you do the
same?
9. What does it mean for a binary to be setuid? is setuid a potential security risk? Why is this important for tools
such as passwd?
10. You have the following foo directory
drwxr-xr-x 7 lance bootcamp 4096 Mar 31 09:15 foo
What chmod command can you run to ensure files created inside that directory will default to having
bootcamp group ownership? Assume the user creating the files is in the bootcamp group.
1.14 Lesson 3: Editors & Version Control
1.14.1 Check in from last week
Is anyone having problems with package management?
Install the packages named git, vim, and emacs
1.14.2 What we’ll discuss
• What’s a text editor?
• Features of an editor
• Version control: Git!
1.14.3 How do you edit files?
1.13. Lesson 2: Homework
39
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.14.4 No. Never, Ever edit code in Word/LibreOffice
• Autocorrect is your worst enemy
• Syntax highlighting is nice
• You are editing plain text, not ‘documents’, i.e. .doc, .rtf, etc
1.14.5 Then what should you use?
• Text editors
– Command-line
– available in your package manager
• For coding, an IDE can be helpful
1.14.6 Integrated Development Environments
• Entire toolchain in one place
• Usually graphical
1.14.7 Sysadmin tools
• Frequently work with remote machines over a ssh connection
– xforwarding can sort of give you a GUI
– Not all remote machines have a desktop environment
• Faster to edit file in place than transfer it down then back up
– Frequent small changes when testing or debugging
• Broken machines often have only terminal via crash cart
Editor Requirements
• Small program, runs in terminal
• Preferably installed by default on most systems
1.14.8 Features of an editor
• Needs to be installed on your system
• Create files
• Move around within the file
• Add or delete text
• Easily automate tedious tasks
40
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.14.9 Text Editors
Note: Quick intro/history: ed editor Pros: low-bandwidth, installed pretty much everywhere, very fast and powerful
for complicated and repetitive tasks Cons: Steep learning curve, different “modes” can be confusing at first Sublime
and other desktop editors: nice for serious programming, but learn the basics of simple text editors even if you want
to be a developer, you won’t always be able to edit your code on your own desktop
• ed -> Vi -> Vim
• Stallman -> lisp -> emacs
Avoid Pico/Nano, Notepad++, SublimeText
1.14.10 Emacs
1.14. Lesson 3: Editors & Version Control
41
OSU DevOps Bootcamp Documentation, Release 0.0.1
Note: Originally released 1976 name from Editor MACros for TECO editor, originally Tape Editor and COrrector at
MIT in the ‘60s
But, along the way, I wrote a text editor, Emacs. The interesting idea about Emacs was that it had a programming
language, and the user’s editing commands would be written in that interpreted programming language, so that you
could load new commands into your editor while you were editing. You could edit the programs you were using and
then go on editing with them.
– Richard Stallman, http://www.gnu.org/gnu/rms-lisp.html
1.14.11 Vim
Note: originally written for Amiga systems (Commodore PCs), 1988 vim released 1991 vimscript, Lua (as of Vim
7.3), Perl, Python, Racket, Ruby, Tcl (tool command language). vi written by Bill Joy in 1976, visual mode for line
editor called ex line editors are from age of teleprinters, no cursors
• Available almost everywhere
• Lightweight
• Design decisions explained in http://docs.freebsd.org/44doc/usd/12.vi/paper.html
• Modal editor (command, insert, visual)
1.14.12 How to choose
• What can the people around you help with?
• Try both; choose one and get good at it
• Have a good answer when people ask why you made that choice
– “Because it’s familiar” is tolerated
– “Because I was initially taught it” is common but accepted (honesty)
42
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
– “Because $usecase” provokes argument but more respected
– “Because I tried both and picked this one” is rare but good
• Your use case as a sysadmin or developer
1.14.13 Modes
How to tell?
-- INSERT --- VISUAL --
144,1
144,77
36%
36%
1.14.14 Commands
Note: Moving around in a file Search / replace Text manipulation, ie: cw, dw, c$, yy / p, x, .
1.14.15 Moving Around
h
j
k
l
0
move
move
move
move
move
one character to the left.
down one line.
up one line.
one character to the right.
to the beginning of the line.
1.14. Lesson 3: Editors & Version Control
43
OSU DevOps Bootcamp Documentation, Release 0.0.1
$ move to the end of the line.
w move forward one word.
b move backward one word.
G move to the end of the file.
gg move to the beginning of the file.
. move to the last edit.
1.14.16 Configuration / customization
Note: there are many many options and pre-existing packages to make editing nice for sysadmins and developers
• .vimrc
• :set
Some sets of Vim plugins and configurations are available
• https://github.com/astrails/dotvim
• https://github.com/carlhuda/janus
Use them for research on what’s available to improve dev productivity
1.14.17 Learning Resources
• $ vimtutor
• http://vim-adventures.com/
1.14.18 Regular expressions
You should know basic substitution:
:%s/foo/bar/g
On IRC, Hamper does rudimentary regex in the form s/foo/bar/ applying only to the most recent comment.
This is not shell globbing
Resources for learning
• RegExr - an interactive Regular Expression editor and debugger
• Regular-Expressions.info - Tutorials and general information
1.14.19 Editor questions?
• Open an editor, find a cheat sheet, try to add some text
• Modify the text: “disemvowel” it
$ vim testvim.txt
<i>
Hello world!
<esc>
:%s/[aeiou]//g
44
$ emacs testemacs.txt
Hello world!
<esc>
<
<alt + x>
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.14. Lesson 3: Editors & Version Control
45
OSU DevOps Bootcamp Documentation, Release 0.0.1
:wq
replace-regexp
[aeiou]
<enter>
<ctrl + x> <ctrl + s>
<ctrl + x> <ctrl + c>
1.15 Lesson 3: Intro to Git
1.15.1 Version Control is Hard
1.15.2 Why Bother?
1.15.3 Better Options: Version Control
Note: Collaboration with multiple developers is important to mention
• Commit = Snapshot of part of your project’s state
• Centralized (SVN, CVS) vs. Decentralized (Git, hg)
• We’ll look at Git today
– Easier to learn other VCS from Git
– Widely used in the open source world
1.15.4 Git
git noun. Brit.informal. 1. an unpleasant or contemptible person.
1.15.5 Using Git Locally
# This initializes a git repo. Use ‘man git-init‘ for more info.
$ git init
# This puts <filename> into the staging area. It isn’t committed yet.
# ‘git diff‘ to see what changes aren’t yet in staging.
$ git add <filename>
Use
# This actually makes the commit. Use ‘git status‘ to see what’s in staging
# but not yet committed. Use ‘git show‘ or ‘git log‘ to see recent commits.
$ git commit -m "I did a thing!"
• Undo things? the git book explains well
• Did I remember to commit that? $ git status
• What commits have I made lately? $ git log
46
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
Figure 1.1: Image from XKCD
Figure 1.2: Image from PhD Comics
1.15. Lesson 3: Intro to Git
47
OSU DevOps Bootcamp Documentation, Release 0.0.1
48
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.15.6 More on commits
Your work goes from unstaged to staging area with git add.
$ git config --global user.name ’Your Name’
$ git config --global user.email [email protected]
• Everything in staging gets wrapped up into an object that contains
– changes
– timestamp
– author info
– parent commit hash
• These live in .git/ in your project directory
• Commits go to other locations with git push
1.15.7 What Not To Do
Figure 1.3:
github/
image from http://arstechnica.com/security/2013/01/psa-dont-upload-your-important-passwords-to-
• Don’t delete the .git files
Note: If you kill them, git loses its memory :(
• Redundant copies of same work
• “oops, undoing that” commits.
– Use git commit --amend
Note: Amending is fine as long as you haven’t pushed yet. It’s generally a bad idea to amend or rebase work that
you’ve already shared with others, unless you really know what you’re doing.
• Don’t wait too long between commits
– Squashing them together later is easy
Note: Commit every time you think you might want to return to the current state. You can revert back to any previous
commit, but there is no way to magically add a commit in where you forgot to make one.
1.15. Lesson 3: Intro to Git
49
OSU DevOps Bootcamp Documentation, Release 0.0.1
• Don’t commit compiled/generated items
Note: Mostly relevant to writing code, .gitignore allows you to avoid dealing with compiled binaries, generated
output, log files, etc
• Don’t commit secrets...
Note: Yes, there are ways to sort of take them down off of GitHub, but somebody might have cloned your repo while
it had the secrets in. Once someone has a piece of information, you can’t just take it away.
1.15.8 Daily workflow
• Pull
• Work
• Add changes
• Commit
• Push
Larger projects have more complex workflows
Note: The picture is of the Git Flow branching model, and you’ll probably see it every single time anyone explains Git
branching and merging to you. If you are working on a larger project or writing code, you’ll likely be using branches,
this allows a project to keep many simultaneous code changes organized.
1.16 Lesson 3: GitHub
• Manage permissions on repos
• Back up your work
• Social/gamification
• Amazing documentation: http://help.github.com
Note: GitHub serves a threefold purpose: It also has amazing documentation which you should all go read right now
and consult whenever you’re the least bit confused. It’s like the Ubuntu forums in that it’s explained in a way the
newbies can understand, but unlike them in that it’s all written by people who know what they’re doing.
1.16.1 Let’s Walk Through
• Creating an account
– Gravatar
– How to read a profile
50
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.16. Lesson 3: GitHub
51
OSU DevOps Bootcamp Documentation, Release 0.0.1
52
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
Note: you just go to github.com and click the account creation links. To make a custom icon, go to gravatar.com and
set up an account using the same email address as you used for github. The picture you upload on Gravatar will then
show up for your github account.
The most important thing about reading profiles is that not all of a person’s repos will display on the front page of their
profile – to see them, got to the ‘repositories’ tab instead of ‘contributions’.
• Creating SSH keys
– ssh-keygen -t rsa
• Uploading your SSH key
• Creating a new repository
• Fork somebody else’s repo
• Edit files online
• Submit a pull request
1.16.2 Help, Everythings’s Broken!
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
Solution ssh-add ~/.ssh/id-rsa or whatever key you have added on github
To [email protected]:edunham/slides.git
! [rejected]
master -> master (non-fast-forward)
error: failed to push some refs to ’[email protected]:edunham/slides.git’
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Merge the remote changes (e.g. ’git pull’)
hint: before pushing again.
hint: See the ’Note about fast-forwards’ in ’git push --help’ for details.
Solution To avoid a messy merge commit, git pull --rebase.
1.16.3 Learn More
• http://git-scm.com/book
• http://try.github.io/levels/1/challenges/1
1.16.4 Hands-On
• Fork the devopsbootcamp dotfiles repo
• Clone a copy of the repo to your VM and make a branch
• Make a commit with a helpful commit message and push to your fork
$
$
$
$
$
ssh-keygen -t rsa # make an SSH key and add it to your account
git clone <url from sidebar of your fork> # clone the repo
cd dotfiles # git commands only work in project directlry
git checkout -b <yourname> # -b creates branch
vim <filename>
1.16. Lesson 3: GitHub
53
OSU DevOps Bootcamp Documentation, Release 0.0.1
#
#
#
$ git
$ git
$ git
’i’ to enter insert mode
<esc> to get back to command mode
:wq to save and quit
add <filename>
commit -m "please use a helpful commit message, not like this one"
push
1.17 Lesson 4: Scripting, Troubleshooting, & Workflow
Note: week 1, 1/9/2014 - log files - git bisect, git log, git blame - scientific method (small changes)
1.17.1 Agenda
• Scripting
– What’s a script?
– What’s a scripting language?
– Common scripting languages
– Bash & Python examples
• Troubleshooting
54
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
– Techniques
– Examples
• Workflow
– Life cycle of a bug
1.17.2 Scripting
1.17.3 What’s a script?
• Piece of code to automate a boring task
• Explanation of how to do what you do
1.17.4 What’s a scripting language?
Note: This is more programming language than scripting specific.
• Compiled vs interpreted
• Expresses logic and actions
– Logic: loops, conditionals, statements
– Actions: File I/O, print to console, interact with system utilities
1.17. Lesson 4: Scripting, Troubleshooting, & Workflow
55
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.17.5 Common Scripting Languages
• Bash
• Python
• Language affects the speed of development and performance of a task
• Most tasks can be done using any language
1.17.6 Bash
• Adds a bit of logic to automate your shell commands
• Saves re-typing things
• Powered by the tools you invoke from the shell
• Scripts can be hard to read at first because they call to external utilities
1.17.7 Python
1.17.8 Why Python?
• Easy to read
• Easy to maintain
• Quick to write
• Lots of libraries
1.17.9 Troubleshooting
1.17.10 Informal method
• Notice that something isn’t working right
• Identify what should be happening
– Define a success criterion (“it works if...”)
1.17.11 If it used to work
• Determine what changed
– Version control history (Git bisect)
– Emails from the system? Logs? (Check for cron jobs or config mgmt)
– Ask others who’ve been working on system
• Use your own notes/documentation
56
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.17. Lesson 4: Scripting, Troubleshooting, & Workflow
57
OSU DevOps Bootcamp Documentation, Release 0.0.1
58
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.17. Lesson 4: Scripting, Troubleshooting, & Workflow
59
OSU DevOps Bootcamp Documentation, Release 0.0.1
60
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.17.12 If it’s never worked for you
• Determine whether it’s possible at all
• Find evidence of similar things working (code, blog posts, stackoverflow)
• If there’s no evidence of anything like this working, you might be Doing It Wrong (tm)
• If there’s documentation of something similar working:
– Confirm that the docs are correct for the versions of things that you’re using
– If they docs are wrong, fix them
– If the docs appear right, figure out what differs between your code and the example
• If there’s sample code, make sure you can run it
– Your goal is minimum viable test case
1.17.13 After finding the problem
• Did the docs tell you how to fix it?
• If you can’t fix the problem, identify why not, and then fix that
• Ask for help
1.17. Lesson 4: Scripting, Troubleshooting, & Workflow
61
OSU DevOps Bootcamp Documentation, Release 0.0.1
– Expert takes 5 minutes to answer a well-asked question
– Newbie can waste hours
1.17.14 Formal method
(from this)
• Identify the problem
• Establish a theory of probable cause (question the obvious)
• Test the theory to determine the cause
• Establish a plan of action to resolve the problem and implement the solution
• Verify full system functionality and, if applicable, implement preventative measures
• Document findings, actions, and outcomes
1.17.15 How to get help
• Don’t ask to ask
• Summarize what’s wrong
• Summarize what you’ve tried and why it hasn’t worked
• Make a specific request, politely
• Pick the right place & time to ask
1.17.16 Documentation
• Man pages
• Wikis
• Google (used wisely)
– Assessing sites’ applicability and reliability
* Who wrote it?
* When?
* Is the other content reliable?
* Is feedback from others visible? If so, what does it say?
1.17.17 Sources of trouble
When using something new
• You probably misunderstood it.
• Maybe their documentation was wrong.
• If neither, then perhaps their code is wrong.
• Submit a ticket or pull request to fix the docs or code
62
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
When something previously working breaks
• Something changed
• Someone updated something
• Figure out who and why; document
1.17.18 Tickets
• Ticket (often sysadmin) or Issue (often developer)
• Ticket comes into tracking system, submitted by a user
• Triage
– Add details to tickets; consolidate duplicates
– Contact submitter if more info needed
– Add tags, milestones, priority, etc.
• Ticket is assigned to someone, who fixes it
• Someone else confirms that the fix works, then ticket is closed
1.17.19 Tickets vs. Issues
• Workflow defined by tracker system
– RT, Redmine, Chiliproject, GitHub issues, mailing lists
• Issues/Bugs are developer work items which need to be included in a release of code
• Tickets are sysadmin work items, often related to systems improvement or maintenance
• Can’t log in because your account got reset: Ticket.
• Can’t log in because the newest release of the software is incompatible with the old database format: Bug.
1.17.20 Some Examples
• Trac
• Chiliproject
• RT
• Bugzilla
1.18 Lesson 5: Web Applications
1.18.1 What is a web app?
Note:
• talk briefly about responsibilities of each component
• talk briefly about http
1.18. Lesson 5: Web Applications
63
OSU DevOps Bootcamp Documentation, Release 0.0.1
• mention Virtual Environments (red box on diagram)
1.18.2 What is this Virtual Environment thing?
Note: Discuss system packages vs local, mention rvm
• Separates application packages from system packages
• Choose versions that differ from the system versions
• Maintain a list of required packages for easy install
• Install modules as a user
• Pip and Gem
64
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.18.3 Frameworks
Ready-made building blocks take care of the tedious details:
• Speaking HTTP
• Connecting code to URLS
• Talking to the database
• Built-in development servers
Popular frameworks:
Django, CakePHP, Ruby on Rails, Flask
Note: Talk about major differences between frameworks, especially Django and Flask
1.18.4 The SystemView app
Display a filtered list of processes on the system
Note:
• explain basic idea of the app
– reference code from lesson 4
1.18.5 Setting Up
Update your vagrant file and vagrant up:
1.18. Lesson 5: Web Applications
65
OSU DevOps Bootcamp Documentation, Release 0.0.1
git pull
vagrant up
vagrant ssh
Note: Changes to the vagrant file: forward port 5050 to 5000 on the vm
1.18.6 Setting Up
Install packages
sudo yum install python-virtualenv*
sudo yum install git
Create a virtualenv
virtualenv systemview_venv
And activate it
source systemview_venv/bin/activate
Note:
• students probably already have git?
• discuss what virtualenv actually does, what is in it
• env variables, etc
• they can put the virtualenv anywhere, discuss locations
• discuss, but don’t use virtualenv tools (mkvirtualenv, use, etc)
• explain what source does
1.18.7 Get the Code
git clone [email protected]:DevOpsBootcamp/systemview.git
Note:
• break here for github account setup, key location (where are they checking code out from? Where is their key
located?), etc
• https://github.com/DevOpsBootcamp/systemview.git for anyone who can’t get their account/key working
1.18.8 Run the Code
cd systemview
python systemview.py
66
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.18.9 Fail
Oops!
Traceback (most recent call last):
File "systemview.py", line 2, in <module>
from flask import Flask, request, session, g, redirect, url_for, \
ImportError: No module named flask
Note: talk about missing modules, we need to install them, this is what the venv is for
1.18.10 Pip
A package manager for Python packages
• Connects to PyPi, a vast repository of Python modules
• Resolves dependencies, installs prerequisites
• Can install packages from a list in a file
1.18.11 Install What’s Missing
Make sure you are in your virtualenv, then:
pip install flask
pip install argparse
Note:
• How do you know if you are in the virtualenv?
• can put requirements in requirements.txt for easy installation
1.18.12 Run and Test!
python systemview.py -i 0.0.0.0 -d
Now go to http://localhost:5050
Note:
• talk about flags
• go to terminal after this slide and talk about the code:
– main module, templates, css, etc
• Point out areas where bugs could be fixed or features added
1.18. Lesson 5: Web Applications
67
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.18.13 Branch and Modify
Create a Github issue for your changes
https://github.com/DevOpsBootcamp/systemview/issues
Create a branch for your changes
git checkout -b my_name
When you have made changes and everything works, push it up
git push origin my_name
Note:
• talk about branching vs forking, get everyone working on a new feature or bug
• use IRC handles for branch names to make sure you are unique and identifiable
1.18.14 Homework
Add a feature or fix a bug, push your changes up.
Github URL:
https://github.com/DevOpsBootcamp/systemview
Github issue tracker:
https://github.com/DevOpsBootcamp/systemview/issues
1.19 Lesson 6: Boot and the Filesystem Hierarchy
Note:
• grub, filesystem stuff based roughly on Frostsnow’s talk
• basics of kernel and differences between virtualization/physical (the picture that kevin draws)
1.19.1 The Linux Filesystem Hierarchy
Note: Based on Wade’s talk https://github.com/clinew/presentation_filesystems/blob/master/presentation.tex
What’s a filesystem?
In computing, a file system is used to control how information is stored and retrieved. Without a file
system, information placed in a storage area would be one large body of information with no way to tell
where one piece of information stops and the next begins.
• (http://en.wikipedia.org/wiki/Filesystem)
68
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.19.2 Filesystem can mean:
• How the system’s files are arranged on the disk
• How the disk actually holds the files
– FAT and NTFS are old but Windows-compatible
– ext3 is standard, ext4 is newer, xfs has fancier journaling
* journaling tracks changes before write
– sysadmins will encounter NFS and its competitors like Gluster
Note: Moving from Windows?
• Binaries, not executables.
• Directories, not folders.
• Read, not load.
• Symbolic links, not shortcuts.
• Write, not save.
1.19.3 The File System
$ ls
bin
boot
dev
etc
home
initrd.img
initrd.img.old
lib
lib64
lost+found
media
mnt
1.19. Lesson 6: Boot and the Filesystem Hierarchy
opt
proc
root
run
sbin
selinux
srv
sys
tmp
usr
var
vmlinuz
69
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.19.4 Installed programs and utilities
/bin
/sbin
/usr/bin
/usr/sbin
/usr/local/bin
/usr/local/sbin
• PATH environment variable
$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• which command
$ which bash
/bin/bash
1.19.5 User-Specific Data & Configuration
• Data stored at /home/<username>
– Desktop environment creates folders Documents, Pictures, Videos, etc.
• Configurations in dotfiles within home (/.)
• Lost+Found is not your desktop trash can
– Lost blocks of the filesystem.
– Usually not an issue.
– If your desktop provides backups of deleted files, they’ll be somewhere in /home/<username>/
1.19.6 Where are drives mounted?
• Raw device appears under /dev.
$ dmesg | tail
[260930.208715] sdb: sdb1
[260930.320756] sd 6:0:0:0: >[sdb] Asking for cache data failed
[260930.320765] sd 6:0:0:0: >[sdb] Assuming drive cache: write through
[260930.320771] sd 6:0:0:0: >[sdb] Attached SCSI removable disk
• USB filesystem under /media, main disk /
• You can manually mount devices with mount
– “Everything’s a file”
– umount to unmount
• /etc/fstab tells things where to mount
• /etc/mtab shows where things are currently mounted
1.19.7 Space on drives
• Use df to see disk free space.
70
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
$ df -h /
Filesystem
/dev/sda8
Size
73G
Used Avail Use% Mounted on
29G
41G 42% /
• Use du to see disk usage.
$ du -sh /home/
21G /home/
• Default output is in bytes, -h for human-readable output.
1.19.8 Three Tiers of Filesystem Hierarchy
• /, essential for system booting and mounting /usr.
• /usr, read-only system data for normal system operation.
• /usr/local, locally-installed software.
– Package managers usually install under / and /usr.
1.19.9 Common Directories
Directory
/bin
/include
/lib
/sbin
/boot
/dev
/etc
/home
/media
/mnt
Contents
Binary files
Header files for C/C++ programs
Libraries
Binary files for root (superuser)
Files essential for booting kernel, initramfs
Virtual filesystem, exports hardware devices
System-wide configurations
Individual users’ data
Removable storage devices
Like media – place to mount disks and things
1.19. Lesson 6: Boot and the Filesystem Hierarchy
71
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.19.10 Common Directories
Directory
/opt
/proc
/root
/run
/sys
/tmp
/var
/usr/share
/usr/src
Contents
“Add-on application software packages”
Virtual filesystem exporting system data
homedir for root
Volatile information accumulated since boot
Virtual filesystem exporting kernel objects
Temporary files
Data which varies – logs, mail, etc.
Architecture-independent, read-only data
Kernel source code
1.19.11 /proc has lots of useful system information
Which Linux kernel version are you running?
$ cat /proc/version
Linux version 3.5.0-17-generic (buildd@allspice) (gcc version 4.7.2
(Ubuntu/Linaro 4.7.2-2ubuntu1) ) #28-Ubuntu SMP Tue Oct 9 19:31:23 UTC 2012
Learn about system’s hardware
$ less /proc/cpuinfo
$ less /proc/meminfo
Some parts of /proc can be written as well as read...
$ echo 3 > /proc/sys/vm/drop_caches # drop caches
1.19.12 Commands for working with filesystems
Creating filesystems
$ mkfs
Mounting filesystems
$
#
#
#
mount
-t for type
-o for options
requires device path and mount point
Loopback devices
$ losetup
$ /dev/loop*
# makes it look like a device instead of a file
1.19.13 devfs
sd*
sr*
/dev/null
72
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
/dev/random
/dev/urandom
/dev/zero
1.19.14 Blocks and dd
• Block size is the size of chunks allocated for files
• dd
– Disk duplicator (or disk dump).
* if=<path>, input file.
* of=<path>, ooutput file.
* bs=<size>, block size.
* count=<size>, number of block to transfer.
$ dd if=/dev/random of=/dev/sda
# What will this do?
1.19.15 Filesystem Consistency
• Metadata vs. data
– Metadata is extra information the filesystem tracks about the file
– Data is the file’s contents
• Filesystem is consistent if all metadata is intact
– fsck is FileSystem Consistency Check
1.19.16 More about Journaling
• Filesystem consistency tool; protections against system freezes, power outages, etc.
• Replaying the journal.
• ext3’s three modes of journaling:
–
journal Data and metadata to journal.
–
ordered Data updates to filesystem, then metadata committed to journal.
–
writeback Metadata comitted to journal, possibly before data updates.
1.19.17 The Boot Process
• Bootstrapping
• Steps in the process
• Boot loaders
• Startup scripts
• Boot levels
1.19. Lesson 6: Boot and the Filesystem Hierarchy
73
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.19.18 Bootstrapping
Note: kernel loaded into memory, initialization tasks, and available to users
Init
• kernel spawns init which is always PID 1
• controls the boot process
• can be a simple script to a binary
• Pull itself up by its own bootstraps
• Automatic and manual booting
• Driver Loading
• Period of vulnerability
– configuration errors
– missing hardware
– damaged filesystems
• init – Always Process ID (PID) #1
– First process to start
– Either a binary or can be a simple script (even a bash shell!)
1.19.19 Steps in boot process
Note:
Kernel
• 1st stage – bootloader, 2nd, boot the kernel
• boot from boot loader
• load into memory
• located in /boot/ on Linux
Hardware config
• locate & initialize hardware
74
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
• print out what it does
System processes
• init, kswapd, pdflush, etc
• init only real process
• Others look like processes for scheduling (appear as [kswapd] with ps)
1. Kernel initialization
2. Hardware configuration
3. System processes
4. Operator intervention (single-user)
5. Execution of start-up scripts
6. Multi-user operation
1.19.20 Booting
Note:
On hardware specific to UNIX (i.e. Sun)
• firmware knows how to use devices
• talk to the network
• understand filesystems
• all accessible via the commandline
1.19. Lesson 6: Boot and the Filesystem Hierarchy
75
OSU DevOps Bootcamp Documentation, Release 0.0.1
BIOS smarter than they used to be
• Not standardized
• Most servers support PXE
• PCs vs Proprietary hardware
– BIOS, UEFI, OpenBoot PROM, etc
• BIOS
– Basic Input/Output System
– Very simple compared to OpenBoot PROM / UEFI
– Select devices to boot from
– MBR (first 512 bytes)
• UEFI
– Unified Extensible Firmware Interface
– Successor to BIOS
– Flexible pre-OS environment including network booting
1.19.21 Boot Loaders (Grub)
Note:
Grub
• next generation PC boot loader
• no need to “re-run grub” config updates
• Grub config
• disks are index based from zero
• grub install commands
• netboot, pretty, serial
• device.map, grub.conf
• robust with weird disk geometry
• Grand Unified Bootloader
• Dynamic fixes during booting
• Can read the filesystem
• Index based – (hd0,0) = sda1
• Grub “version 1” vs. “version 2”
– Version 2 has more features, but more complicated
– Latest Debian, Ubuntu and Fedora use v2
76
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
grub> root (hd0,0)
grub> setup (hd0)
grub> quit
(Specify where your /boot partition resides)
(Install GRUB in the MBR)
(Exit the GRUB shell)
grub-install
1.19.22 Single User Mode
Note:
Show on VM
• enter grub, hit ESC, pick kernel, hit “e” for edit
• use arrows
Typically ask for root password
• What is it used for?
• Troubleshoot problems
• Manual Filesystem Checks
• Booting with bare services
• Fix boot problems
• Add “single” to kernel option
• Solaris/BSD
– boot -s
1.19.23 Startup Script Tasks
Note: Verbose and print out description of what its doing.
Old days were to manually adjust scripts, not anymore. Most are configurable now.
1.19. Lesson 6: Boot and the Filesystem Hierarchy
77
OSU DevOps Bootcamp Documentation, Release 0.0.1
• Setting up hostname & timezone
• Checking disks with fsck
• Mounting system’s disks
• Configuring network interfaces
• Starting up daemons & network services
1.19.24 System-V Boot Style
Note:
• System-V Most common today
• Show system changing between different run levels.
• Slightly different between Distros
• Linux derived from System-V originally
• Alternative init systems
– systemd - Fedora 15+, Redhat 7+ and Debian* (dependency driven)
– upstart - Ubuntu, Redhat 6 (event driven, faster boot times)
Run levels:
level 0
level 1 or S
level 2 through 5
level 6
sys is completely down (halt)
single-user mode
multi-user levels
reboot level
1.19.25 /etc/inittab
Note: Look at inittab
• Tells init what to do on each level
• Starts getty (terminals, serial console)
• Commands to be run or kept running
• inittab not used with systemd or upstart
78
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
# The default runlevel.
id:2:initdefault:
# What to do in single-user mode.
~~:S:wait:/sbin/sulogin
# What to do when CTRL-ALT-DEL is pressed.
ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now
# terminals
1:2345:respawn:/sbin/getty 38400 tty1
T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100
1.19.26 init.d Scripts
Note:
sshd init script
• case statement
• functions
• chkconfig
• One script for one service/daemon
• Start up services such as sshd, httpd, etc
• Commands
– start, stop, reload, restart
• sshd init script
$ service sshd status
openssh-daemon (pid 1186) is running...
$ service sshd restart
Stopping sshd:
Starting sshd:
[
[
OK
OK
]
]
1.19.27 Starting services on boot
Note: Show sshd script show list, adding, removing, enabling, disabling
• rclevel.d (rc0.d, rc1.d)
• S = start, K = stop/kill
• Numbers to set sequence (S55sshd)
• chkconfig / update-rc.d
– Easy way to enable/disable services in RH/Debian
• Other distributions work differently
1.19. Lesson 6: Boot and the Filesystem Hierarchy
79
OSU DevOps Bootcamp Documentation, Release 0.0.1
$ chkconfig --list sshd
sshd
0:off 1:off 2:on
3:on
4:on
5:on
6:off
$ chkconfig sshd off
$ chkconfig --list sshd
sshd
0:off 1:off 2:off 3:off 4:off 5:off 6:off
1.19.28 Configuring init.d Scripts
Note: show sendmail & network config examples for CentOS
/etc/defaults seems to be more common between UNIX’s
• /etc/sysconfig (RH) or /etc/defaults (Debian)
• source Bash scripts
• Daemon arguments
• Networking settings
• Other distributions are vastly different
$ cat /etc/sysconfig/ntpd
# Drop root to id ’ntp:ntp’ by default.
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"
1.19.29 Shutting Down
Note: Modern systems are less touchy with hard resets, but still need to be careful. Only for emergencies.
Shutdown -h
• Not Windows, don’t reboot to fix issue
• Can take a long time (i.e. servers)
• Reboot only to
– load new kernel
– new hardware
– system-wide configuration changes
• shutdown, reboot, halt, init
• wall - send system-wide message to all users
$ wall hello world
Broadcast message from root@devops-bootcamp (pts/0) (Fri Jan 31 00:40:29 2014):
hello world
80
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.19.30 Homework
1.20 Lesson 7: Databases
1.20.1 Filesystems Review Questions
• What might happen to a busy ext2 volume on power loss?
• ext3?
• ext4?
Note:
• ext2 may not be consistent upon restart
• ext3 and ext 4 are not
• but consistency only guarantees metadata is intact
• What might happen to a busy ext2 volume on power loss?
• ext3?
• ext4?
1.20.2 But what about our poor data?
• Possibly gone, like the wind
• Or worse: Half completed writes!
• General purpose operating systems, by design, don’t understand structured data
1.20.3 Enter the Database
A database system’s fundamental goal is to provide consistent views of structured data using the tools the
operating system makes available.
Chief among them is fsync(2)
Note: fsync instructs the operating system to flush all writes to disk before returning
1.20.4 Structure
SQL databases are structured around Relational Algebra
• Tables
– Columns are fields
– Rows define a relation between fields
• A Primary key is a set of columns that uniquely identify rows in a table
• A Foreign key is a column that matches the primary key of another table
1.20. Lesson 7: Databases
81
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.20.5 Relational Algebra Visualized
Note: joins are the principle use of relations.
1.20.6 Installing MySQL
$ yum install mysql-server
$ /sbin/service mysqld start
$ /usr/bin/mysql_secure_installation
82
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.20.7 Managing MySQL
$ /sbin/service mysqld status
$ mysqladmin -p ping
$ mysqladmin -p create nobel
1.20.8 Configuration
• /etc/my.conf
• The most important MySQL tuning rule:
– almost always prefer InnoDB
Note: we’re going to add: default_storage_engine = InnoDB
1.20.9 Users & Permissions
$ sudo mysql -p
mysql> CREATE USER ’vagrant’@’localhost’
IDENTIFIED BY ’password’;
mysql> GRANT ALL PRIVILEGES ON nobel.*
TO ’vagrant’@’localhost’
WITH GRANT OPTION;
1.20.10 Importing Data
$ wget http://osl.io/nobel -O nobel.sql
$ mysql -p nobel < nobel.sql
$ mysql -p nobel
SHOW TABLES;
DESCRIBE nobel;
1.20.11 Basic Queries
4 basic operations on data:
• SELECT
• INSERT
• UPDATE
• DELETE
1.20.12 SELECT
1.20. Lesson 7: Databases
83
OSU DevOps Bootcamp Documentation, Release 0.0.1
SELECT
yr, subject, winner
FROM
nobel
WHERE
yr = 1960;
1.20.13 Practice
• Who won the prize for Medicine in 1952?
• How many people were awarded the 1903 Nobel in Physics?
• How many prizes were awarded to Linus Pauling?
• How many people have won more than once? (Difficult)
1.20.14 INSERT
INSERT VALUES
(’2013’,’Literature’,’Herta Müller’)
INTO
nobel;
Note: this data stops at 2008, so lets insert some 2009 awards
1.20.15 Practice
In 2009:
• Barack Obama won the Peace Prize
• Elinor Ostrom and Oliver E. Williamson won the prize in Economics
• http://en.wikipedia.org/wiki/List_of_Nobel_laureates
1.20.16 UPDATE
UPDATE
nobel
SET
winner=’Andrew Ryan’
WHERE
subject=’Peace’ AND yr=’1951’;
Note: obviously Andrew Ryan deserves the peace price for his work in the Rapture planned community
1.20.17 Practice
• Brigid Tenenbaum Medicine prize in 1952
84
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.20.18 DELETE
DELETE FROM
nobel
WHERE
yr = 1989, subject = peace;
Note: peace prizes can be controversial, and perhaps there’s a political interest in censoring our database?
1.20.19 Further Reading, Resources, etc.
• Codd, E.F. (1970). “A Relational Model of Data for Large Shared Data Banks”. Communications of the ACM
13 (6): 377–387.
• sqlzoo.net
• CS 440: Database Management Systems
1.20.20 Hands-On: Make a Database
• Create a new database
mysql> create database systemview
mysql> GRANT ALL PRIVILEGES ON systemview.*
TO ’vagrant’@’localhost’
WITH GRANT OPTION;
• Grant a user privileges on your new database
Note: challenge them to do this based on the material in the last hour, maybe also demo the mysql console. Make
sure everyone remembers the username and password for the next step.
1.20.21 Databases in Applications
Applications love databases.
• Application data - the information to be displayed and manipulated
• User data - complex authentication and authorization
• Logging, statistics, state and session data, etc...
Note: All the various things an app might use a database for - note that the vast majority of web apps use them for
something
1.20.22 Native SQL
Most languages allow you to speak directly to a database
Python:
1.20. Lesson 7: Databases
85
OSU DevOps Bootcamp Documentation, Release 0.0.1
#!/usr/bin/python
import MySQLdb
db = ("localhost","testuser","test123","nobel" )
cursor = db.cursor()
cursor.execute("SELECT subject, yr, winner FROM nobel WHERE yr = 1960)
data = cursor.fetchall()
for winner in data:
print "%s winner in %s: %s " % (winner[0], winner[1], winner[2])
db.close()
Note: Note the plain SQL statement, recognizable from earlier. Point out the cumbersome nature of creating the
connection, creating a cursor, sending the sql, getting data from the cursor (iterating over it if you want multiple
results), etc. Similar interfaces exist for virtually all languages.
1.20.23 Introducing the ORM
Object Relational Mapper
• Maps an Object in an application to a database table or relationship
• Talks SQL to the database, your favorite language to you
• Lets you point to different databases with the same syntax
• Intelligently manages transactions to the database
Note: Make sure people know what you mean by “object”, mention possible difference between Postgres, sqlite,
MySql, etc. Objects may map to one table, but might also incorporate relationships. ORMs also often optimize queries
and manage transactions to make database queries as efficient as possible (like all other magic, though, sometimes this
can backfire).
1.20.24 Life With a Python ORM
Look, ma! No SQL!
for subject, yr, winner in session.query(Nobel).filter_by(yr=1960):
print "%s winner in %s: %s " % (subject, yr, winner)
Much easier to read and understand, but requires some setting up first.
Note: Of course we actually have to do a lot of setup work - setting up the model, engine, session, etc - but you do
that once and can interact with the database as much as you want, without worrying about the cursor or connection.
Note that we have no SQL in this statement, it is pythonic and has pythonic methods. The database table is now an
object.
86
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.20.25 Setting Up the Magic - SqlAlchemy
SqlAlchemy - a popular Python ORM, frequently used in Flask apps (like SystemView!).
To use it, we’ll need to:
• Import sqlalchemy
• Create a “model” - a representation of our data in code
• Create an “engine” and connect it to the database
• Create a session to store the model instances and transactions
Note:
Model A object with all the properties, attributes, etc of our data, can also include code to manipulate
that data in order to represent a specific view (i.e. automatically returning sorted results). It’s just a
python class, instances are just python objects.
Engine This handles the authentication with the database, it’s like the MySQLdb.connect above.
Session An in-memory record of your changes to objects - all the orm objects you instantiate live int he
session, and are only saved to the database when you say so.
1.20.26 Let’s Databasify Systemview
Project:
• Store search terms, then provide them as links on the search page, so you can just click the most common terms
you search for.
What else? Ideas?
Note: Solicit ideas for another column or two, maybe number of times the term is used (easy incrementing example),
or number of results from the least search.
1.20.27 Hands On
• Install the following packages:
sudo yum install python-devel
sudo yum install mysql-devel
• Check out systemview from GitHub (if you don’t have it already)
git clone [email protected]:DevOpsBootcamp/systemview
1.20.28 Hands On (Cont...)
• Switch to ‘save-search’ branch
git checkout -tb save-search origin/save-search
• Activate your virtualenv
1.20. Lesson 7: Databases
87
OSU DevOps Bootcamp Documentation, Release 0.0.1
source <path to virtualenv>/bin/activate
• Install the requirements
pip install -r requirements.txt
Note: Talk about git branches again, explain tracking, git pull for people who already have it cloned, etc. Talk about
the virtualenv, have people create a new one if they have lost the one they made last time. Talk about pip and what
requirements.txt is all about - point out how easy it is to set up an app this way. Make sure requirements.txt contains
sqlalchemy.
DANGER! - people will need mysql-dev package! name varies by distribution, for centos it is libmysqlclient-dev
1.20.29 Goals
• Connect the app to your new database
• Add a new column
• Save data to that column whenever someone searches
• Fetch the data from that column and display it on the search page
• challenge: limit the returned result to only 5 terms
http://docs.sqlalchemy.org/en/rel_0_9/orm/tutorial.html
Note: The code in the repo should have a simple model with one column, ‘term’, you can make a models.py,
or just put it all in one file. If you separate them, talk about MVC. The code should start an sqlalchemy engine and
session, save the search term normalized (lowercased, stripped), the column should be set to unique. Make sure the
code handles the case of the term already existing in the database (when you add a count, increment the count when
the term exists). You should probably initialize the db directly in the code, otherwise you’ll have to open up a python
console, import the app and run the db update.
1.21 Lesson 8: Security & Authentication
1.21.1 What is security?
se·cu·ri·ty
sikyoorit/
noun
• the state of being free from danger or threat.
• the safety of a state or organization against criminal activity such as terrorism, theft, or espionage.
• procedures followed or measures taken to ensure the safety of a state or organization.
• the state of feeling safe, stable, and free from fear or anxiety.
88
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.21.2 Types of security
• Physical security
• Software security
• Network security
– Active vs. passive
1.21.3 Authentication vs. Authorization
Authentication Are they who they say they are?
Authorization Are they allowed access here?
• Authentication requires proof of identity
• Authorization requires authentication, plus permission from an authority
1.21. Lesson 8: Security & Authentication
89
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.21.4 Identity
Persistent vs. authoritative
Imagine an identity thief who takes out lines of credit in their victim’s name then pays all the bills on time...
• Is their identity persistent?
• Is their identity authoritative?
How about a project maintainer who never uses their real name online, but uses the same handle and email address
across all sites?
1.21.5 Identity
How about a project maintainer who loses the domain which was hosting their email, and thus changes addresses
abruptly?
If you’re a sysadmin who works with multiple projects, you will run into these concerns often.
1.21.6 Principle of Least Authority
• User & Group management
• ACLs
• File permissions
1.21.7 System Security
• Close ports
• Firewalls
• Process isolation
• nmap vs. fail2ban
90
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.21.8 Other Attacks
• Social engineering
– Pretexting
– Phishing
– Baiting
– Quid Pro Quo
– Tailgaiting
• Zero-Day vulnerabilities
Note: Social engineering leverages those person-to-person skills us computer folks are so well known for not having.
Can someone let me borrow their laptop for a minute? I just want to tell my mom I’ll be home late tonight... honest!
Pretexting is when someone contacts you with a pretext. They sound like they’re authentic because they know something about you which they probably got off Facebook or something else. “Hi, I’m calling from Chase Bank and I
noticed that your card might have been used at a location where fraud was committed. I have your name as Bob Smith
and your date of birth as May third, 1992. Can you read me your card number and the three digits on the back?”
Phishing is something we’ve all been warned about. We all know that eBay and Paypal aren’t going to email us asking
for our bank account information. Just don’t fall for it and you’ll be okay.
Baiting is a little more interesting. Ever walk down the street and notice a thumb drive or SD card on the ground? Hey
– free thumb drive, right? Let’s just put it in the computer, what could go wrong? Plenty. If it’s too good to be true, it
probably is.
Quid pro quo is a trade – I’ll give you something if you give me something. Would you trade your password for a
chocolate bar? According to the BBC, someone tried this outside a subway station in London in 2004, and more than
seventy percent of people took that trade! Thirty-four percent gave it up for free. Don’t do that.
I suspect many of you who live in the dorms have been told about tailgating. You’re unlocking the dorm door and
someone comes up and says “hey, I forgot my key in my room, can you let me in” or maybe they’re wearing a Domino’s
1.21. Lesson 8: Security & Authentication
91
OSU DevOps Bootcamp Documentation, Release 0.0.1
uniform and are carrying a pizza. You’re a nice person, you want to help them. Don’t do it.
And that leaves us with zero-day vulnerabilities. The term ‘zero day’ refers to the amount of time that the folks who
write and maintain the code have had to fix it. If they don’t know about it, they can’t fix it. This is why it’s so important
for us to report vulnerabilities when we discover them as was discussed earlier.
1.21.9 What to do if you discover a vulnerability
First, test and document to verify that it exists.
Then, disclose it privately to those responsible for fixing it
Provide examples – it’s basically a bug report, but through private channels (not public tracker yet!)
Give them time to release a patch before announcing it
Some places have bug bounties
1.21.10 Passwords
1.21.11 Good Passwording
• http://bash.org/?244321
1.21.12 Server Side
• Rainbow Table
• Hashing / salt
• bcrypt/ scrypt
92
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.21. Lesson 8: Security & Authentication
93
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.21.13 Password Managers
• Password managers (LastPass, 1Password, KeepPass*)
– Works with phones and other things
• pass http://www.zx2c4.com/projects/password-store/
• vim -x passwords.txt
• http://world.std.com/~reinhold/diceware.html
Note: http://makezineblog.files.wordpress.com/2013/01/fractal-rainbow-table-runner-1.jpg We use passwords for everything we do online. Some (hopefully) semi random grouping of letters, numbers, and symbols which when combined with a username allow you to authenticate with a server or process. There are a couple common attacks on
passwords, the most common of which is called a dictionary attack. This uses the fact that words are easier to remember than random characters, so it abuses human memory in order to greatly reduce the search space for passwords.
pwgen + a password manager will help you have better passwords which you don’t even have to remember! DEMO
Storing passwords on the server side is a whole other matter. One of the primary issues of concern is what happens if
your server gets compromised. Lets say for instance that you just have a giant text file that has the form “username
password” on each row. This would be super fast to to lookup users in, but if that file ends up in the wrong hands, you
lose. A better option is to not store the passwords directly, but to store some representation of the password. This is
where a hash comes in. Essentially a hash is a one way function that is fast to calculate, deterministic in output, but
_very_ hard to reverse. If you store the hash of a password you can hash what they send you to verify who they are.
Again we must consider what happens if our database was compromised. Since these hashes are deterministic and
computing power is so cheap, we can precalculate what passwords correspond to what hashes, these precomputed files
are called rainbow tables. To avoid the issue with rainbow tables we ‘salt’ our passwords. This adds a small random
string to each password so that the search space for precomputing possible passwords becomes tera/petabytes large.
Enough about passwords, we now move into more interesting things called keys!
94
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.21.14 Keys
• Better than passwords
• Symmetric vs Asymmetric
• Diffie-Hellman / RSA
1.21.15 Key Exchange
1.21.16 RSA
• Math
• Math
• More Math
• Don’t be shy
Note: Keys are password files. These can be used in place of a password for authentication and encryption.
Symmetric keys essentially work like passwords. They are basically a one-time pad where both parties need to know
the key to enable data to be stored and retrieved. Asymmetric keys work by encrypting with a public key (one everyone
can see), but only being able to be decrypted by the private key (which you shouldn’t show anyone).
1.21. Lesson 8: Security & Authentication
95
OSU DevOps Bootcamp Documentation, Release 0.0.1
The fundamental problem with communication is that if you don’t have a preshared key between two users, everything
you say is being listened to and presumably logged.
Diffie Hellman key exchange is probably the most important result in cryptography. It allows two users to communicate in plaintext (non-encrypted) and trade their public keys in order to generate a shared secret so then they can
communicate with encryption. RSA is an algorithm that follows Diffie-Hellman and is the most common way to do
key exchange.
1.21.17 SSH
• Password vs Keys
• Passphrases
• authorized_keys
• Automation
$ ssh -D 9999 [email protected]
$ ssh -R 2222:localhost:22 freshblue.lake
Note: ssh is secure shell and provides a shell to a unix machine over the ‘net by using RSA to encrypt communications
between a client and server. Passwordless login, refuse connections without keys, tunneling. Commands at the end
are run unecrypted.
Passphrases work by adding a password to a key file. Add your friends public keys to authorized_keys so they don’t
need a password to login.
ssh-agent, .ssh/config, /etc/ssh/sshd_config
DEMO Make ssh-keys, post to pastebin.osuosl.org
1.21.18 Brief History of Time (line of GPG)
• P(retty)G(ood)P(rivacy)
• Phil Zimmermann
1.21.19 GPG
• E-mail privacy
• Why you should use GPG
• Why people don’t use GPG
• Keys, signing, keyservers
• Encryption
1.21.20 Ways to use GPG
• Enigamail
• mutt
96
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.21. Lesson 8: Security & Authentication
97
OSU DevOps Bootcamp Documentation, Release 0.0.1
• Command line
$ gpg --encrypt manateessecrets.jpg.exe
1.21.21 Certificates and HTTP
• Certificate Authorities
• https
• ssl/tls
$ openssl req -new -x509 -key /etc/ssl/private/privkey.pem \
-out /etc/ssl/certs/cacert.pem -days 1095
1.21.22 Man in the Middle
Note:
• 650 CAs
• Attacks on https/ssl
• Future
DEMO sslsniff
98
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.21.23 WiFi
• wep
• wpa
• wpa2
• Wireshark
– Demo
Note:
• Attacks
• mschapv2
DEMO Wireshark
1.21.24 Crypto-wares
• Files
– Tarsnap, SpiderOak, rsync over ssh
• Communications
– VPN
– TextSecure/ RedPhone
– Tor
– https everywhere
• Security
– Metasploit, BEEF
– AirCrack, sslstrip
1.21.25 Math!
• Primes
• Number Theory
• Fields
• Elliptic Curves
Note: DEMO rot13
1.21.26 One Last Thing
• https://priv.ly ( proudly hosted at the OSL)
• “I have nothing to hide“
• jeremykun.com
1.21. Lesson 8: Security & Authentication
99
OSU DevOps Bootcamp Documentation, Release 0.0.1
• thoughtcrime.org
• https://www.schneier.com/
1.22 Lesson 8: Web application security
Figure 1.4: image source: https://info.cenzic.com/rs/cenzic/images/2013-vulnerability-summary_290x250.png
1.22.1 Web application security
• Who needs to worry about web application security?
– Everyone!
100
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
• What kinds of attacks are seen in the wild?
– Many!
• What can devops do about these attacks?
– A lot!
Note: Everyone needs to worry about web application security. You need to worry, because you’re learning how to
write web applications. You want to avoid making decisions which could lead to exposing vulnerabilities and letting
bad people use your service to do bad things. You also need to worry even if you’re not writing web applications,
because you’re using web applications. The web is still a wild and wooly place, and the last line of defense for the
user is their own common sense.
What kinds of attacks are seen in the wild? The image shows a dizzying array of acronyms and shorthand but we’ll be
going over those in a little more detail.
And what can devops like us do about these attacks? Plenty – wait and see.
1.22.2 Code Injection
• Attacks
– SQL Injection
– Cross-Site Scripting (XSS)
– Cross-Site Request Forgery (CSRF)
– Remote Code Execution
• Defenses
– Sanitize your inputs!
http://bobby-tables.com/ https://docs.djangoproject.com/en/dev/ref/contrib/csrf/ http://guides.rubyonrails.org/security.html
Note: These types of attacks consist of code that is introduced into the application causing unexpected behavior.
This code can be introduced unintentionally by typical users who use quotes or ampersands in their inputs as well as
intentionally by nefarious folks.
The comic demonstrates a classic SQL injection attack. Bobby took advantage of the school’s software not properly
sanitizing their inputs by including a command to drop the students table, causing the kind of chaos often seen in xkcd
comics.
Cross-site scripting works much the same way: someone posts a comment on a blog which includes Javascript which
gets executed when you view the comment. When it is executed, it does something horrible like send them your cookie
for that blog site.
1.22. Lesson 8: Web application security
101
OSU DevOps Bootcamp Documentation, Release 0.0.1
Cross-site request forgeries are similar but instead of Javascript you’ll see image links that really point to another site
like your bank, hoping that your cookies will let them transfer money from your accounts to theirs.
Remote site execution is what it sounds like: just like the SQL injection attack, but instead running a shell command
on the web server. I think by now you all have enough experience with running commands on your virtual machines
to know how bad that could be.
Luckily, each of these threats can be addressed the same way: listen to Bobby’s mom and sanitize your inputs! There’s
a web site dedicated to helping developers with SQL injection threats which I’ve listed above, but the same concepts
apply to the other threats. Want to stop cross-site scripting? Don’t allow users to contribute arbitrary Javascript in
comments. Want to stop cross site request forgeries? Make sure your GET requests are free of side-effects, and
include tokens in your forms. As a bonus, Django will do that last bit for you if you let it – check out that second
link up there for more details. That third link is the Rails security guide and provides advice on these issues as well as
many others.
1.22.3 Web Server-Specific Attacks
Figure 1.5: image source http://news.netcraft.com/wp-content/uploads/2014/02/apache-vulns1.png
• Version-Based
• Configuration-Based
Note: There is a constant battle between developers and the bad guys – one side discovers vulnerabilities, the other
side fixes them. One of the easiest things to do to keep the bad guys out is to use the most up-to-date version of your
web server, regardless of whether it’s Apache or IIS or nginx.
The graph above shows the most popular versions of Apache as of February 2014 according to Netcraft. Apache
encourages admins to run the latest major release of the 2.4 stable branch, which is Apache 2.4.7. How many of those
102
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
releases do you see in that image? That’s right – none. Heck, two of the top fifteen are EOLed – they aren’t even
receiving security updates any longer! This is bad. Don’t be like them.
But it’s not enough to run the latest version. You should also make sure your configuration files are updated as well.
Some default configurations will include accounts or passwords which can be guessed by hackers. Other times certain
features will be enabled by default, which can introduce vulnerabilities you don’t expect even though you’re not using
those features. Read the release notes when you update your software. Pay attention to details. They will. You should
too.
1.22.4 Problems with Design and Implementation
• Authentication and Authorization
• Session Management
• Information Leakage
• Unauthorized Directory Access
Note: The remaining threats facing the typical web developer come down to design and implementation problems.
The fine points of authentication and authorization have been discussed already: make sure that all your actions are
authorized by authenticated users and you should be okay.
Also, don’t let your cookies have infinite lifetimes. Better to have your users occasionally log in again than let
them be vulnerable to those cross-site attacks we covered before. Pro tip: PHP has a default setting for “session.cookie_lifetime” of zero, which means they never expire. If you’re using PHP, fix that.
Information leakage is pretty sneaky. Let’s say your app allows users to request a password reset by entering their
email addresses. If your app behaves differently when valid and invalid addresses are input, congratulations, you’re
leaking information. Unauthorized directory access is a specialized form of information leakage – while it’s nice to let
people know how to contact your staff, you might not want to let them download everyone’s email address and such.
1.22.5 What Not to Do: The Exercise
1.22.6 Getting Up to Date
• ssh into your vagrant environment
• change directory to your local systemview repo
$ cd ~/projects/systemview
• Make sure your local copy is up to date
$ git pull
• If you’ve modified code you’ll need to follow these instructions
$ git stash save "some witty name about your work"
$ git pull --rebase
1.22. Lesson 8: Web application security
103
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.22.7 Let’s Check out Dean’s (not so) Awesome Code
$ git checkout <not so awesome code branch goes here>
1.23 Lesson 9: Networking
1.23.1 Who is this talk for?
Someone with little or no networking knowledge.
ECE/CS 372 at OSU covers this content, more or less
1.23.2 What is a network?
“a group or system of interconnected people or things“
To us, a network is:
• Electronic devices
• Sending signals over wire, fiber, or radio
• Communicating data using a standardized protocol
1.23.3 What is a protocol?
“A set of agreed upon rules for communication“
• Defines sequence & format of packets being sent
1.23.4 The OSI Model
Open Systems Interconnection
• Layers of abstraction
Note: “Create a layer of easily localized functions so that the layer could be totally redesigned and its protocols
changed in a major way... without changing the services expected from and provided to adjacent layers”
1.23.5 Layer 1: Physical
Networking Hardware
• Connector shapes
• Wire, optical fiber, or radio signal specifications
RS-232
104
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.23. Lesson 9: Networking
105
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.23.6 Layer 2: Data Link
MAC: Media Access Control
• MAC address should be globally unique
• ARP: Address Resolution Protocol (between layer 2 & 3)
• NDP (neighbor discovery protocol) used in IPv6
• Flow control & error checking
1.23.7 Layer 3: Network
Packet forwarding and routing
Network and host addressing
• IPv4
• IPv6
1.23.8 Layer 4: Transport
Interact directly with program same-order delivery, reliability, flow control, and congestion avoidance
TCP Transmission Control Protocol
• used by HTTP, HTTPS, SMTP, POP3, IMAP, SSH, FTP, Telnet
UDP User Datagram Protocol
• No error checking built in
• No retransmission delays
• VoIP, media, games
1.23.9 Get your hands dirty
In a linux terminal run::
ip a
These will display information about your network interfaces.
See also:
ifconfig
iwconfig
1.23.10 Example output:
user@host:~$ ip a
...
2: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 33:77:00:44:66:33 brd ff:ff:ff:ff:ff:ff
3: wlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
106
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
link/ether 24:77:33:44:55:66 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.55/24 brd 192.168.1.255 scope global wlan1
inet6 fe80::2677:3ff:fed4:538c/64 scope link
valid_lft forever preferred_lft forever
1.23.11 Netmask:
Decimal IP Address
192.168.1.55
255.255.255.0
Binary IP Address
11000000.10101000.00000001.00110111
11111111.11111111.11111111.00000000
Part of address
Network (Decimal)
Network (Binary)
Host (Decimal)
Host (Binary)
Corresponding address
192.168.1.0
11000000.10101000.00000001.00000000
0.0.0.55
00000000.00000000.00000000.00110111
Available Hosts: 192.168.1.[1-254]
Broadcast address: 192.168.1.255
1.23.12 Netmask Example:
Decimal IP Address
192.168.90.55
255.255.192.0
Binary IP Address
1.23.13 Netmask Example:
Decimal IP Address
192.168.90.55
255.255.192.0
Binary IP Address
11000000.10101000.01011010.00110111
11111111.11111111.11000000.00000000
Part of address
Network (Decimal)
Network (Binary)
Host (Decimal)
Host (Binary)
Corresponding address
192.168.64.0
0.0.26.55
1.23.14 Netmask Example:
Decimal IP Address
192.168.90.55
255.255.192.0
Binary IP Address
11000000.10101000.01011010.00110111
11111111.11111111.11000000.00000000
Part of address
Network (Decimal)
Network (Binary)
Host (Decimal)
Host (Binary)
Corresponding address
192.168.64.0
11000000.10101000.01000000.00000000
0.0.26.55
00000000.00000000.00011010.00110111
Available Hosts: 192.168.[64-127].[1-254]
1.23. Lesson 9: Networking
107
OSU DevOps Bootcamp Documentation, Release 0.0.1
Broadcast Address: 192.168.127.255
1.23.15 Routes
user@host:~$ route
Kernal IP routing table
Destination
Gateway
default
foo.osuosl
link-local
*
192.168.1.0
*
Genmask
0.0.0.0
255.255.0.0
255.255.255.0
Flags
UG
U
U
Metric
0
1000
2
Ref
0
0
0
Use
0
0
0
Iface
wlan1
wlan1
wlan1
user@host:~$ route -n
Kernel IP routing table
Destination
Gateway
0.0.0.0
192.168.1.1
169.254.0.0
0.0.0.0
192.168.1.0
0.0.0.0
Genmask
0.0.0.0
255.255.0.0
255.255.255.0
Flags
UG
U
U
Metric
0
1000
2
Ref
0
0
0
Use
0
0
0
Iface
wlan1
wlan1
wlan1
1.23.16 Bootstrapping
What happens when your computer connects to a network?
1. Duplex and speed negotiation
2. Static or dynamic configuration is applied
1.23.17 Static Configuration
Must in advance know:
• IP Address
• Netmask
• Default Gateway
• DNS Servers (optional in some cases)
1.23.18 Dynamic Configuration
All of the statically defined parameters are retrieved over the network via DHCP
But how do you communicate over the network without a network configuration?
1.23.19 Reserved IPv4 Addresses
• 127.0.0.1
• 192.168.0.0
• 172.16.0.0
• 10.0.0.0
• 169.254.0.0
108
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.23.20 Public vs Private Address
NAT Network Address Translation
• lose end-to-end traceability
• hides internal network topology
• allows use of private IP’s over public internet
• conserves limited public IP’s
1.23.21 Network Devices
1.23.22 Network Devices
1.23.23 Control Layer
Connection oriented vs Connectionless
1.23.24 Collisions
CSMA CA All Wireless networks use this Carrier Sense Multiple Access with Collisions Avoidance
CSMA CD Carrier Sense Multiple Access with Collisions Detection
Why is this important?
http://articles.latimes.com/2007/aug/15/local/me-lax15
1.23. Lesson 9: Networking
109
OSU DevOps Bootcamp Documentation, Release 0.0.1
110
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.24 Lesson 10: Networking part 2
1.24.1 Agenda
• Networking review
• DNS
• Web Server Setup
1.24.2 Networking Review
1.24.3 Networking Review
1.24.4 Networking Review
1.24.5 Networking Review
1.24.6 Networking Review
1.24.7 Intro to DNS
I get the OSL’s home page when visiting osuosl.org
1.24. Lesson 10: Networking part 2
111
OSU DevOps Bootcamp Documentation, Release 0.0.1
112
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.24. Lesson 10: Networking part 2
113
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.24.8 How do I know where the page came from?
HOST(1)
BIND9
HOST(1)
NAME
host - DNS lookup utility
SYNOPSIS
host [-aCdlnrsTwv] [-c class] [-N ndots] [-R number]
[-t type] [-W wait] [-m flag] [-4] [-6] {name}
[server]
DESCRIPTION
host is a simple utility for performing DNS lookups. It is
normally used to convert names to IP addresses and vice
versa. When no arguments or options are given, host prints
a short summary of its command line arguments and options.
1.24.9 OSL
$ host osuosl.org
osuosl.org has address 140.211.15.183
How did it know?
1.24.10 DNS
Domain Name System
• Port 53
My laptop asked a DNS server how to get to osuosl.org.
114
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.24.11 Hierarchy of domains
1.24.12 Pop quiz
• What’s 15 * 823?
• 12345
1.24.13 Recursive DNS
• Always gives real answer or error
• Vulnerable to cache poisoning
1.24. Lesson 10: Networking part 2
115
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.24.14 Non-recursive
• Uses cache or referral
• Includes all authoritative-only
– root and top-level domain servers
1.24.15 Another Quiz
• What’s 15 * 823?
• How did you know so quickly?
1.24.16 Caching
• You cached the answer.
• DNS can be cached at routers, ISPs, and DNS servers to improve performance.
– Negative caching: Remember fails
• TTL
1.24.17 /etc/resolv.conf
Configuration for BIND (Berkeley Internet Name Domain tool)
• most common DNS resolver
• current version is BIND 9
$ cat /etc/resolv.conf
# Generated by resolvconf
domain wireless.oregonstate.edu
nameserver 128.193.15.13
nameserver 128.193.15.12
Can only handle recursive name servers, no referrals
1.24.18 /etc/hosts
Used to skip looking up the DNS
Useful for testing web sites
Avoid malicious sites
$ cat /etc/hosts
127.0.0.1 devops-bootcamp32.osuosl.org devops-bootcamp32 localhost
localhost.localdomain localhost4 localhost4.localdomain4
::1
localhost localhost.localdomain localhost6 localhost6.localdomain6
116
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.24.19 Load Balancing
Multiple IPs bound to a single hostname are returned in random order
$ host google.com
google.com has address
google.com has address
google.com has address
google.com has address
google.com has address
google.com has address
google.com has address
google.com has address
google.com has address
google.com has address
google.com has address
173.194.33.131
173.194.33.132
173.194.33.133
173.194.33.134
173.194.33.135
173.194.33.136
173.194.33.137
173.194.33.142
173.194.33.128
173.194.33.129
173.194.33.130
1.24.20 Records
Note: rfc 1035
A hostname -> IPV4 address
AAAA hostname -> IPV6 address
CNAME like an alias, “Go look up this name’s record”
PTR
• Pointer to a canonical name, returns name and stops
• Used in reverse DNS
SOA
• Start of Authority for a zone (such as osuosl.org)
• Administrator contact info, timers, serial number
MX Email (more on this next week)
1.24.21 Reverse DNS
Note: http://support.simpledns.com/kb/a45/what-is-reverse-dns-and-do-i-need-it.aspx
Reverse segments, then end with in-addr.arpa
$ host osuosl.org
# could also use dig
osuosl.org has address 140.211.15.183
$ dig 183.15.211.140.in-addr.arpa
...
;; AUTHORITY SECTION:
15.211.140.in-addr.arpa. 10795 IN
1.24. Lesson 10: Networking part 2
SOA ns1.auth.osuosl.org. hostmaster.osuosl.org. 1398356710 300 90
117
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.24.22 Web Apps: A Bit of Review
• We created a python app called Systemview using the Flask framework
• We tested Systemview by running Flask’s built-in webserver on the command line
• Systemview ran on a special port we had to open up on the virtual machine
1.24.23 What We Want To Do
• Install a production-quality web server on a standard port
• Serve Systemview using that web server
• Party
1.24.24 Why?
• Flask’s web server is not robust or secure
• We want to use standard ports for our web apps
• We may want to run multiple apps on one server
• Web server administration is cool
1.24.25 What is a Web Server
Note: Webserver software, not hardware
1.24.26 Webservers Talk HTTP
They don’t run code (well, they kinda do)
• PHP, Python, Ruby, C don’t run in your browser
• Separate servers (usually) run that code, and send the output of the code to the web server to send to your
browser
• Sometimes those separate servers are web server modules
Note: Apache modules generally run in the apache process itself
1.24.27 A Digression: AJAX, JSON and APIs
• Browsers render HTML/CSS (layout)
• Browsers execute Javascipt (logic)
• Javascript can dynamically update the layout
• Javascript can handle user interaction
• Javascript can call back to the server for more data
118
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.24. Lesson 10: Networking part 2
119
OSU DevOps Bootcamp Documentation, Release 0.0.1
• Javascript can process data
• Javascript is Client Side Logic
1.24.28 AJAX
• Asynchronous Javascript And XML
• An http request initiated by Javascript
• Javascript listens in the background
• The app sends a response containing data
• Javascript processes the data
• Traditionally XML, but now mostly JSON
Note: Lots of issues around security, javascript calling many servers, gathering data, calling servers outside the
domain of the originating page, etc. Install RequestPolicy and NoScript, just to see who that web page is talking to
while you read it.
1.24.29 JSON
JavaScript Object Notation
{"menu": {
"id": "file",
"value": "File",
"popup": {
"menuitem": [
{"value": "New", "onclick": "CreateNewDoc()"},
{"value": "Open", "onclick": "OpenDoc()"},
{"value": "Close", "onclick": "CloseDoc()"}
]
}
}}
1.24.30 XML
The same text expressed as XML:
<menu id="file" value="File">
<popup>
<menuitem value="New" onclick="CreateNewDoc()" />
<menuitem value="Open" onclick="OpenDoc()" />
<menuitem value="Close" onclick="CloseDoc()" />
</popup>
</menu>
1.24.31 APIs
• When web apps talk to web apps
• When javascript talks to a web app
120
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
• When curl talks to a web app
They talk HTTP, using clearly defined GET or POST params to initiate actions on the remote application.
curl http://graph.facebook.com/12345/friendlists
curl https://api.github.com/users/osuosl/repos
curl http://pub.sandbox.orcid.org/v1.1/0000-0001-7857-2795/orcid-bio
Note: Take a look at the source of a web page, look at all the javascript! How much of it is talking to Google, to
Facebook, etc?
1.24.32 Let’s Install a Web Server!
yum install httpd
1.24.33 Apache
What’s this httpd thing?
“A patchy web server” - born of many patches to NCSA’s HTTPD (1995)
• Venerable, tested, solid
• Old, complex, slow (not really that slow)
• Many modules for executing code
• Many modules for all kinds of other things too
1.24.34 Let’s Serve Some Web
Apache’s DocumentRoot is the default place where it will look for files to serve. It maps “/” in the URL to a location
on disk
http://localhost:8080/index.html
^
"/" is the DocumentRoot
We’ll write some HTML in the DocumentRoot for Apache to serve.
1.24.35 But First, Config Files
/etc/httpd/conf/httpd.conf
DocumentRoot "/var/www/html"
<Directory "/var/www/html">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>
Note: Just looking, we are not editing the configs here. Note the DocumentRoot and Directory
1.24. Lesson 10: Networking part 2
121
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.24.36 Wait, What am I Writing Again?
HTML: Hyper Text Markup Language
Go to the DocumentRoot and create an html file:
cd /var/www/html
vim index.html
<html>
<head>
<title>This is only a test!</title>
</head>
<body>
<p>Nothing to see here, move along</p>
</body>
</html>
Point your browser to: http://localhost:8080/index.html
Note: HTML, is it code? Is it a language? Can you do logic with it? What happens if you forget the <html>? The
browser does the rendering, the web server doesn’t care, it just sends the data along. HTTP Content-Type header
says what kind of data.
1.24.37 Voila!
• Apache receives a request for /index.html
• It translates “/” into /var/www/html using the DocumentRoot directive
• It looks in /var/www/html for the file “index.html“
• It finds your file and sends its contents, along with HTTP headers, back to your browser
Note: Have a look at the page source. Edit the file, remove <html>, etc, look at source again. If time allows, use
developer tools, firebug, etc to look at http headers
1.24.38 But I Want to Run Code!
Let’s put some PHP code in the DocumentRoot:
vim index.php
<html>
<head>
<title>This is only a test!</title>
</head>
<body>
<?php print "Hey, this is PHP!" ?>
</body>
</html>
Then go to http://localhost:8080/index.php
122
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.24.39 What Went Wrong?
Apache doesn’t know what PHP is, it needs a module to execute the PHP code and return data it can serve
yum install php
service httpd restart
Note: Pop quiz - where do you look to find out what went wrong? Look at log files, talk about them, then look at
page source.
1.24.40 Voila, Again.
How does Apache know what to do with index.php?
/etc/httpd/conf.d/php.conf
<IfModule prefork.c>
LoadModule php5_module modules/libphp5.so
</IfModule>
<IfModule worker.c>
LoadModule php5_module modules/libphp5-zts.so
</IfModule>
AddHandler php5-script .php
AddType text/html .php
DirectoryIndex index.php
Note: CentOS, and most distribution system packages put these conf files for modules in place for you. httpd.conf
includes everything in conf.d - similar for Nginx
1.24.41 Ok, But I Want To Serve a Python App...
There’s a module for that! (Actually several, but we are going to use this one)
WSGI: Web Server Gateway Interface
• Standardized interface for python apps to talk to web servers
• Works with many different servers
• Allows separation of python app and web server processes
Note: talk about mod_python - runs python scripts directly, not bad for single scripts, but unwieldy for applications
and frameworks.
1.24.42 Sounds Great, Let’s Go!
yum install mod_wsgi
Let’s clone the systemview app into a reasonable location while we are at it
1.24. Lesson 10: Networking part 2
123
OSU DevOps Bootcamp Documentation, Release 0.0.1
cd /var/www
git clone https://github.com/DevOpsBootcamp/systemview.git
cd systemview
git checkout wsgi
Note: Talk about the location - can be anywhere, but be consistent - /var/www is actually not in the web root, not
accessible by default, don’t put things under the docroot!
1.24.43 Don’t Forget Virtualenv!
(in the systemview/ directory)
virtualenv --no-site-packages venv
source venv/bin/activate
pip install -r requirements.txt
And lets make sure everything is owned by the web server:
chown -R apache ../systemview
Note: Web server user/group ownership is a major source of breakage - get cloning/pulling as the wrong user will
change perms on files, possibly breaking things
1.24.44 What Makes an App WSGI?
systemview.wsgi
activate_this = ’/var/www/html/systemview/venv/bin/activate_this.py’
execfile(activate_this, dict(__file__=activate_this))
import sys
sys.path.insert(0, ’/var/www/html/systemview’)
from systemview import app as application
1.24.45 Configuring Apache for Systemview
/etc/httpd/conf.d/systemview.conf
WSGISocketPrefix /var/run
WSGIDaemonProcess systemview user=apache group=apache threads=5
WSGIScriptAlias /systemview /var/www/systemview/systemview.wsgi
<Directory /var/www/systemview>
WSGIProcessGroup systemview
WSGIApplicationGroup %{GLOBAL}
Order deny,allow
Allow from all
</Directory>
(Look for this in systemview/docs/systemview.conf)
124
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
Note: This will go into a vhost some day
1.24.46 Even More Voila
http://localhost:8080/systemview
There are a lot of steps to getting this app up, wouldn’t it be nice to automate this?
Note: Future topics - configuration management and automated deploys, virtual hosts, best practices for app location,
Nginx, UWSGI, PHP-FPM, etc
1.24.47 Homework
• Deploy Systemview’s master branch with Apache (we merged the database code)
• Read about Apache Virtualhosts
• Install Nginx and UWSGI, deploy Systemview
1.25 Lesson 11: Devops & Configuration Management Intro
1.25.1 Session Summary
• Devops introduction
• Configuration management introduction
1.25.2 Devops Introduction
What is it?
“A software development method that stresses communication, collaboration and integration between
software developers and information technology (IT) operations professionals.” [Wikipedia]
1.25.3 Definition of Devops
• Software Engineering (Dev)
• Technology Operations (Ops)
• Quality Assurance (QA)
Figure 1.6: Wikipedia (cc)
1.25. Lesson 11: Devops & Configuration Management Intro
125
OSU DevOps Bootcamp Documentation, Release 0.0.1
Figure 1.7: photo by http://thoriseador.deviantart.com/ (CC)
1.25.4 The old view
“Dev” side being the “makers”
“Ops” side being the “people that deal with the creation after its birth”
This siloed environment has created much harm in the industry and the core reason behind creating Devops.
Burn down those silos!
1.25.5 History of Devops
• mid-2000s
“Hey, our methodology of running systems seems to still be in a pretty primitive state despite years
of effort. Let’s start talking about doing it better”
• Velocity Conf 2008/2009 - increased presentations on “Agile System Administration“
• Agile 2008 Conf - “Agile Infrastructure” BOF – nobody showed up!
• 2009 DevOpsDays in Ghent, Belgium - Patrick Debois
1.25.6 The Agile Approach
• Iterative, incremental
• Requirements change often thus need to be adaptive
• Very short feedback loop and adaptation cycle
• Quality focus
Manifesto:
• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
126
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.25.7 Adapting Agile to Ops
• Widening the principles towards infrastructure
“Infrastructure as code” - i.e. configuration management
• Integrating ops with dev, QA and product in the product teams
• Continuous Integration
“Give your developers a pager and put them on call”
• Utilizing more specific metric and monitoring schemes
1.25.8 Better Tools enable Devops
Explosion of new tools over the past few years:
• Release tools (jenkins, travisci, etc)
• Config Management (puppet, chef, ansible, cfengine)
• Orchestration (zookeeper, noah, mesos)
• Monitoring & Metrics (statsd, graphite, etc)
• Virtualization & containerization (AWS, Openstack, vagrant, docker)
1.25.9 It’s not NoOps
• Existing ops principles, processes and practices have not kept pace
• Business & dev teams need more agility to keep up with competitors
• Deep dev skill set + Deep ops skill set == awesomesauce
• Ops people need to do a little dev
• Dev people need to do a little ops
1.25.10 What is Devops Video
1.25.11 Devops Explained: No Horse Manure
1.25.12 Configuration Management
What is it?
“Configuration management is the process of standardizing resource configurations and enforcing their
state across IT infrastructure in an automated yet agile manner.” [PuppetLabs]
1.25.13 History of CM
• mid-1990s – “snowflake system”; few systems
• Rise of Unix-like systems and commodity x86 hardware increased the need
• CFEngine – First release 1993; v2 released in 2002
1.25. Lesson 11: Devops & Configuration Management Intro
127
OSU DevOps Bootcamp Documentation, Release 0.0.1
• mid-2000s through present
– More agile CM systems emerged developed with the cloud in mind
• 2008
– provisioning and management of individual systems were well-understood
1.25.14 Infrastructure as code
• CM enables ops to define their infrastructure in code
• Install packages, configure software, start/stop services
• Ensure a state of a machine
• Ensure policies and standards are in place
• Provide history of changes for a system
• Repeatable way of rebuild a system
• Orchestrate a cluster of services together
1.25.15 CM Platforms
• CFengine
– Lightweight agent system. Manages configuration of a large number of computers using the client–server
paradigm or stand-alone.
• Puppet
– Puppet consists of a custom declarative language to describe system configuration, distributed using the
client–server paradigm.
1.25.16 CM Platforms (part 2)
• Chef
– Chef is a configuration management tool written in Ruby, and uses a pure Ruby DSL for writing configuration “recipes”. Also a client-server model.
• Ansible
– Combines multi-node deployment, ad-hoc task execution, and configuration management in one package.
Utilizes SSH with little to no remote agents.
1.25.17 Puppet Example
• Install apache and start the service
• Puppet Domain Specific Language (DSL)
package { "apache":
name
=> "httpd",
ensure => present,
}
128
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
service {
name
ensure
enable
require
}
"apache":
=> "apache",
=> running,
=> true,
=> Package["apache"],
1.25.18 Chef Example
• Install apache and start the service
• Ruby code
package "apache" do
package_name "httpd"
action :install
end
service "apache" do
action [:enable, :start]
end
1.25.19 CM Platform Comparison
• CFEngine scales like mad, not very agile
• Puppet
– Uses a list of dependencies and figures out what order to run it in
– The Puppet DSL can become a blocker and a problem, puppet also has scaling issues
• Chef
– Executes commands and scripts as they are listed with minimal amount of dependencies
– Using ruby offers both its advantages and disadvantages
• Each platform offers its own level of complexity
1.25.20 References
• http://theagileadmin.com/what-is-devops/
• http://itrevolution.com/the-convergence-of-devops/
• http://en.wikipedia.org/wiki/DevOps
• http://en.wikipedia.org/wiki/Agile_software_development
• What is DevOps? - In Simple English (video)
• DevOps Explained: No Horse Manure (video)
1.25. Lesson 11: Devops & Configuration Management Intro
129
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.25.21 Traditional Development Workflow
Scenario: Developer Mary Smith wants to deploy SystemView to a server administered by Ivan Bofh, a strict oldschool sysadmin.
email conversation link
1.25.22 Email #1
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
On April 3, 2013, at 4:22 PM, Mary Smith <[email protected]> wrote:
Ops team,
As discussed in the release schedule distributed by Mr. Bossman on 2/5, the
development team is ready to deploy our flagship product SystemView this week.
We will need Python 3.4 an Virtualenv on the production server, as well as a
correctly configured Nginx vhost to direct users to the site.
When we log into the production server to deploy the app’s code, we’ll need
permission to write to /var/www and all of /etc for configuration reasons.
Please also create the user and tables detailed in the attached spreadsheet on
our MySql 5.7 database.
Mary Smith
Lead Developer, CruftWare SystemView product division
1.25.23 Email #2
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
On April 5, 2013, at 9:15 AM, Ivan Bofh <[email protected]> wrote:
Mary,
Our production systems are standardized to CentOS 6, so Python is only
supported up to version 2.6. The Python 2.6 version of virtualenv can be
installed after you work with legal to file documentation of a full security
audit of the package.
Providing any account, let alone root, to developers on a production system is
absolutely out of the question. Just document the app’s deployment process
clearly and we’ll handle it.
Ivan Bofh
Senior Systems Engineer, CruftWare
1.25.24 Email #3
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
130
On April 5, 2013, at 11:32 AM, Mary Smith <[email protected]> wrote:
Ivan,
That sounds like it will be simpler to just install the dependencies directly
on the server instead of using virtualenv. I should be able to include this
in the Jenkins configuration, as long as the CI users is running as root.
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
Speaking of which, the development team will need access to Jenkins or other
continuous integration in order to automatically update the site when changes
are pushed.
Is mysql-dev installed yet? Also please confirm that the database is at
systemview-prod.mysql57.cruftware.com.
Mary Smith
Lead Developer, CruftWare SystemView product division
1.25.25 Email #4
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
On April 6, 2013, at 10:08 AM, Ivan Bofh <[email protected]> wrote:
Mary,
Why do you need to use Jenkins? I Googled it and it looks like a non-standard
and immature implementation of some of CFEngine’s features. Just send me a
CFEngine configuration file for the settings that you need. The updates can be
done with an SVN post-commit hook.
Due to administrative decisions that Mr. Bossman explained in a company-wide
memo a couple of months ago, absolutely no dev libraries may be installed on
production servers. Servers are for serving, not for compiling.
Ivan Bofh
Senior Systems Engineer, CruftWare
1.25.26 Email #5
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
On April 6, 2013, at 10:14 AM, Mary Smith <[email protected]> wrote:
Do you at least have the Nginx vhost and uWsgi installation ready?
Mary Smith
Lead Developer, CruftWare SystemView product division
On April 6, 2013, at 1:53 PM, Ivan Bofh <[email protected]> wrote:
We don’t use Nginx or uWsgi. The specs should have said to convert the app to
work with Apache and mod_wsgi for production deployment.
Ivan Bofh
Senior Systems Engineer, CruftWare
1.25.27 Email #6
> On April 6, 2013, at 2:37 PM, Mary Smith <[email protected]> wrote:
>
> Ivan,
>
> What’s the URL for the database?
>
1.25. Lesson 11: Devops & Configuration Management Intro
131
OSU DevOps Bootcamp Documentation, Release 0.0.1
> Mary Smith
> Lead Developer, CruftWare SystemView product division
>
On April 6, 2013, at 4:22 PM, Ivan Bofh <[email protected]> wrote:
Mary,
You’ll have to contact Sharon Negative ([email protected]), our DBA, and
file a ticket to get the database access. She won’t be back from vacation for
another 2 weeks so it might take awhile.
Ivan Bofh
Senior Systems Engineer, CruftWare
1.25.28 DevOps Workflow
Scenario: DevOps-oriented developer Simon Johnson wants to deploy SystemView to a server administered by Ada
Opdev, a DevOps-oriented sysadmin.
irc conversation link
1.25.29 IRC #1
14:03 < JnomiS> AdaOpdev: hey, all the systemview tests are passing on
python 3.4
14:04 <@AdaOpdev> yay! I’ll spin up a VM on the cluster with the production
cookbook
14:04 < JnomiS> that’ll be at sysview23dev.internal.ourcorp, right?
14:05 <@AdaOpdev> actually we’re migrating over to a new test cluster
14:05 <@AdaOpdev> could you use sysview23.dev.ourcorp instead?
14:06 < JnomiS> sure
14:06 <@AdaOpdev> it’s set up for passwordless ssh login with your ldap
account
14:07 < JnomiS> ok, awesome.
14:07 < JnomiS> thanks!
1.25.30 IRC #2
15:12 < JnomiS> hmm, I’ve been building mysql-python for the app with the
mysql dev libraries, but it doesn’t look like you have
those in production
15:22 < JnomiS> also, what database should i connect to from the dev
instance? I’ve been using MySql 5.7 in testing
15:25 <@AdaOpdev> just get me a list of the names and databases you’ll
need, and I’ll plug them into our MySql Chef cookbooks.
15:25 <@AdaOpdev> I checked the ORM docs, and you’re not actually using any
features that changed between MySql 5.5 (which is what
we’ve got in production right now) and 5.7
15:26 < JnomiS> okay, I’ll run the tests with mysql5.5
15:27 < JnomiS> everything seems to be working fine locally
132
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.25.31 IRC #3
16:00 < JnomiS> ugh, I totally forgot -- I’ll need to get root on the dev vm
so I can install uwsgi and nginx
16:00 <@AdaOpdev> We actually manage UWsgi and Nginx through chef as well.
Have you written cookbooks before?
16:02 < JnomiS> nope, I’ve always just used yours :)
16:05 <@AdaOpdev> http://reiddraper.com/first-chef-recipe/ and
https://www.digitalocean.com/community/articles/how-to-create-simple-chef-cookbooks
are a couple of good places to start
16:05 < JnomiS> thanks!
1.25.32 IRC #4
16:52
16:53
16:54
16:54
16:54
16:55
16:55
16:57
16:57
16:58
< JnomiS> do we have jenkins deployed anywhere yet?
< JnomiS> I’d like to get continuous integration set up
<@AdaOpdev> I’ve heard of Jenkins but not worked with it much
< JnomiS> yeah, it’ll automate that deploy hook mess we used to have
<@AdaOpdev> cool!
<@AdaOpdev> let’s talk to our boss about getting an instance
provisioned
< JnomiS> okay, I’ll email him about it and cc you
<@AdaOpdev> thanks
<@AdaOpdev> for now let’s keep using Chef for everything we can
< JnomiS> okay, sounds good.
1.25.33 Non-DevOps
• Poor communication, territorialism (silos)
• Development environment wildly different from production
• Sysadmin averse to changes because environment is hard to test
• Little willingness to cooperate or educate (trust/teamwork)
• It can get even worse
– More people in email thread = more confusion
– Bikeshedding about top post vs bottom post
– Mary is surprisingly clear about her exact requirements
1.25.34 DevOps
• Developer tests on VM with same config as production
• Realistic expectations about access and compatibility
• More automation, configuration management
• Sysadmin can debug the code and help developer
• Developer and sysadmin help and educate one another (Chef, Jenkins)
• Distributed tasks mean fewer choke points (single person who can block task)
1.25. Lesson 11: Devops & Configuration Management Intro
133
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.26 Lesson 12: Configuration Management
Note: starting off with cs312 content
1.26.1 Large site management
• Configuration Management
• Automation
• Centralized
• Standardization
• History
Note:
• Automating most of what you do
• Scaling the configuration management is key
• Centralized so that a team can work together
• History via SCM (source control management)
1.26.2 What config management does
• Verify permissions for security/mgmt purposes
• Distribute config files and scripts
• Control batch jobs
• Ensure packages are up to date
• Look for file changes
Note: Ensures that critical configs always are set like you want. No mucking around by a bad admin
1.26.3 Configuration Management
• System-wide settings
– sshd, ntp, ldap, etc
• Group settings
– Web, email, database
• Standardize settings globally
• Easy to troubleshoot
134
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.26.4 CF Management Tools
• CFEngine
– Mature, widely used
• Puppet
– Getting mature, different concept
• Chef
– Very new
Note: Different concepts Feature set different between them Each have their own issues
1.26.5 Puppet
Let’s install puppet
$ yum install puppet
1.26.6 Declarative Configuration
• We ‘declare’ the desired state of the system
• Puppet does the necessary work to make the system match our declaration
• We can save these declarations in a repository, just like code
1.26.7 Puppet Resources
Puppet knows about “resources” on the system
$ puppet describe -l
Look at all those things Puppet can manage out of the box. We’re most interested in these:
file
group
package
service
user
-
Manages files, including their content, owner ...
Manage groups
Manage packages
Manage running services
Manage users
1.26.8 Resources Have Attributes
# let’s look at the vagrant user
$ sudo puppet resource user vagrant
user { ’vagrant’:
ensure
gid
groups
home
password
=>
=>
=>
=>
=>
’present’,
’500’,
[’wheel’],
’/home/vagrant’,
’$1$aDsSD/Uu$.tXG5wN.TSit1AP5ZyphB0’,
1.26. Lesson 12: Configuration Management
135
OSU DevOps Bootcamp Documentation, Release 0.0.1
password_max_age
password_min_age
shell
uid
=>
=>
=>
=>
’99999’,
’0’,
’/bin/bash’,
’500’,
}
We can declare a value for any of those attributes, and Puppet will make it happen.
Note: the password is a password hash, as appears in /etc/shadow - don’t put passwords in puppet manifests!
1.26.9 Puppet Manifests
Puppet keeps its declarations in manifest files. We can write a manifest to create a user:
$ sudo su $ vim users.pp
user {’yournamehere’:
ensure
=> ’present’,
home
=> ’/home/yournamehere’,
groups
=> [’wheel’, ’vagrant’],
shell
=> ’/bin/tcsh’,
}
1.26.10 Pull the Strings
Lets run our manifest.
> puppet apply user.pp
Notice: Compiled catalog for devops-bootcamp.osuosl.org in
environment production in 0.12 seconds
Notice: /Stage[main]/Main/User[yournamehere]/ensure: created
Notice: Finished catalog run in 0.13 seconds
Note: we are using stand-alone mode, manually running an individual manifest
1.26.11 Declarations Are Idempotent
Lets run our manifest again.
> puppet apply user.pp
Notice: Compiled catalog for devops-bootcamp.osuosl.org in
environment production in 0.12 seconds
Notice: Finished catalog run in 0.02 seconds
The state of the system is already what we declared it should be, so applying the manifest again doesn’t change
anything.
Note: idempotency is important, the puppet master daemon will run periodically, and it is important that running the
same commands over and over does not have cumulative effects
136
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.26.12 Packages and Services
We can declare that our system should have certain things installed and running.
apache.pp:
package{’httpd’:
ensure => ’present’
}
service{’httpd’:
ensure => ’running’,
enable => ’true’,
require => Package[’httpd’],
}
Note: The ‘service’ block makes sure that the httpd service is started, and that it is enabled, the ‘require’ directive
tells the service that it must wait until the package ‘httpd’ is processed. Services are anything you would start with
“service x start” and packages anything you would install with “yum install x”
1.26.13 Puppet Config
Where does Puppet keep its configuration files?
Note: the audience really ought to know where to start looking by this point
1.26.14 /etc/puppet
$ ls /etc/puppet
auth.conf modules
puppet.conf
• puppet.conf - systemwide configuration
• auth.conf - puppet agent configuration
• modules - we’ll talk about that later
Note: there isn’t much of anything we need to worry about in any of the config files
1.26.15 The Site Manifest
We
want
to
move
beyond
running
individual
manifests
on
‘/etc/puppet/manifests/site.pp‘ is the place to put your site’s configuration.
the
command
line.
$ mkdir /etc/puppet/manifests
$ vim /etc/puppet/manifests/site.pp
1.26.16 But First, Nodes
• Nodes are defined in the site manifest
• A node is a single machine, identified by its FQDN (Fully-Qualified Domain Name).
1.26. Lesson 12: Configuration Management
137
OSU DevOps Bootcamp Documentation, Release 0.0.1
• You can define many nodes.
• You can add declarations to a node definition.
• A special ‘default’ node will be used if a node’s name can’t be found.
We will put our configurations in the default node for now.
Note: a node can inherit from another node, but this is discouraged
1.26.17 An Example Site Manifest
node default {
file {’/etc/issue’:
path
=> ’/etc/issue’,
mode
=> 644,
ensure => present,
content => "Welcome to the DevOps Bootcamp VM.\n",
}
package{’httpd’:
ensure => ’present’
}
service{’httpd’:
ensure => ’running’,
enable => ’true’,
require => Package[’httpd’],
}
}
Note: have we talked about /etc/issue? The file resource lets you declare the filename, ownership, and contents. You
can also have it copy files from the module onto the node instead of manually inserting content here.
1.26.18 The Master and the Agent
Puppet uses a Master/Agent architecture.
• The Master reads the ‘site.pp‘ and listens for an Agent to contact it.
• Agents run on nodes, they contact the master to get their configuration
• Master and Agent can be on the same machine.
• When they are on different machines, they need an SSL certificate to authenticate
Run the master on your vm:
$ puppet master
Note: the master will background by default and log to syslog, but you can run it in the foreground with –nodaemonize and get extra logging on stdout with –verbose
138
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.26.19 The Agent
The agent will look for its master on the host ‘puppet‘ by default. Lets add the hostname ‘puppet‘ to our local
host definition in /etc/hosts, so it will look on the local machine.
$ vim /etc/hosts
127.0.0.1
devops-bootcamp.osuosl.org devops-bootcamp localhost
localhost.localdomain localhost4 localhost4.localdomain4 puppet
^^^^^^
Now run the agent in test mode:
$ puppet agent --test --verbose
Note: the agent will also background by default, the –test flag prevents that and shows us what is going on. In a
production environment, the master and agent would always be running in the background, usually started as services
on boot.
1.26.20 Modules
We can keep adding configurations to site.pp, but it’s going to get long and messy. Let’s use modules instead.
• Modules are classes
• Modules encapsulate a set of related configurations
• Modules make it easy to apply configurations to many nodes
• Community created modules already exist for almost everything
Note: community or puppetlabs modules vary in quality, always read the docs thoroughly
1.26.21 Module Structure
/etc/puppet/modules/
modulename/
files/
some_file
manifests/
init.pp
some_other_manifest.pp
Note: that files directory is served to the puppet agent like a fileserver, file resources can declare their source attribute
like “puppet:///modules/module_name/some_file” and the file will be copied into place
1.26.22 The Bootcamp Apache Module
#
$
$
$
$
Let’s create a module for our Apache configuration.
cd /etc/puppet/modules
mkdir bootcamp_apache
mkdir bootcamp_apache/manifests
vim bootcamp_apache/manifests/init.pp
1.26. Lesson 12: Configuration Management
139
OSU DevOps Bootcamp Documentation, Release 0.0.1
class bootcamp_apache {
package{’httpd’:
ensure => ’present’
}
package{’mod_wsgi’:
ensure => ’present’
}
service{’httpd’:
ensure => ’running’,
enable => ’true’,
require => Package[’httpd’],
}
}
Note: it is good practice to namespace the class name of your modules, so instead of just ‘apache’, we use bootcamp_apache, which won’t collide with any other apache related module.
1.26.23 Site.pp Modularized
node default {
file {’/etc/issue’:
path
=> ’/etc/issue’,
mode
=> 644,
ensure => present,
content => "Welcome to the DevOps Bootcamp VM.\n",
}
include bootcamp_apache
}
Note: the include statement assumes a module located in modules/ under the pupper config dir. The name is the class
name of the the module, which is not necessarily the directory name the module is stored under (but it is much easier
to name them the same)
1.26.24 Community Modules
We need MySql installed for our SystemView app, as well as a database, user, and permissions. We could do all that
with package, service and file resources, but there is a better way, the puppetlabs-mysql module.
https://github.com/puppetlabs/puppetlabs-mysql
(It’s in Git, how convenient!)
$ cd /etc/puppet/modules/
# We’ll clone into a directory named mysql, because that’s the module name
$ git clone https://github.com/puppetlabs/puppetlabs-mysql.git mysql
We can include this module’s class into our site manifest or our own modules.
1.26.25 The Bootcamp Mysql Module
We want to create a database and users, so lets make a module and not clutter up the site.pp
140
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
$
$
$
$
cd /etc/puppet/modules
mkdir bootcamp_mysql
mkdir bootcamp_mysql/manifests
vim bootcamp_mysql/manifests/init.pp
class bootcamp_mysql {
class { ’::mysql::server’ }
}
::mysql::server causes Puppet to install MySql and makes available many methods for managing MySql.
Note: Calling the ‘mysql’ class essentially includes that module, which includes a package declaration insuring mysql
is installed. It is easy to explore the module files and see what is in it.
1.26.26 Databases, Users, and Grants
class bootcamp_mysql {
class { ’::mysql::server’ }
mysql_database { ’systemview’:
ensure => ’present’,
charset => ’utf8’,
collate => ’utf8_swedish_ci’,
}
mysql_user { ’vagrant@localhost’:
ensure => ’present’,
}
mysql_grant { ’vagrant@localhost/systemview.*’:
ensure
=> ’present’,
options
=> [’GRANT’],
privileges => [’ALL’],
table
=> ’systemview.*’,
user
=> ’vagrant@localhost’,
}
}
Note: the mysql module has a lot of stuff in it, there isn’t time to get into it all.
1.26.27 Test It Out
$ puppet agent --test --verbose
1.26.28 Further Reading
• http://docs.puppetlabs.com/learning/introduction.html
• https://github.com/puppetlabs/puppetlabs-mysql
1.26. Lesson 12: Configuration Management
141
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.27 Lesson 13: Contributing to Open Source
1.27.1 Getting started in Open Source
• Carlos’s CS419 in a nutshell
• Relevant books
– The cathedral and the bazaar (Eric Raymond)
– Producing Open Source Software (Karl Fogel)
– Open Advice (Lydia Pintscher)
1.27.2 Joining vs Starting a Project
• Scratch an itch.
• Research first
• Revive dead project vs. rewrite
• Get involved with communities even if you plan to start your own
– Learn from their examples
1.27.3 Know your licenses
Note: I am not a lawyer.
http://opensource.org/licenses is pretty cool
” Freely used, modified, and shared”
MIT/X11 Short, permissive, says attribution and no liability. Doesn’t discuss copyright. Can convert to
Apache 2
Apache Short, permissive, goes in every file, grants patent rights from contributors to users, author keeps
copyright. Plays nice with GPL3 (?)
BSD Attribution, keep copyright, no liability
AGPL Demands source distribution even when software not distributed (for cloud/hosted)
GPL Viral, copyleft. Viral = infects entire program if it links to GPL library or uses a single line of
GPL’d code
LGPL Fixes library linking issue with GPL
CC Non-code content
• MIT/X11
• Apache
• BSD
142
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
• AGPL/GPL/LGPL
• Creative Commons
• http://choosealicense.com/
1.27.4 Assessing a new community
• Elitism vs welcomeness
• Communication style
• Documentation and guides
• Faster or slower change?
1.27.5 Getting involved
• Lurk more
• Make accounts
– Mailing lists
– IRC channels
– Wikis
– Issue trackers
• Your nick is your reputation
• It’s okay to make mistakes... But learn from them.
1.27.6 Finding a project
Note: First contributions will be metric of how nice they are to newbies
1.27. Lesson 13: Contributing to Open Source
143
OSU DevOps Bootcamp Documentation, Release 0.0.1
144
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
There’s a thing where older project members get grumpy at newbies because they’ve answered the question over and
over... read docs/faq then improve them
• Openhatch
• Easy bugs
• GSOC submitters who didn’t get enough interns
• Search by language
• Search by project type – find something that interests you (web dev? bioinformatics? video games?)
• Your immediate payment for contributions will be satisfaction, so pick something satisfying
1.27.7 First steps
Note: It will feel like you have only a vague idea what you’re doing. This means you’ve found a project that’s
challenging and that you’ll learn from.
• Lurk awhile then ask
• Write a test
• Fix a typo
• Deploy and update the installation docs
1.27.8 DevOps Concerns
• Configurations often managed in public repos
1.27. Lesson 13: Contributing to Open Source
145
OSU DevOps Bootcamp Documentation, Release 0.0.1
146
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
• Root can’t be handed out to just anyone
• Build trust, contribute to project consistently
• Practice with the tools they use
1.27.9 Your Homework
• Find a project that you’d like to get involved with this summer
• Join IRC, mailing lists, etc.
• Pull the code and run its tests using what you’ve learned
• Find something you can contribute to the project
• Discuss how it’s going in #devopsbootcamp on irc.freenode.net
1.27.10 Questions?
Any questions about anything from this year?
• Conferences: OSBridge, OSCON may have free expo hall passes
• In Corvallis? Want to come to the OSL and see what we do, pair program, etc.?
• No meeting next week – please leave feedback!
1.28 Lesson 14: Review
1.28.1 Today’s Goals
• Working VM
• Run Systemview app
• Connect to Systemview from browser in host
• Understand what’s happening
1.28.2 How to set up a VM
1.28.3 Enable VirtualBox (VT extensions)
Install VirtualBox
Install Vagrant
1.28.4 Linux
# clone
git clone https://github.com/DevOpsBootcamp/devopsbootcamp-vagrant.git
# start up
cd devopsbootcamp-vagrant
1.28. Lesson 14: Review
147
OSU DevOps Bootcamp Documentation, Release 0.0.1
vagrant up
# access vm
vagrant ssh
1.28.5 IRC
• Use the LUG guide
• #devopsbootcamp on irc.freenode.net
• Also join the Mailing List
1.28.6 GitHub Account
SSH keys
ssh-keygen -t rsa
Your public key is in ~/.ssh/id_rsa.pub by default.
GitHub -> Account Settings (icon in upper right) -> SSH keys -> Add SSH key
1.28.7 Joining the Github Org
• Ping Ramereth in the IRC channel to add you
1.28.8 Installing the Web App
1.28.9 Prerequisite tools
On host machine, in devopsbootcamp-vagrant directory:
git pull
vagrant up
vagrant ssh
1.28.10 The Magic Script
Now that you’re in the guest machine:
[email protected]:DevOpsBootcamp/catch-up.git
cd catch-up
./catch-up.sh
1.28.11 Branches
Start one with
148
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
git checkout -b branchname
Which are you on
git branch
Switch with
git checkout branchname
1.28.12 Connecting to the App
In the guest machine, with virtualenv activated, python systemview.py
Point the browser of your host machine at http://127.0.0.1:5050
If changes in the app don’t show up in your browser, use F5 to hard refresh
1.28.13 Where am I?
1.28.14 In virtual machine?
• Did you vagrant ssh?
1.28.15 In a repo?
• git status
1.28.16 On a branch?
# Show current branch
$ git branch
# create new branch, called branchname
$ git checkout -b branchname
1.29 Lesson 16: Email
• Email service
– How it works
– Configuration Postfix
– Planning
1.29. Lesson 16: Email
149
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.29.1 Email: System Components
• Mail User Agent (MUA)
• Mail Transport Agent (MTA)
• Delivery Agent (MDA)
• Access Agent (MAA)
Note:
MUA
• lets users read & compose mail
• Thunderbird, mutt, etc
MTA
• routes messages to other machines
• sendmail, postfix, exim, qmail
MDA
• places messages in local store
• mail.local, procmail
MAA access to mail store (i.e IMAP, POP)
1.29.2 Email: System Components
Note: The most confusing part about email is understanding the routing. Knowing the different components is
important to fully grasping it.
150
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
1.29.3 Transport Agents
Accept mail form user agent
Postfix More common, easier to configure & use
Sendmail Highly configurable, steep learning curve
Exim Similar to Postfix
Qmail Logging is horrid, but some people like it
Note: Postfix is the easiest to learn and understand, but queue management is a “black box”
Sendmail & qmail is great for high volume sites, but postfix/exim still perform great.
Sendmail has great options for queue management
Features to look out for:
• SASL (authenticated SMTP)
• Queue Management
1.29.4 Delivery Agents & Message Stores
• procmail – great filtering
• maildrop – newer procmail-like
• mail.local
• Message Stores
– mbox – one large file, locking problems
– maildir – one file per message, great for IMAP
Note: Consider scaling issues for the mailstore.
Generally maildir is the best & most compatible option
1.29.5 Anatomy of a Mail Message
• Envelope
– Destination email address
• Headers
– Record of variety of important information
– Great for tracking down problems
• Body of the message
Note:
Headers:
• Know how to identify and track queue id’s
• Originator starts at the bottom
1.29. Lesson 16: Email
151
OSU DevOps Bootcamp Documentation, Release 0.0.1
• Headers can be forged
• X- Headers non-RFC headers
• Message ID is always unique
1.29.6 MTA Log Files
• Track emails via queue ID
– Look something like: 03CE18819A
• Tracking via message ID
• Informational fields
– to, from, status, relay, etc
• Log files differ between each MTA
Note: Being able to read log files is important.
1.29.7 Configuring Postfix
• /etc/postfix
– main.cf – main config file
– master.cf – postfix process config file
– /etc/aliases – local email forwarding
• Set to relay email to central MTA
– relayhost = [smtp.osuosl.org]
– myorigin = osuosl.org
– /etc/aliases – root: [email protected]
Note:
relayhost: [smtp.osuosl.org] vs. osuosl.org
• [smtp.osuosl.org] goes directly to smtp.osuosl.org
• ‘osuosl.org’ does DNS lookup and uses MX
Make sure you run “newaliases” after updating /etc/aliases
Reloading postfix is ideal too
To test email: echo “this is a test” | mail root@localhost
1.29.8 Sendmail
• Config files created via m4
– Makefile
152
Chapter 1. Contents:
OSU DevOps Bootcamp Documentation, Release 0.0.1
• Always edit the .mc files not the .cf files
• Remember to rebuild .cf files with make
• Extremely configurable
Note: Config files in /etc/mail usually Primary file to edit should be sendmail.mc
1.29.9 Email: Viruses & Spam
• Virus
– Clamav
– Ensure freshclam is running too
• Spam
– Spamassassin
• All-in-one
– Amavis
• Check abuse emails
Note: Make sure you have enough CPU & RAM for Spam checking Neglecting abuse emails may get you blacklisted
For larger infrastructures, have dedicated machines to process spam Important to keep these updated
1.29.10 Email: Infrastructure Implementation
• Small sites
– Can have MTA/MDA/etc all on the same server
• Medium sites
– Separate MTA from MDA
• Large sites
– Split outgoing mail and incoming
Note: Consider resources, redundancy, & scalability. MDA is hardest to scale.
• Look at Cyrus Murder for large scalability
• dovecot is another option
1.29.11 Email: Security
• On General servers:
– Only listen on localhost
– Don’t allow other hosts to relay through it
– Relay all outbound mail through central host
1.29. Lesson 16: Email
153
OSU DevOps Bootcamp Documentation, Release 0.0.1
• On Email servers:
– Restrict relaying to trusted networks
– Implement antivirus & spam protection
Note: Always test new configurations to ensure spammers can’t relay mail through your server Having dedicate
outbound servers will ensure they always catch spam/viruses/etc
154
Chapter 1. Contents:
CHAPTER 2
Indices and tables
• genindex
• modindex
• search
155
OSU DevOps Bootcamp Documentation, Release 0.0.1
156
Chapter 2. Indices and tables
Bibliography
[Wikipedia] http://en.wikipedia.org/wiki/DevOps
[PuppetLabs] http://puppetlabs.com/solutions/configuration-management
157