[Devel] [Sysadmin] Puppet files

[Asheesh Laroia]

Hi Jack, and anyone else who's interested in server move updates,

(Other people who don't care about the server move can probably mute this 
thread.)

I've gotten to a good point to explain my vision for how we can use Puppet 
to make the new server quite comprehensible. This is a long email with 
that vision.

Here's what I think we should do:

How to use what I've made so far
--------------------------------

First, you should read http://people.redhat.com/dlutter/puppet-app.html . 
I mean it! It's old but good.

You should "git clone" the new https://gitorious.org/openhatch/oh-puppet 
repository on the new machine. For now, it doesn't matter much where in 
the filesystem you clone it to. You don't need to clone it as root; you 
can clone it as your own user ID.

You should run 'puppet -v node.pp' as root (perhaps with sudo) and watch 
as Puppet (verbosely, hence '-v') explains what it is doing, and why.

You'll notice that Puppet creates the OpenHatch deploy user and does the 
right "git clone".

More background info
--------------------

The way I see it, the Puppet configuration we end up with should be 
idempotent -- when you run Puppet, it checks the current state of the 
system, and only acts if something needs changing. So if you run it a 
second time with the same Puppet config files, it will do nothing.

The goal of the files is to describe how to create a (nearly) 
fully-working OpenHatch server, including our nginx and Apache 
configuration, creating the Roundup service, and creating the wiki. It 
should be possible to test the Puppet files by creating a new VM on your 
own machine and running this stuff. In fact, that's how I recommend 
developing the scripts, rather than developing them on the OSU OSL 
machine.

(I say "nearly": I think our SSL private key shouldn't be in there. That's 
probably okay.)

I realize that, since we have a flexible configuration format and an 
interpreter for it, we're going to end up "programming" the server, 
optimizing the Puppet scripts for readability and cleanliness. If you're 
cynical, you might worry that we'll end up "wasting" time improving "code 
quality" (quality of the Puppet files). I accept that's a risk but I think 
we'll end up with a good balance of spending time improving them and 
learning Puppet where the time investment helps us spend less time on 
sysadmin tasks later (and also make more efficient use of volunteers).

The official way to make changes to the server should be to change the 
Puppet file, not to be to run commands on the main server. Except...

...I still think we should keep using the "deploy" script that we have 
now. The script has the advantage of running as just one unprivileged user 
ID, and I (for one) really like how quickly it works.

Adding someone to the "Login Team" (as well as removing them) will be 
easy: we'll just add their SSH key information to the Puppet configuration 
file, commit to git, and tell Puppet on the main server to run itself with 
the freshest configuration from git. Ta-da.

Things we are doing "unusually"
-------------------------------

We're not using the puppet daemon at all (called puppetmaster), neither 
for distributing config files, nor for passing Puppet files to the server. 
We just have the one server, and we can store all of the fun stuff in git.

Things we are doing "wrong"
---------------------------

We're using a non-idiomatic filelayout. Honestly, we should fix that. 
We're supposed to use "modules" and "manifests". I'm still a Puppet newbie 
so I did it in the easiest way I could put it together.

Similarly, I don't understand everything about Puppet classes yet. There's 
some "->" syntax that I don't understand for expressing dependencies, and 
it seems pretty cool. There's some "::" thing that I don't understand yet, 
either.

Next steps
----------

Jack, please try the above on the OSL server. Anyone else can follow along 
on a VM on their own machine, if you don't have root on the OSL box.

Once you and I are on the same page about the way this will work, then I 
add a few more things into the Puppet config. (That means: Jack, I'm 
waiting on your ACK before I move forward. That way I can take a quick 
break from Puppet-ification and relax for a while!)

Other links
-----------

As I learn and find resources I like, I'm bookmarking them at 
http://bookmarks.makesad.us/insipid.cgi?tag=puppet .  I especially 
recommend the cool-headed 
http://current.workingdirectory.net/posts/2011/puppet-without-masters/ . 
You can read them and/or suggest other links that would be useful.

P.S. Wow, this email got long.

-- Asheesh.