Chef on steroids

The problem

At Picklive, we love Ruby and therefore we love Chef. But Chef has two little glitches: he’s a bit too talkative and he doesn’t like SSL.

Since our main activity is gambling, we have to pass strict security audits and then displaying full backtraces of Chef errors to everyone on is not seen in a good light. Unfortunately, Chef logging level is not very configurable and it could be really verbose - even running in production mode. And hacking it is not an option since it would makes it hard to maintain up-to-date. So we have to filter error pages as HTTP 4xx and 5xx. It’s now that another great piece of software comes into play: Nginx.

Let’s proxy Chef through Nginx and return a custom error page instead of Chef backtraces. While we’re at it, we could put all Chef traffic over SSL to avoid sending potential sensitive information over the hazardous Internet.

Set up Nginx as proxy

Just let’s add this to our host configuration:

location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X_FORWARDED_PROTO $scheme;
    proxy_set_header Host $host;

    proxy_connect_timeout 4;
    proxy_read_timeout 30;
    proxy_send_timeout 30;

    proxy_redirect off;
    proxy_max_temp_file_size 0;

    access_log /var/log/nginx/ upstream;
    proxy_pass http://localhost:4000;

Finally we say to chef to listen only on localhost with -h localhost and then restart the services.

Everything is broken!

Just try to upload a cookbook to test whether the proxy is working:

$ knife cookbook upload picklive -l debug
DEBUG: /var/folders/ti/tik1tgFzHlBl3tMqXZGAaE+++TQ/-Tmp-/chef-picklive-build20110825-1460-1wrs8m6-0/picklive/recipes/application.rb has not changed
DEBUG: Committing sandbox
DEBUG: Signing the request as xxx
DEBUG: Sending HTTP Request via PUT to
/Users/xxx/.rvm/rubies/ree-xxx/lib/ruby/x.x/net/http.rb:xxxx:in `error!': 405 "Not Allowed" (Net::HTTPServerException)


Obviously, we didn’t expect this weird 405 "Not Allowed" error. But if we look the line just above, we can spot what’s the problem. Yes, I’m talking about this whereas we were expecting it in port 4000 - the port upon which Chef is listening. So let’s launch Wireshark and see what’s going on. It starts with some classic GET/POST requests to end up on a 201 Created followed by the PUT request above. The problem is that Chef sends us an URL to a new resource without the :4000:


So the problem is on the server-side.

Let’s sniff the traffic between Nginx and Chef with ngrep - a tcpdump-like but more suitable to analyse HTTP traffic.

$ sudo ngrep -W byline -d lo port 4000

T -> [A]
PUT /cookbooks/picklive/0.1.0 HTTP/1.0.
Connection: close.
Accept: application/json.

T -> [AP]
HTTP/1.1 201 Created.
Content-Type: application/json; charset=utf-8.
Connection: close.
Server: thin x.x.x codename Flaming Astroboy.

As we can see, the :4000 is not removed by Nginx thus it’s Chef which doesn’t set it at the beginning of the chain. Why?! Well, just take a look at our Nginx configuration again before using some dirty hack like


Oh wait, maybe if we set the right port here:

proxy_set_header Host $host:4000;

Oh yeah! It was that, we are now able to use knife again.

The actual work

Now, filtering error messages and putting traffic over SSL is a piece of cake.

Filter error messages

Simply use the error_page directive offered by Nginx:

# Return a dummy page for all standard errors because chef-server is too verbose.
proxy_intercept_errors on;
error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 495 496 497
           500 501 502 503 504 505 506 507 /50x.html;
location = /50x.html {
    root  /var/www/nginx-default;


Keep it simple, stupid.

ssl on;
ssl_certificate     /etc/ssl/<%= node[:chef][:server][:certificate][:file] %>;
ssl_certificate_key /etc/ssl/<%= node[:chef][:server][:certificate][:key] %>;


Posted on 02 September 2011 by Cedric @infertux.
blog comments powered by Disqus