Just Enough Developed Infrastructure

Virtualbox Savestate Restore Network Problems

(2010-08-30) - Comments

Virtualbox Savestate and restore does not restore the network

I make extensive use of Virtualbox to test my installations. When I automate the installation , I use the savestate and restore to rollback changes I do to the virtual machine. Recently the network of the virtual machine would not be accessible anymore after the restore. A network ping from both inside or outside the virtualmachine fails. Several tickets have been created for it, and the bug seems to pop up every new update or so Fixed in 25205 , re-introduced in Ticket 5666, Ticket 5654 .

The first work-around I found was:

On a linux machine restart the network stack with a /etc/init.d/network restart . But that is not very portable, and it's hard to automate because the network is gone.

A more generic work-around is to temporary disable the network link:

$ VBoxManage controlvm <yourvmname> setlinkstate1 off
$ VBoxManage controlvm <yourvmname> setlinkstate1 on
$ sleep 2

NAT: re-establish port-forwarding after savestate / restore state

Another quite annoying bug is that virtualbox sometimes looses it's NAT mappings Ticket 25402 There is no other way then re-executing the NAT mapping.

Capturing the Screen or Video of Your Virtual Machines Using X, Vnc, Rdp or Native ways

(2010-08-30) - Comments
With this blogpost we continue the previous investigation to interact with a virtual machine using X, VNC , RDP or native ways with a virtual machine. This time instead of sending keystrokes we are looking for capture screenshots or even capture the complete session as a video.

Interacting with X-Windows

Capturing a screenshot:

Grabbing a screen of an X-Windows session is easy: in order to grab the screen on an X-session on :1 issue the following command:

$ import -window root -display :1 screenshot.png

Recording a video:

I got the idea of using the ffmpeg command for capturing the X-session from

http://gasubasu.com/2009/05/06/flash2video/

So I installed ffmpeg using macports

$ sudo port install ffmpeg

When I executed it, it complained about Unknown input format: 'x11grab'

$ ffmpeg -f x11grab -vc x264 -s vga -r 30 -b 2000k -g 300 -i :1.0 session-recording.avi
FFmpeg version 0.6, Copyright (c) 2000-2010 the FFmpeg developers
  built on Aug 29 2010 16:56:51 with gcc 4.2.1 (Apple Inc. build 5664)
  configuration: --prefix=/opt/local --enable-gpl --enable-postproc --enable-swscale --enable-avfilter --enable-avfilter-lavf --enable-libmp3lame --enable-libvorbis --enable-libtheora --enable-libdirac --enable-libschroedinger --enable-libfaac --enable-libfaad --enable-libxvid --enable-libx264 --enable-libvpx --enable-libspeex --enable-nonfree --mandir=/opt/local/share/man --enable-shared --enable-pthreads --disable-indevs --cc=/usr/bin/gcc-4.2 --arch=x86_64
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     1.11. 0 /  1.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0
Unknown input format: 'x11grab'

There was no variant or option for macports to have ffmpeg take this flag for compiling. So I downloaded the sourcecode from the ffmpeg repository:

$ wget http://www.ffmpeg.org/releases/ffmpeg-0.6.tar.gz
$ cd ffmpeg-0.6
$  ./configure --enable-x11grab --enable-gpl

Although I had specified the --enable-x11grab option, ffmpeg still complained about the Unknown input format.

I got some clues here and here. And after a look at the the configure file, I found that in order to work, it required the Xfixes and Xext libraries:

enabled x11grab                         &&
check_header X11/Xlib.h                 &&
check_header X11/extensions/XShm.h      &&
check_header X11/extensions/Xfixes.h    &&
check_func XOpenDisplay -lX11           &&
check_func XShmCreateImage -lX11 -lXext &&
check_func XFixesGetCursorImage -lX11 -lXext -lXfixes

So after installing those libraries:

$ sudo port install xorg-libXext
$ sudo port install xorg-libXfixes
$  ./configure --enable-x11grab --enable-gpl --enable-nonfree --extra-cflags="-I/opt/local/include -I/opt/local/include/X11"  --extra-ldflags=-L/opt/local/lib

....
Enabled indevs:
x11_grab_device
...

Now with the input x11_grab_device available, I was all set to record a running session.

$ ./ffmpeg -f x11grab -vc x264  -s xga -r 30 -b 2000k -g 300 -i :1.0 session-recording.avi
FFmpeg version 0.6, Copyright (c) 2000-2010 the FFmpeg developers
  built on Aug 30 2010 09:11:02 with gcc 4.2.1 (Apple Inc. build 5664)
  configuration: --enable-x11grab --enable-gpl --enable-nonfree --extra-cflags='-I/opt/local/include -I/opt/local/include/X11' --extra-ldflags=-L/opt/local/lib --disable-shared
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libswscale     0.11. 0 /  0.11. 0
[x11grab @ 0x10100b000]device: :1.0 -> display: :1.0 x: 0 y: 0 width: 1024 height: 768
[x11grab @ 0x10100b000]shared memory extension  found
[x11grab @ 0x10100b000]Estimating duration from bitrate, this may be inaccurate
Input #0, x11grab, from ':1.0':
  Duration: N/A, start: 1283153847.281641, bitrate: 754974 kb/s
    Stream #0.0: Video: rawvideo, bgra, 1024x768, 754974 kb/s, 30 tbr, 1000k tbn, 30 tbc
File 'session-recording.avi' already exists. Overwrite ? [y/N] y
Output #0, avi, to 'session-recording.avi':
  Metadata:
    ISFT            : Lavf52.64.2
    Stream #0.0: Video: mpeg4, yuv420p, 1024x768, q=2-31, 2000 kb/s, 30 tbn, 30 tbc
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding
frame=  310 fps= 30 q=2.0 Lsize=    2810kB time=11.83 bitrate=1945.1kbits/s dup=0 drop=21    
video:2796kB audio:0kB global headers:0kB muxing overhead 0.502798%

This worked fine! But the recording only captured 1024x768 (flag = xga), when I tried to have ffmpeg capture a larger screen 1600x1024 (flag wsxga), it told me an error about Can't get shared memory . No clue why. For now, the xga option is enough for me.

./ffmpeg -f x11grab -vc x264  -s wsxga -r 30 -b 2000k -g 300 -i :1.0 session-recording.avi
FFmpeg version 0.6, Copyright (c) 2000-2010 the FFmpeg developers
  built on Aug 30 2010 09:11:02 with gcc 4.2.1 (Apple Inc. build 5664)
  configuration: --enable-x11grab --enable-gpl --enable-nonfree --extra-cflags='-I/opt/local/include -I/opt/local/include/X11' --extra-ldflags=-L/opt/local/lib --disable-shared
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libswscale     0.11. 0 /  0.11. 0
[x11grab @ 0x10100b000]device: :1.0 -> display: :1.0 x: 0 y: 0 width: 1600 height: 1024
[x11grab @ 0x10100b000]shared memory extension  found
[x11grab @ 0x10100b000]Fatal: Can't get shared memory!

Interacting with VNC

Capturing a screenshot:

I found two projects that touted the ability to capture the screen of a VNC session:

None of them really worked for me. So I resorted back to using Xvfb as an intermediate:

$ vncpasswd mypasswordfile
$ Xvfb  :1 -screen 0 1024x768x24 -fbdir /var/tmp/
(EE) AIGLX error: dlopen of /usr/X11/lib/dri/swrast_dri.so failed (dlopen(/usr/X11/lib/dri/swrast_dri.so, 5): image not found)
(EE) GLX: could not load software renderer
(EE) XKB: Couldn't open rules file /usr/X11/share/X11/xkb/rules/base
(EE) XKB: No components provided for device Virtual core keyboard
$ DISPLAY=:1 rdesktop -u username -p password -d domain remotehost
$ import -window root -display :1 screenshot-rdp.png

Recording a video:

To record an vnc session to flash I found the following pointers:

I went for the C version of vnc2swf :

$ wget http://www.unixuser.org/~euske/vnc2swf/vnc2swf-0.5.0.tar.gz
$ cd vnc2swf-0.5.0
$  LDFLAGS=-L/usr/X11R6/lib  ./configure 
$  LDFLAGS=-L/usr/X11R6/lib  make
$ ./vnc2swf
usage: ./vnc2swf [<options>] <out.swf> <host>:<display#>
       ./vnc2swf [<options>] <out.swf> -listen [<display#>]

<options> are standard Xt options, or:
              -shared
              -viewonly
              -fullscreen
              -passwd <passwd-file>
              -noauto
              -encodings <encoding-list> (e.g. "raw copyrect")
              -geometry <geometry>
              -bgr233
              -owncmap
              -truecolour
              -depth <depth>
              -framerate <frames-per-sec>
              -startrecording
              -nowindow
              -nostatus
              -clippinggeometry <geometry>

Àn alternative is to ffmpeg as explained in the X-session, with vncviewer displayed in Xvfb session

Interacting with RDP

Capturing a screenshot:

I found 0 tools to capture a screenshot directly from an RDP (at least a tool on Linux or MacosX). While poking around I found a reference to VNC inside the code of Rdesktop: there was mention of rdp2vnc

$ ./configure --help |grep vnc
  --with-libvncserver-config=CMD  use CMD as libvncserver-config
  --with-libvncserver     make rdp2vnc

I almost got it to work by checking out the latest svn

$ svn co https://rdesktop.svn.sourceforge.net/svnroot/rdesktop rdesktop
$ cd rdesktop

It required the installation of LibVNCServer library - http://libvncserver.sourceforge.net/

./configure --prefix /opt
make
make install

$ cd rdesktop
$ ./configure  --with-libvncserver  --x-includes=/opt/local/include/ --x-libraries=/opt/local/lib --with-libvncserver-config

But in the end - nothing and I was facing a Compile problem

So of to the X-session fallback using rdesktop and xvfb:

$ Xvfb  :1 -screen 0 1024x768x24 -fbdir /var/tmp/
(EE) AIGLX error: dlopen of /usr/X11/lib/dri/swrast_dri.so failed (dlopen(/usr/X11/lib/dri/swrast_dri.so, 5): image not found)
(EE) GLX: could not load software renderer
(EE) XKB: Couldn't open rules file /usr/X11/share/X11/xkb/rules/base
(EE) XKB: No components provided for device Virtual core keyboard
$ DISPLAY=:1 vncviewer -FullColor --Passwordfile mypasswordfile localhost
$ import -window root -display :1 screenshot-vnc.png

Recording a video:

The only way (I found) is to capture the RDP session as a video is to use ffmpeg as explained in the X-session, with rdesktop displayed in Xvfb session.

Virtual Server helpers:

Capturing screenshots with Vmware Esx

The Vsphere API allows the creation of a task by using CreateScreenshot_Task() command. I used the VMware VI (vSphere) Java API - http://vijava.sourceforge.net/ and coded an example to create the screenshot. The file itself is saved by vmware in the datastore root, so you need something like scp to grab it from there.

import java.net.URL;
import com.vmware.vim25.*;
import com.vmware.vim25.mo.*;

public class takescreenshot 
{
    public static void main(String[] args) throws Exception
    {
        String url="https://localhost/sdk"; 
        String username="root"; 
        String password="thepassword"
        long start = System.currentTimeMillis();
        ServiceInstance si = new ServiceInstance(new URL(url), username, password, true);
        long end = System.currentTimeMillis();
        System.out.println("time taken:" + (end-start));
        Folder rootFolder = si.getRootFolder();
        String name = rootFolder.getName();
        System.out.println("root:" + name);
        ManagedEntity[] mes = new InventoryNavigator(rootFolder).searchManagedEntities("VirtualMachine");
        if(mes==null || mes.length ==0)
        {
            return;
        }
        
        VirtualMachine vm = (VirtualMachine) mes[0]; 
        Task screentask=vm.CreateScreenshot_Task();
        si.getServerConnection().logout();
    }
}

For an example using the perl API have a look at the script at http://communities.vmware.com/docs/DOC-10497

Screencapture and recording a video in Virtualbox

I found that virtualbox itself has a way to capture the screen (--capture) and to record the session to file. Internally it uses ffmpeg. I wasn't able to run it on my mac, as it is not supported on that platform, but it might work for you.

$ VBoxVRDP --help
Oracle VM VirtualBox Headless Interface 3.2.8
(C) 2008-2010 Oracle Corporation
All rights reserved.

Usage:
   -s, -startvm, --startvm <name|uuid>   Start given VM (required argument)
   -v, -vrdp, --vrdp on|off|config       Enable (default) or disable the VRDP
                                         server or don't change the setting
   -p, -vrdpport, --vrdpport <ports>     Comma-separated list of ports the VRDP
                                         server can bind to. Use a dash between
                                         two port numbers to specify a range
   -a, -vrdpaddress, --vrdpaddress <ip>  Interface IP the VRDP will bind to 
   -c, -capture, --capture               Record the VM screen output to a file
   -w, --width                           Frame width when recording
   -h, --height                          Frame height when recording
   -r, --bitrate                         Recording bit rate when recording
   -f, --filename                        File name when recording.  The codec
                                         used will be chosen based on the
                                         file extension

Sending Keystrokes to Your (Virtual) Machines using X, Vnc , Rdp or Native ways

(2010-08-29) - Comments

Options overview

The most common way to interact with a virtual machine is by remote login via ssh. This blogpost is about a different way of interaction: it will show you how to send keystrokes (or mouse) directly to the remote screen of the machine. This can be used for instance for kickstarting a machine before the network is up (typing linux), or automating things that require screen interaction.

In general three types of remote screen sessions exist:

  • VNC: is a graphical desktop sharing system that uses the RFB protocol
  • RDP: A proprietary developed by Microsoft
  • X-session: based on the X Window System (commonly X or X11)

Most virtualization solutions allow you to activate one of these remote session options:

Keycodes versus keys

Before we start of with the different solutions, I would like to point out that sending keystrokes to these remote sessions is not the same as printing a character to the screen.

Keyboards and for that matter virtual keyboards interact with scancodes : every key on the keyboard is assigned a scancode. This has a few consequences:

  • An uppercase letter is a combination of a scancode for Shift and a scancode for the letter
  • The same scancode on a different keyboard layout can cause a different letter to be send: f.i. scancode 10 (hex) generate a 'Q' on a Qwerty keyboard but 'A' on an Azerty keyboard

Most of the tools assume the use of a US Layout keyboard. A good overview of the different scancodes can be found at http://www.win.tue.nl/~aeb/linux/kbd/scancodes-1.html

Interacting with X-Windows

Before VNC and RDP existed , people already had the option of using an X-client to an X-server.

P.S. I know you are probably looking for RDP or VNC interaction, but this description will come in handy as we will use this through Xvfb.

I found two tools that help sending keystrokes:

I couldn't get xte to compile on my mac but xdotool is conveniently available in macports.

What is xdotool? This tool lets you simulate keyboard input and mouse activity, move and resize windows, etc. It does this using X11's XTEST extension and other Xlib functions. Additionally, you can search for windows and move, resize, hide, and modify window properties like the title. If your window manager supports it, you can use xdotool to switch desktops, move windows between desktops, and change the number of desktops.

$ sudo port install xdotool
--->  Computing dependencies for xdotool
--->  Fetching xdotool
--->  Verifying checksum(s) for xdotool
--->  Extracting xdotool
--->  Applying patches to xdotool
--->  Configuring xdotool
--->  Building xdotool
--->  Staging xdotool into destroot
--->  Installing xdotool @2.20100818.3004_0
--->  Activating xdotool @2.20100818.3004_0

To use xdotool (and avoid the error message "Error: XTEST extension unavailable
on '(null)'") you need to enable the XTEST extension. If you're using Apple's
X11.app, the command to do so is: 

defaults write org.x.X11 enable_test_extensions -boolean true 

If you're using the MacPorts X11.app, use: 

defaults write org.macports.X11 enable_test_extensions -boolean true 

This only needs to be done once.
--->  Cleaning xdotool

Sending the key 'a' to your X-sessions

$ xdotool key 'a'

The tool has a lot of ways of interacting with the windows, and for our purposes it has a option to send a key and even mouse .

$ xdotool
Usage: xdotool <cmd> <args>
Available commands:
  getactivewindow
  getwindowfocus
  getwindowname
  getwindowpid
  search
  selectwindow
  help
  version
  click
  getmouselocation
  key
  keydown
  keyup
  mousedown
  mousemove
  mousemove_relative
  mouseup
  type
  windowactivate
  windowfocus
  windowmap
  windowmove
  windowraise
  windowsize
  windowunmap
  windowreparent
  windowkill
  set_window
  behave
  set_num_desktops
  get_num_desktops
  set_desktop
  get_desktop
  set_desktop_for_window
  get_desktop_for_window
  get_desktop_viewport
  set_desktop_viewport

Interacting with VNC

Using Ruby-vnc

The easiest way I found to interact with a VNC session is by using the excellent ruby-vnc library http://code.google.com/p/ruby-vnc/

The example on the website gives you a good idea on how it works:

# launch xclock on localhost. note that there is an xterm in the top-left
Net::VNC.open 'localhost:0', :shared => true, :password => 'mypass' do |vnc|
  vnc.pointer_move 10, 10
  vnc.type 'xclock'
  vnc.key_press :return
end

Using vncviewer and xvfb

If you are not that confident with ruby, there is another option. Xvfb allows you to create a virtual screen on which you can interact. Together with the vncviewer (X-client) we can use it with xdotool to interact with a vnc session.

The first step is to create a vnc password file, as with vncviewer the password can not be supplied on the commandline.

Create password file name 'mypasswordfile'

$ vncpasswd mypasswordfile
Password:
Verify:

Start Xfvb screen (:1 means another display and 24 is the colordepth)

$ Xvfb  :1 -screen 0 1024x768x24 -fbdir /var/tmp/
(EE) AIGLX error: dlopen of /usr/X11/lib/dri/swrast_dri.so failed (dlopen(/usr/X11/lib/dri/swrast_dri.so, 5): image not found)
(EE) GLX: could not load software renderer
(EE) XKB: Couldn't open rules file /usr/X11/share/X11/xkb/rules/base
(EE) XKB: No components provided for device Virtual core keyboard

Now we can start vncviewer (that logs in automatically) within the Virtual Frame buffer

$ DISPLAY=:1 vncviewer -FullColor --Passwordfile mypasswordfile localhost

Now we are back to using xdotool to interact with the session.

Interacting with RDP

Using properjavardp

Properjava rdp - http://properjavardp.sourceforge.net/ is a full implementation of the RDP protocol in Java. It doesn't seem to be maintained and several 'forks' are available.

I used the original source to get it running:

  • Import the src directory in a java project
  • Add the src files like RdpPacket_Localised for the src-1.4 directory
  • Add the jar files to the project (log4j.jar, ..)

The following code will give you an idea on how you can use it

package net.propero.rdp;

import java.net.InetAddress;
import net.propero.rdp.rdp5.Rdp5;
import net.propero.rdp.rdp5.VChannels;

public class SendKeysRdp {
    public static void main(String[] args) {
        int logonflags = Rdp.RDP_LOGON_NORMAL;
        Rdp5 RdpLayer = null;
    
        VChannels channels = new VChannels();
        RdpLayer = new Rdp5(channels);
        Common.rdp = RdpLayer;      
        try {
/*
 *                  RdpLayer.connect(Options.username, InetAddress
                            .getByName(server), logonflags, Options.domain,
                            Options.password, Options.command,
                            Options.directory);         
 */
            RdpLayer.connect("Username", InetAddress.getByName("192.168.2.30") , logonflags, "Domain","Command", "","Directory");
            RdpLayer.sendInput(Input.getTime(), Input.RDP_INPUT_SCANCODE, Input.RDP_KEYPRESS, 0x1f, 0);

        } catch(Exception e){
            System.out.println(e.toString());
        }
    }
}

Using rdesktop and Xvfb

$ sudo port install rdesktop
$ Xvfb  :1 -screen 0 1024x768x24 -fbdir /var/tmp/
(EE) AIGLX error: dlopen of /usr/X11/lib/dri/swrast_dri.so failed (dlopen(/usr/X11/lib/dri/swrast_dri.so, 5): image not found)
(EE) GLX: could not load software renderer
(EE) XKB: Couldn't open rules file /usr/X11/share/X11/xkb/rules/base
(EE) XKB: No components provided for device Virtual core keyboard

$ DISPLAY=:1 rdesktop -u <username> -p <password> -d <domain> remotehost

Directly from C code re-using code from rdesktop

I found a link to an xrdp overflow tool that seems to use the rdp_send_scancode function to send the keystroke.

http://webcache.googleusercontent.com/search?q=cache:GIucABEgkT8J:packetstormsecurity.org/0904-exploits/xrdp-overflow.txt+rdp_send_scancode&cd=7&hl=nl&ct=clnk&gl=be&client=firefox-a

Another cool trick I picked up, was the way to launch commands using rdesktop :

http://www.singularity.be/2008/03/using-rdesktop-to-script-windows.html

Virtualbox option

Virtualbox provides a way to send keyboard scancodes directly using it's excellent API. The option keyboardputscancode allows to specify the hex code of the scancode. You can also send multiple keycodes after each other. In my experience this doesn't work well for a long sequence. It seems the buffer it limited and your best option is to send the different keycodes in multiple calls to the command line.

VBoxManage controlvm        <uuid>|<name>
                            pause|resume|reset|poweroff|savestate|
                            acpipowerbutton|acpisleepbutton|
                            keyboardputscancode <hex> [<hex> ...]|

Using a non-headless solution

In order to automate visual tasks, there exist a lot of macro/recording tools, the downside of them is that they need an actual display to run. When scripting you are mostly looking for headless solutions though. The disadvantage of headless solutions is that they are cumbersome to create as you have to move the pointer and the keys at the exact spot.

Sikuli

When you are interacting with a session you as a person do a lot more then just typing things. You yourself also locate which window to focus, what button to click.

Sikuli is a great tool that tries to help you in those tasks. It uses image recognition tools to find the correct place on the screen. Think of it a visual scripting language for screen interaction. An example script looks like this:

It allows you use these commands within java and it seems that someone is working on replacing the java.awt.robot to a vnc version

More background can be found at:

The function type() simulates keyboard typing just as a user types text in a application. However, type() doesn't work for different keyboard layouts other than QWERTY, such as DVORAK. We provide a workaround paste() since 0.9.7 (20100127). The function paste() transfers text through system's clipboard, which is fully independent of keyboard layouts. A sample usage that paste "network" into a search box is shown as follows.

Other options

For completeness I provide different options I found useful

$ osascript -e 'tell application "System Events" to keystroke "LOGIN_NAME"'; \
$ osascript -e 'tell application "System Events" to keystroke tab'; \
$ osascript -e 'tell application "System Events" to delay 0.5'; \
$ osascript -e 'tell application "System Events" to keystroke "PASSWORDHERE"'; \
$ osascript -e 'tell application "System Events" to delay 0.5'; \
$ osascript -e 'tell application "System Events" to keystroke return'

Replace LOGIN_NAME and PASSWORD with the proper values
Run as "root" user

ssh tricks - the usual and beyond

(2010-08-27) - Comments
SSH is an amazing beast. I nearly use it everyday and I'm amazed every time I learn something new. The following is a list of my tricks in the bag. It starts with the usual tricks that you find all over the place, but I hope there will be some new tricks for you too.

What's your best trick? Share it in the comments with the world. Nobody can know enough of ssh!

The basics:

Password-less login:

This is usually the first thing start doing when want automation with ssh

#Create a new keypair
$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/Users/patrick/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/patrick/.ssh/id_dsa.
Your public key has been saved in /Users/patrick/.ssh/id_dsa.pub.
The key fingerprint is:
87:66:b7:a0:f6:0e:6a:71:2c:5d:ee:5f:17:2a:b7:2f patrick@localhost
The key's randomart image is:
+--[ DSA 1024]----+
|                 |
|                 |
|                 |
|        ..       |
|     o oS o   .  |
|    o ++.+ . . . |
|     ++.  o + .  |
|    .o o.  +Eo   |
|   ..  .o.. .o.  |
+-----------------+
$ cat ~/.ssh/id_dsa.pub | ssh user@remotehost "cat - >> ~/.ssh/authorized_keys"
$ ssh user@remotehost

Install your keys on a remote server:

$ ssh-copy-id -i ~/.ssh/id_dsa.pub user@remotehost
#Alternative 
$ cat ~/.ssh/id_dsa.pub | ssh user@remotehost "cat - >> ~/.ssh/authorized_keys"

Passphrase automation:

If you have protected your keys with a passphrase (which you should), then it is annoying to re-enter that all the time. You can avoid that by running your environment inside an ssh-agent and using ssh-add to enter the passphrase once.

$ ssh-add ~/.ssh/id_dsa
Need passphrase for /home/mah/.ssh/id_dsa (you@example.com).
Enter passphrase:
$

Pseudo Terminal :

some commands like sudo require a pseudo terminal to be activated

$ ssh -t patrick@remotehost sudo cat /etc/passwd

Avoid lastlog:

Log in without appearing in lastlog/w and who output.

$ ssh -T user@hostname.com

Piping

Example of using piping to backup over the network

$ ufsdump 0uf - /dev/md/rdsk/d33 | ssh r280n "dd obs=32k ibs=32k of=/dev/rmt/0n"

Rsync over ssh

$ rsync -avz -e "ssh -i /home/thisuser/cron/thishost-rsync-key" remoteuser@remotehost:/remote/dir /this/dir/

Tunnels and firewall-piercings:

X-forwarding:

$ ssh -X patrick@remotehost
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Warning: No xauth data; using fake authentication data for X11 forwarding.
Last login: Fri Aug 27 20:27:40 2010

Port forwarding:

Set up a localforward from the remote machine port 25 to a local port 9025

$ ssh -L 9025:localhost:25 patrick@remotehost

No command:

Sometimes you just want to setup a forward with having a shell

$ ssh -N -L 9025:localhost:25 patrick@remotehost

KeepAlive:

Getting tired of those timeouts by the firewall? Have ssh send a keepalive/

Put the following options in your $HOME/.ssh/ssh_config

    KeepAlive yes
    ServerAliveInterval 60

Socks Daemon for proxying: (-D)

Sometimes it's interesting to start a socks daemon. You can configure this in your browser to surf as it seems to come from the remote machine.

$ ssh -D 9999 patrick@remotehost

Tunneling over an http proxy:

Corporate firewalls often only allow http to go outside. See corkscrew

ProxyCommand /usr/bin/corkscrew proxy-ip 8080 %h %p ~/.ssh/myauth

Chaining ssh hopping:

Host pc1.example.org pc2.example.org
ForwardAgent yes
ProxyCommand ssh -qax bastion.example.org /usr/bin/nc -w 120 %h %p

Netcat mode:

Starting from openssh 5.4: we can have ssh act as netcat. (-W) This connects stdio on the client to a single port forward on the server. This allows, for example, using ssh as a ProxyCommand to route connections via intermediate servers.”

$ ssh -p 443 -W remotehost2:23 patrick@remotehost
Trying remotehost2...
Connected to remotehost2.
Escape character is '^]'.

User Name : ^]
telnet> close
$

Mounting over ssh:

Sometimes it's nice to mount a remote directory over ssh. Fuse and sshfs are your friend

$ sshfs remote-user@remote.server:/remote/directory /mnt/remote-fs/

http://fuse.sourceforge.net/sshfs.html

VPN Tunneling:

Did you know that ssh can do layer 2 and 3 VPN tunneling?

Check out ssh -w. Example from manpage:

$ ssh -f -w 0:1 192.168.1.15 true
$ ifconfig tun0 10.0.50.1 10.0.99.1 netmask 255.255.255.252

SSH http multiplexer:

sslh lets one accept both HTTPS and SSH connections on the same port. It makes it possible to connect to an SSH server on port 443 (e.g. from inside a corporate firewall) while still serving HTTPS on that port. http://www.rutschle.net/tech/sslh.shtml

Speed

Compression

If you are working on a slow link, compression (-C) and using a simple cipher (-c blowfish) saves you speed

$ ssh -C -c blowfish patrick@remotehost

Multiplexing - ControlMaster:

Another great way to speed up ssh is to re-use the same connection when you connect multiple times to the same host

$ mkdir –p ~/.ssh/connections
$ chmod 700 ~/.ssh/connections

Add this to your ~/.ssh/config file:
Host *
ControlMaster auto
ControlPath ~/.ssh/connections/%r_%h_%p

Managing keys

Ignore Hostkeys:

When you're re-installing a machine over and over again, you often want to get rid of the hostfile key verification. This is what you need:

$ ssh user@host -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null

Check if hostkey exists:

k$ ssh-keygen -F 192.168.2.152
# Host 192.168.2.152 found: line 31 type RSA
192.168.2.152 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwHH15HpeJo21wyqpe2iFM8/0CtoYnE9DDXfCewws7iMhM+vgp7pjnaC83IgAt7G/x/VDHcbnyuI4odrGSEAE5wm7LNuT6uSfQMbXCayE+uoOIrAVhf41ZnAFQrs/+Mutk5LFEjPPNhuriq5ltBT4UwMlYQMa5z/SzmxV0ZAGXks5GMDz0o89yUwRarRfsGudASEtzUxgnxnOo5STBMZOdQ0GNEVdfJDgfJDAOi34T1FidpCqAtm8akYuB+Qsj3/hDQmIT+GsKYaGNZvz8ZNnPBAc9kWlS6VqXXNreyEeu7AmHDWXjMP3NW1tsibmZ8zeOSZdmEVEiuaYCIvERDq3MQ==

Remove a hostkey:

$ ssh-keygen -R 192.168.2.152
/Users/patrick/.ssh/known_hosts updated.
Original contents retained as /Users/patrick/.ssh/known_hosts.old

Get hostkey of remote server:

$ ssh-keyscan remotehost
# remotehost SSH-2.0-OpenSSH_5.2
remotehost ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyREFGMBB6Qi1uoEYIk4GlqLXdS26moAxmV69UX0icQjp0Rw53xZ/2L0ZQwhsUiFV1vq4QfZNeUO142IzBgSspgsJZ7wJq213tsE7WIJGIBqvWnhU3vJuL9wgYT8f6BAvLoEfapFhLy24TDmn2DXldJAYgo8MnUbRrJlvnhQZPpd5cDWCXkzPGQE8r7REZsAWbWNlVOFRvZioPoGCGYMtsDWSBelBISGkedoNpTSpRkMmBAnsHBfvIzDPoTDYL4PZR0jJ8MaJrDhRtD4caRw4HVyhzSa3/FCpcm09PyBRabH/CyxNSOZjLc2+N9Ph9AKeTNgvmxP70wx668XaGYwCrQ==

SSH DNS Keys

Instead of using your local hostfile, you can store your keys in DNS. Have a look at sshfp to do the job. Then you can specify that ssh needs to

$ ssh localhost -o "VerifyHostKeyDNS=yes"
yes authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 2d:d3:29:bd:4d:e2:7d:a3:b0:15:96:26:d4:60:13:34.
Matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?

SSH Escape Sequences:

It often happens to me that I'm working into an ssh shell that used forwarding. I always thought there was no way to change the forwarding rules and that I had to logout. It seems not! SSh has an internal shell activated by a tilde. Seeing is believing!

Escape sequences are only recognized after a newline and are initiated with a tilde (~) unless you modify it with the -e flag.

Hit ENTER ~? on a running ssh session to see a list of escapes:

Supported escape sequences:

~. – terminate connection
~B – send a BREAK to the remote system
~C – open a command line
~R – Request rekey (SSH protocol 2 only)
~^Z – suspend ssh
~# – list forwarded connections
~& – background ssh (when waiting for connections to terminate)
~? – this message
~~ – send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)
~. and ~# are particularly useful.

Visualize hostkeys:

Every host key has it's own visual fingerprint

$ ssh -o VisualHostKey=yes patrick@localhost
Host key fingerprint is 9f:a0:03:c1:63:8b:b8:c6:d6:83:cb:22:33:cb:83:cc
+--[ RSA 2048]----+
|                 |
|   .             |
|    =            |
| . o +           |
|. . o   S        |
|..o  . . o .     |
|== o  o   o      |
|@E. .  .         |
|+B.              |
+-----------------+

Security hacks

Local Password sniffing:

If you have process that connects to your ssh and you want to see the password it's using, then strace is your friend.

$ ps axuww | egrep 'PID|ssh'
#Now become root and attach to the running daemon with strace, changing the PID as appropriate:

$ sudo strace -f -e 'read,write' -p12345

Remote Password sniffing:

A more passive way of listening into ssh sessions (v1) is using dsniff - Dsniff

Fingerprint fuzzing:

This one is to lure a lazy administrator into accepting your certificate. It generates keys with an almost similar fingerprint. http://freeworld.thc.org/papers/ffp.html

SSH Honeypot:

And to go totally security. Launch your own ssh honeypot and capture all the remote commands (and typos) with Kippo

Need more?

Top 50 SSH Helper tools - OMG!

Share yours! I'm definitely interested

Webby Rss feed and Syntax Highlighting

(2010-08-26) - Comments

When I first started blogging I used Wordpress, while it was nice at that time, it felt rather cumbersome to use for posting technical stuff like code samples. Also having everything in the database make it difficult to use my favorite text editor to manage my content. So I decided to move to Webby : it allows you to use the power of ruby erb inside you html pages without the overkill of running a rails Another advantage is that it creates static html so it's both fast and secure.

I've been running it now for more then a year and have been happy ever since. This blogpost shares some of my scripts/enhancements I've used:

  • Wordpress exporter to Webby : this was my first challenge, how do I get everything from my wordpress out of that database. I wrote a little script that reads from the RSS feed of the wordpress and export everything into webby structure . It's not the most robust script (it was early stages ruby for me at that time), but you'll get the idea.

  • RSS feed in Webby: I didn't find any good rss feed code, most of it didn't pass the rss validation. So here you can find my take on it. As a bonus the script tries to make your images absolute, which is important for having your images displayed in some rss feedreaders.

  • Highlighting with Coderay : webby includes a way to highlight parts of the code. This works very well, but I found myself missing a way to include a file instead of pasting the code directly into the blogpost. This has many advantages: I can edit the file separate and the correct syntax highlighting in my favorite text editor still works. And the escaping works better for ruby code as there can not be a problem with escaping special characters. After a while I switched to Ultraviolet because it has way more syntaxes supported. The downside is that it requires the onigurama library . This isn't really a problem on Mac using Macports.

patricks-iMac:jedi-webby patrick$ sudo port install oniguruma4
patricks-iMac:jedi-webby patrick$ gem install ultraviolet -- -I/opt/local/lib
Building native extensions.  This could take a while...
ERROR:  Error installing ultraviolet:
    ERROR: Failed to build gem native extension.

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb -I/opt/local/lib
checking for main() in -lonig... no
creating Makefile

make
gcc -I. -I. -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin10.0 -I. -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE   -fno-common -Wall  -c oregexp.c
oregexp.c:2:23: error: oniguruma.h: No such file or directory

#Solution
patricks-iMac:jedi-webby patrick$ gem install ultraviolet -- --with-opt-dir=/opt/local

This is all for now.

Codify helper (with Ultraviolet): put inside your webby-root/lib

# require uv
if try_require 'uv'
require 'enumerator'

Loquacious.configuration_for(:webby) {
  desc <<-__
    Options for CodeRay syntax highlighting. See the CodeRay home page
    (http://coderay.rubychan.de/) for more information about the available
    options.
  __
  codify {
    desc 'The language being highlighted (given as a symbol).'
    lang :ruby

    desc 'The file you want to read instead of a string'
    lang :file

    desc 'Include line numbers in :table, :inline, :list or nil (no line numbers).'
    line_numbers nil

    desc 'Where to start line number counting.'
    line_number_start 1

    desc 'Make every N-th number appear bold.'
    bold_every 10

    desc 'Tabs will be converted into this number of space characters.'
    tab_width 8
  }
}

module Webby::Helpers
module CodifyHelper

  # The +codify+ method applies syntax highlighting to source code embedded
  # in a webpage. The CodeRay highlighting engine is used for the HTML
  # markup of the source code. The page sections to be highlighted are given
  # as blocks of text to the +coderay+ method.
  #
  # Options can be passed to the CodeRay engine via attributes in the
  # +codify+ method.
  #
  #    <% codify( :lang => "ruby", :line_numbers => "inline" ) do -%>
  #    # Initializer for the class.
  #    def initialize( string )
  #      @str = stirng
  #    end
  #    <% end -%>
  #    
  # The supported Codify options are the following:
  #
  #    :lang               : the language to highlight (ruby, c, html, ...)
  #    :file               : the file to highlight
  #    :line_numbers       : include line numbers in 'table', 'inline',
  #                          or 'list'
  #    :line_number_start  : where to start with line number counting
  #    :bold_every         : make every n-th number appear bold
  #    :tab_width          : convert tab characters to n spaces
  #
  
  def codify( *args, &block )
    opts = args.last.instance_of?(Hash) ? args.pop : {}

    parent=File.dirname(@page.path)
     file=opts[:file]
    text=""
    if (!file.nil?) 
      begin
        filename=parent+File::SEPARATOR+file
        file=File.open(filename,"rb")
        text=file.read
        syntax=guess_syntax(filename)
        if !syntax.nil? && opts[:lang].nil? then
          opts[:lang]=syntax
       end
      rescue
        puts "Error reading code file"+filename
      end
    else
      text = capture_erb(&block)
    end
    
    return if text.empty?

   defaults = { :lang => "ruby", :line_numbers => false , :theme => "mac_classic"}
   lang = opts.getopt(:lang, defaults[:lang]).to_s
   line_numbers = opts.getopt(:line_numbers, defaults[:line_numbers])
   theme = opts.getopt(:theme, defaults[:theme])

    lang=case lang
      when "shell","sh" then "shell-unix-generic"
      when "text" then :plain_text
      else
        lang
      end
    out = %Q{<div class="UltraViolet">\n}
    out << Uv.parse(text, "xhtml", lang.to_s, false,"pastels_on_dark")
    out << %Q{\n</div>}

    # put some guards around the output (specifically for textile)
    out = _guard(out)

    concat_erb(out, block.binding)
    return
  end

  def guess_syntax(filename)
    filetype=Uv.syntax_for_file(filename)[0][0]
    return filetype

  end

end  # module CodefyHelper

register(CodifyHelper)

end  # module Webby::Helpers
end  # try_require

# EOF

Codify helper (with Coderay): put inside your webby-root/lib

if try_require 'coderay'
require 'enumerator'

Loquacious.configuration_for(:webby) {
  desc <<-__
    Options for CodeRay syntax highlighting. See the CodeRay home page
    (http://coderay.rubychan.de/) for more information about the available
    options.
  __
  codify {
    desc 'The language being highlighted (given as a symbol).'
    lang :ruby

    desc 'The file you want to read instead of a string'
    lang :file

    desc 'Include line numbers in :table, :inline, :list or nil (no line numbers).'
    line_numbers nil

    desc 'Where to start line number counting.'
    line_number_start 1

    desc 'Make every N-th number appear bold.'
    bold_every 10

    desc 'Tabs will be converted into this number of space characters.'
    tab_width 8
  }
}

module Webby::Helpers
module CodifyHelper

  # The +codify+ method applies syntax highlighting to source code embedded
  # in a webpage. The CodeRay highlighting engine is used for the HTML
  # markup of the source code. The page sections to be highlighted are given
  # as blocks of text to the +coderay+ method.
  #
  # Options can be passed to the CodeRay engine via attributes in the
  # +codify+ method.
  #
  #    <% codify( :lang => "ruby", :line_numbers => "inline" ) do -%>
  #    # Initializer for the class.
  #    def initialize( string )
  #      @str = stirng
  #    end
  #    <% end -%>
  #    
  # The supported Codify options are the following:
  #
  #    :lang               : the language to highlight (ruby, c, html, ...)
  #    :file               : the file to highlight
  #    :line_numbers       : include line numbers in 'table', 'inline',
  #                          or 'list'
  #    :line_number_start  : where to start with line number counting
  #    :bold_every         : make every n-th number appear bold
  #    :tab_width          : convert tab characters to n spaces
  #
  
  def codify( *args, &block )
    opts = args.last.instance_of?(Hash) ? args.pop : {}

    parent=File.dirname(@page.path)
     file=opts[:file]
    text=""
    if (!file.nil?) 
      begin
        filename=parent+File::SEPARATOR+file
        file=File.open(filename,"rb")
        text=file.read
        syntax=guess_syntax(filename)
        if !syntax.nil? && opts[:lang].nil? then
          opts[:lang]=syntax
       end
      rescue
        puts "Error reading code file"+filename
      end
    else
      text = capture_erb(&block)
    end
    
    return if text.empty?

    defaults = ::Webby.site.coderay
    lang = opts.getopt(:lang, defaults.lang).to_sym

    cr_opts = {}
    %w(line_numbers       to_sym
       line_number_start  to_i
       bold_every         to_i
       tab_width          to_i).each_slice(2) do |key,convert|
      key = key.to_sym
      val = opts.getopt(key, defaults[key])
      next if val.nil?
      cr_opts[key] = val.send(convert)
    end

    #cr.swap(CodeRay.scan(text, lang).html(opts).div)
    out = %Q{<div class="CodeRay">\n<pre>}
    out << ::CodeRay.scan(text, lang).html(cr_opts)
    out << %Q{</pre>\n</div>}

    # put some guards around the output (specifically for textile)
    out = _guard(out)

    concat_erb(out, block.binding)
    return
  end

  def guess_syntax(filename)
    extension=case File.extname(filename).downcase
      when ".rb" then :ruby
      when ".html",".html" then :html
      when ".sh" then :sh
      when ".css" then :css
      when ".js" then :java_script
      when ".diff" then :diff
      when ".yaml" then :yaml
      when ".json" then :json
      when ".java" then :java
      when ".xml" then :xml
      when ".txt" then :plaintext
      when ".py" then :python
      when ".xml" then :xml
      when ".c" then :c
      when ".sql" then :sql
      else nil
    end
    return extension

  end

end  # module CodefyHelper

register(CodifyHelper)

end  # module Webby::Helpers
end  # try_require

# EOF

Rss feed file

---
title:     JEDI - Just Enough Developed Infrastructure
subtitle:  www.jedi.be
description: development, infrastructure and other stuff
site:      www.jedi.be
author:    Patrick Debois
email:     Patrick.Debois@jedi.be
extension: xml
layout:    nil
dirty:     true
filter:    
- erb
# <?xml-stylesheet type="text/css" media="screen" href="http://www.jedi.be/css/blueprint/screen.css"?>
---
<%- require 'rexml/document' -%>
<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/">
    <channel>
        <atom:link href="http://www.jedi.be/blog/feed/" rel="self" type="application/rss+xml" />
        <title><%= h(@page.title) %></title>
        <description><%= h(@page.description) %> </description>
        <link>http://<%= @page.site %>/blog</link>
        <pubDate><%= Time.now.gmtime.rfc822 %></pubDate>
        <generator>webby rss script</generator>
        <lastBuildDate><%= Time.now.gmtime.rfc822 %></lastBuildDate>
        <managingEditor><%= @page.email %> (<%= @page.author %>)</managingEditor>
        <webMaster><%= @page.email %> (<%= @page.author %>)</webMaster>
        <language>en</language>
     <%- @pages.find( :limit => 10,
       :in_directory => 'blog',
       :recursive => true,
       :sort_by => 'created_at',
       :reverse => true).each do |article|

       next if article.blog_post.nil?
   -%>
      <item>
            <title><%= h(article.title) %></title>
            <link>http://<%= @page.site %><%= article.url %></link>
            <% if article.guid.nil? %>
          <guid isPermaLink="false">http://<%= @page.site %><%= article.url%></guid>
            <% else %>
          <guid isPermaLink="false"><%= article.guid %></guid>
            <% end %>
          <pubDate><%= article.created_at.gmtime.rfc822 %></pubDate>
            <comments>http://<%= @page.site %><%= article.url %>#comments</comments>
            <description><%= normaltext=render(article); 
            normaltext.gsub!('<img src="/', '<img src="http://'+@page.site+'/')
            normaltext.gsub!("<img src='/", "<img src='http://"+@page.site+'/')

             h(normaltext) %></description>
        
            <%- if !article.keywords.nil? %>
              <%- article.keywords.each do |keyword| %>
                  <category><![CDATA[<%= keyword %>]]></category>
                <%- end %>
         <% else %>
                  <% if !article.tags.nil? %>
                  <%- article.tags.each do |tag| %>
                          <category><![CDATA[<%= tag %>]]></category>
                    <% end %>
              <% end %>
           <% end %>
          <content:encoded><![CDATA[<%= REXML::CData.new(render(article)) %>]]></content:encoded>
        </item>
    <% end %>
  </channel>
</rss>

Wordpress to webby conversion script

#!/usr/bin/env ruby
require 'rubygems'
require 'rfeedparser'
require 'pp'
require 'net/http'
require 'uri'

def write_feed(feed)
  pp feed.title
  pp feed.subtitle
  pp feed.links[0].href

  @site_url=feed.wp_base_blog_url
  #feed.wp_base_site_url
  #feed.updated
  #feed.generator
  #feed.updated_time
  #feed.updated_parsed
  #feed.wp_tag_name
  #feed.wp_tag_slug
  #feed.wp_cat_name
  #feed.wp_category_parent
  #feed.wp_category
  #feed.wp_wxr_version
end

def write_entry(e)
  #File.open(local_filename, 'w') {|f| f.write(doc) }
  
  base=e.link.gsub(@site_url+'/','').gsub(e.wp_post_name+'/', '')
  path=@webby_dir+"/"+base+'/'+e.wp_post_name
  FileUtils.mkdir_p path
  filename="index.html"
  @counter=@counter+1
  
  #write entries
  puts "[#{@counter}]creating "+path+"/"+filename
  File.open(path+"/"+filename, 'w') { |f| 
    f.write("---\n") 
    f.write("title: "+"\""+e.title+"\""+"\n")
    f.write("author: "+e.author+"\n")
    f.write("guid: "+e.guid+"\n")
    format="%Y-%m-%d %H:%M:%S.0 +00:00"
    #created_at: 2009-09-10 22:16:41.382708 +02:00
    f.write("created_at: "+ Time.parse(e.wp_post_date).strftime(format)+"\n")
    f.write("blog_post: "+ "true\n")
    f.write("filter:\n")
    f.write("  - erb\n")
    f.write("  - basepath\n")
#    f.write("  - tidy\n")

    tags=Hash.new
    categories=Hash.new
    rssterms=e.tags
    rssterms.each { |t|
      scheme=t['scheme']
      term=t['term']
      if scheme=='tag'
        tags[term]=''   
      end
      if scheme=='category'
        categories[term]=''   
      end
    }
    f.write("categories:\n")
    categories.keys.each { |c| 
      f.write("   - #{c}\n")
    }
    f.write("tags:\n")
    tags.keys.each { |c| 
      f.write("   - #{c}\n")
    }

    #f.write("  - maruku\n")
    f.write("---\n")
    #fixing empty lines to be a hard break
    f.write(e.content[0].value.gsub(/\n\n/,"\n<br>\n")) 
    
    #fixing URL's for the images and relative paths
  }
  #fixing directory urls
  
  #browsing subdirectories index.html?


  puts "---------------------------"
  
  
  return
  
  puts e.link

  puts counter.to_s+":"
  #e.wp_post_date
  pp e.wp_post_type
  pp e.title
  #e.summary_detail[0]
  #e.post_meta
  #e.wp_meta_key
  pp e.wp_post_name
  #e.wp_post_date_gmt
  pp e.author
  pp e.wp_post_id
  pp e.wp_status
  #e.wp_comment_status
  #e.summary
  #e.guidislink
  #e.title_detail[0]
  #e.wp_meta_value
  pp e.wp_post_parent
  #e.wp_ping_status
  #pp e.content[0].value
  pp e.content[0].type
  pp e.content[0].language
  pp e.links[0].href
  #e.links[0].rel
  #e.links[0].type
  pp e.link
  #e.wp_post_password
  #e.wp_menu_order
  #e.wp_post_id
  #e.excerpt_encoded
  pp e.updated
  #e.updated_time
  e.updated_parsed
  
end

def download_attachment(e)
  url=e.wp_attachment_url
  puts url
  base=@webby_dir+"/"+url.gsub(@site_url+'/','').gsub(e.wp_post_name+'/', '')
  directory=File.dirname(base)
  filename=File.basename(base)
  puts directory
  puts filename
  FileUtils.mkdir_p directory
  
  myURI = URI.parse(url)
  pp myURI
  Net::HTTP.start(myURI.host) { |http|
    resp = http.get(myURI.path)
    open(directory+'/'+filename, "wb") { |file|
      file.write(resp.body)
     }
  }
end

def write_entries(entries)
  @counter=0
  entries.each { |e|
    
    if e.wp_post_type == 'post' && e.wp_status == 'publish'

      write_entry(e)
  end

    if e.wp_post_type == 'attachment'
      #we should download it      
      download_attachment(e)
  end

  }
end

if ARGV.length < 2
  puts "Usage: wp2ruby <feed_url> <webby_dir>"
  exit
end

@feed_url=ARGV[0]
@webby_dir=ARGV[1]
@site_url=""

wpr=""
begin
  wpr=FeedParser.parse(@feed_url)
rescue
  puts "Error parsing feed"+$!
  exit
end

write_feed(wpr.feed)
write_entries(wpr.entries)

The leaning of life - History of the Silos

(2010-07-07) - Comments


'Break down the silos', that is a rallying cry that you will often hear amongst devops people: the word silo in an enterprise context usually has a bad connotation. Still they keep on existing and I figure there must be good reasons for that. Maybe if we understand the reasoning behind silos better, we stand a better chance overcoming the downsides of their existence.

As usual wikipedia is quite helpful to find the origin of the word Silo effect and explains:

  • The silo effect gets its name from the farm storage silo; each silo is designated for one specific grain, and a lack of communication causes departmental thinking to lack ideas from other departments.
  • A second slightly more academic, suggestion as to the term silo effect focuses on the gradual draining of the entire silo's grain from a remarkably small opening in the bottom. The homogeneous state of the entire volume of grain makes it highly susceptible to small changes as they occur further and further down.

Julian Simpson (aka The @builddoctor) said in his presentation Silos are for Farmers and right he is, but why do they keep existing?

My initial findings on the internet seemed to indicate two other reasons:

We discussed it further at Agile Open Benelux 2010 earlier this year but we didn't get to the root of the matter IMHO. We all have a hunch or certainty that this is a bad thing. Still we fail to explain why they are so pre-dominantly existent in big enterprises. Sure, people see managers as wanting their own empire, but why is it those people have the idea they are doing a good job?

After chatting with Pascal Van Cauwenberghe , he suggested I read Rebirth of American Industry - A study of Lean Management by William H. Waddell & Norman Bodek and now things have become much clearer to me.

The book explains on the difference in culture in manufacturing between the Sloan and the Toyoda model (aka as Lean). The sloan model was used at the successful (ahum) General Motors, and has been copied over to other management cultures. Some of it's design decisions are:

  • Management knows it best: Sloan by design has put people in different groups and put management on top of it, to make sure it works
  • It values the concept of ROI and ROS which makes the divide even further, as unfinished products as they float from division to another can be seen as assets (or is it inventory), even if they are of no use to the actual customer.
  • People are seen as a variable cost that can be gotten under control by automation. Toyota has made a point of never firing people because they see people as fixed asset. Some argue this is one of the reasons of the current problems of Toyota but nevertheless I believe in it.

Toyota/Ford or lean for that matter focus on different aspects:

  • Cycle time: the time it takes to go through the whole process. If this can be shortened it means that the process can run much smoother
  • Quality: they have a zero defect tolerance, where they understand whatever defect that will occur during manufacturing will be easier and less costly to fix as soon as possible within the process
  • Synchronization: make the different processes work together in lock step together and also work at the requested speed of the customer (pull not push)

They pressure the whole of the cycle and not each specific part of the chain. Off course, this is my quick interpretation, so if you want to know more , I urge you to read the book.

You might ask where does this lead me , I'm in IT ? Well, when hearing many of the stories and interpretation of devops, I see a lot of similarity between lean and devops subjects:

  • The enabler: just as the electric motor enabled automation within manufacturing, virtualization and in it's extend cloud has made many more things possible
  • Reducing waste: similar to the focus of the engineers in manufacturing, people are trying to improve cycle time by eliminating waste. F.i. cucumber-nagios reduces the waste of rewriting monitoring scripts when you already have testing scripts. Config Management tools like chef or puppet reduce the setup time of a machine again decreasing cycle-time.
  • Avoid Batches: by using continuous deployment in small steps, people avoid both large inventory of features and better understanding of when things fails because they do it in small steps.
  • Poka Yoke/FailProof: using Continuous Integration with test, they create a harness to make sure things don't fail. Some companies are getting so confident that they let new employees deploy their own code, because they trust their failproofing of the system.
  • Takt Time: the ability to produce at the requested speed of the customer. Sysadmins make sure that the last mile to the customers keeps on working and even though they don't develop software, they still add value to the process. Also the fact that scaling can happen
  • Andon/Stop the line: It requires the complete attention of everybody to fix it, as the cycle time is broken. Teams performing Continuous Integration already have this habit for the build status. It should always be fixed first. Similar sysadmins that have monitoring software that goes red, assign it the highest priority. To take it even further both should be focused to fix each other's problems if one of these go wrong.
  • Kanban: the equivalent has been introduced in Software development by David Anderson, to visualize the process and work out the required buffers, waste and other things to improve in your process. It is described in more detail here by Stephen Nelson-Smith
  • Measuring cycle time: a lot of companies are actively providing metrics of their # of deploys, errors they have (or tickets). While they improve things they can measure their progress. If you don't measure things you don't know what's happening.
  • Zero defect: it doesn't matter if its a functional defect or things like security and performance. Things need to be fixed as early as possible in the process
  • Listen to the people on the floor: Lean instructs management to listen to people that do the job and encourages them to watch and learn first. They are not better then anyone else and t
  • Understanding the tools you use: it might be a biased observation, but Open Source allows you better to inspect problems and potentially fix them. It's one of it's strengths of using it.
  • Single piece flow: Single piece flow is the ideal state where parts are manufactured one at a time, and flow throughout the manufacturing and supply chain as single unit, transferred as customer’s order. You could the fact that people push out single feature out all of the time as a kind of single piece flow.

It's just common sense? We'll lost of people have tried to introduce lean within their enterprise and failed (maybe not on paper). It requires a culture and not only going through the mojo: oh yes, we have a CI system and do puppet or chef, hurraah! There's more to it then just tools, you also need the culture. Agreed culture is hard to change and you might start with the behavior first Israel Gat, but only the behavior will only get you so far.

Another topic that keeps crawling up is, will sysadmins disappear? Engineers have never disappeared from factories either. There will always be someone needed to manage the machines. If your management doesn't value people they risk that you don't improve things because you fear for your job. But if you can overcome that fear, make sure you are open for other work at the same place: maybe spend more time in testing phase, or even become partly a programmer. Just like the HR policy of Toyota you can be retrained, the rest is up to you. So it is not about being a sysadmin or a devops: it doesn't matter what label you put on it, as long as you are improving things. And when you improve things make sure they improve things for the complete cycle and for the customer.

Some of these thought might still see very blurry. I hope to refine this in the future, but I wanted to get you informed on the book anyway. Any comments, feedback , corrections are more then welcome!

Velocity - Devopsdays US 2010 and beyond

(2010-07-07) - Comments

This year on 22-24 of June I attended Velocity 2010, a conference about Web Performance and Operations. And I must say, it was a blast. The talks were quite interesting, and I learned a lot of new products to check out. Most of the talks will be put online so you can enjoy them for yourselves. But what I liked most of it, was the networking and talking with like minded, interested people. Some people were clearly there for business and when you told them you do some consultancy for unknown or smaller companies and they didn't see the dollars immediately, I could see them shift, haha. Luckily there were a lot of other people too, willing to share there ideas. At velocity, the format of only talks/presentations makes that hard so I had to catch up in the hallways. Catching up with people you only from twitter and suddenly you get to talk to them face to face, makes you understand that twitter is so limited bandwidth for sharing ideas.

In a lot of talks there were hints to devops thinking, something I off course appreciated very much. Even the job board was full of people looking for 'devopsy' people. Funny to see how the term took off and everybody now has an opinion on what it is. This was pretty much like Andrew described in his post of Devops the Elephant in the room . At devopsdays US itself it became even more explicit: everyone was going on about devops means X , no according to me devops means Y.



To me, it seemed like the first get together of the 'Anonymous alcoholics': the first step is to recognize that you have problems. And that was the exact purpose we had in mind for this edition: that people started sharing their experiences with each-other. The panel experiment turned out perfect and got a lot of discussion going (including some heated ones). There were no final conclusions but there were a lot of thoughts flowing around. And that's where we should be aiming next: turning all these stories in a kind of what worked and why. And then we can start helping other people! The term devops is not yet one year old, so we still have some time AND work to do.

All of this was kindly hosted by LinkedIn and recorded by InfoQ. There was a lot of brainpower in the room including some of my new and old time heroes. Until the videos come online you can already enjoy the almost realtime transcriptions by Gene Kim co-author of the Visible Ops book.

Daniel Cukier was so kind of doing an interview with me to get my initial thoughts recorded and there is the opening video. That's it for now folks!

Introduction video of Devopsdays US 2010

Interview by Daniel Cukier

Puppetcamp 2010 Europe - a Wonderful Gathering

(2010-05-19) - Comments
Puppetcamp Europe 2010

I have been following 'Puppet' from Puppetlabs for several years now. I love the idea of 'Infrastructure as Code'. Some people say: All the cool things happen in the US.

But now the original developer and current CEO Luke Kanies and a lot of Puppet adepts are joining together for the next Puppetcamp in Europe, Belgium , Ghent on 27 and 28 of May 2010

While handling the registrations for the event, I got an insight on the experience level of the attendees, and I must say, I'm really impressed!. Some people manage over a 1000 nodes in their companies and their code-fu level are extra-ordinary. I'm sure the event is going to rock as hell.

The event is split over two days and there will be six speakers: Check the complete schedule at the puppetcamp event website.
And in the afternoon there will be an Open Space. This will create a great opportunity to talk directly to all these awesome puppeteers.		 Puppet Labs logo

If you want to part of it, you better hurry up and register. Only some places are left and time is running out.

New Book on Web Operations

(2010-05-19) - Comments

In the beginning of this year I was lucky to be asked by John Allspaw former Chief Engineer at Flickr and Jesse Robins CEO at Opscode to write a chapter on Monitoring for the upcoming book on Web Operations. When he told me the names working on the book, I felt really excited and humble among these great names of industry. I want to thank all the people helping me review my chapter. It was a great learning experience and now I know how bad my English is as a non-native speaker.

The book will be released at the Oreilly Velocityconf 2010 and whatever money it will raise will go to 826 Valencia National Foundation .

Book Cover Web Operations Pre/Order your copy at Oreilly , amazon.com or whatever bookshop you want.

How Hudson Hit Puppet With a Cucumber

(2010-05-19) - Comments

On 25 March the UKUUG organized their 'UKUUG Spring Conference'. I must say, it was a long time ago since I went up to a Unix User group conference. It was really fun to see people again so passionate about their subjects, although a lot of the talks had a rather academic edge. Being in Manchester, several of the devops crowd came down and we had a nice chat with them.

Together with Julian Simpson aka as the illustrious 'BuildDoctor'. We presented a session on how to use the tools Hudson, Puppet and Cucumber together. Initially we intended to do more technical demos but it turned out to be a devops talk, trying to attract people to the ideas around it.

Here are the slides of the presentation. Enjoy!

How Hudson hit Puppet with a Cucumber
Next