As the last Package available for Rasbian got some “small problems” regarding
ykclient return value (106): Server response signature was invalid (BAD_SERVER_SIGNATURE)
it is necessary to build it by your own untill the packages are updated.

1. Install tools

apt-get install autoconf libtool automake make libcurl4-gnutls-dev libykclient3 libusb-1.0-0-dev libpam-dev

2. Create Temp Folder for git clones

mkdir /root/temp

3. Build YubiKey C Library

cd /root/temp/apt
git clone git://github.com/Yubico/yubico-c.git
cd /root/temp/yubico-c
autoreconf --install 
./configure && make check && make install

4. Build yubico-c-client

cd /root/temp/
git clone git://github.com/Yubico/yubico-c-client.git
cd /root/temp/yubico-c-client
autoreconf --install 
./configure && make check && make install

5. Build yubikey-personalization

cd /root/temp/
git clone git://github.com/Yubico/yubikey-personalization.git
cd /root/temp/yubikey-personalization
git submodule init
git submodule update
autoreconf --install
./configure && make check && make install

6. Build yubico-pam

cd /root/temp/
git clone git://github.com/Yubico/yubico-pam.git
cd /root/temp/yubico-pam
autoreconf --install
./configure && make check && make install

7. Get the first 12 Chars

read -p "Enter OTP: " s && echo ${s:0:12}

8. Get Yubikey API Key and ID

9. Create “THE MASTER FILE”

mkdir /etc/yubikey_mappings/
vim /etc/yubikey_mappings/authorized_yubikeys

Enter the desired username with the 12char from Step 7
Example: mustermann:dwjdwakdjkaw

10. Edit /etc/pam.d/sshd to include the Yubikey Modul:

vim  /etc/pam.d/sshd

Insert the following line at the beginning. (Replace the Values from the Api ID and Key from Step 8

 auth required /usr/local/lib/security/pam_yubico.so id=nnnnn key=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk authfile=/etc/yubikey_mappings/authorized_yubikeys debug

11. Modify /etc/ssh/sshd_config to include the follwing:

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
PasswordAuthentication yes

12. Append the folling at the end of the line containing pam_unix.so in /etc/pam.d/common-auth

try_first_pass

Example:

auth    [success=2 default=ignore]      pam_unix.so nullok_secure try_first_pass

13. DEBUG DEBUG
In case you want to see something:

touch /var/run/pam-debug.log
chmod go+w /var/run/pam-debug.log

14. Restart SSH and Test

/etc/init.d/ssh restart

Login in while viewing the /var/run/pam-debug.log
You need to enter your password and before hitting enter use the yubikey to generate the otp and login.

Thanks to:
https://v00d00.co/2013/08/10/yubikey-ssh-2-factor-authentication-with-ubuntu-13-04/