jump to navigation

Using rsync remotes with git annex December 17, 2012

Posted by Rich in Uncategorized.
Tags:
add a comment

I’m a big fan of git and I really like using git annex to keep track of my large binary files (for my research, this usually means pdfs and other image files).

One shortcoming of git annex is that it doesn’t work very well on a git repository if you don’t have git annex installed. This is a non-trivial matter due to all of the Haskell dependencies if you want to compile it, but at least some pre-compiled binaries have been available of late.

However, this doesn’t help me, since my problem is that although my personal computers are up-to-date with Linux, my work computer uses a Linux distribution from about 2005. The dependencies for git annex are too new for this relic–even the version of glibc is too old! I was able to get git installed on my work filesystem but not git annex. This poses a problem when I update my git repository at work because all of the links to the git annex content are broken.

I thought I found a solution by adding an “rsync remote” on my personal computers. I set up a work remote by using (the line continuations are put in for clarity):

git annex initremote work type=rsync \
rsyncurl=work.com:~/research/project/.git/annex \
encryption=none

Now I can copy my annex contents to my work directory and it’s available to use there, right? Not quite.

For some reason, the way the annex contents are stored differs when you use a local git repository and when you use an rsync or “directory” special remote. For a local repo, they’re stored like this:

.git/annex/objects/ab/cd/key/key

Whereas for an rsync/directory special remote as I’ve set it up above:

.git/annex/abc/def/key/key

The way ab, cd, abc, and def are computed is based on the MD5 sum of the key, but the computation for ab/cd differs from abc/def. So even though I can copy all the annex contents to my work git repo, none of the files in the repo are pointing to the annex content correctly.

This behavior has bothered me for a long while, so tonight I finally did something about it. I found some helpful information here which gave me enough information to create a script to fix these symlinks.

I’ve put the script in my github repo under the MIT license. This has already been quite useful for my work, and if someone else can benefit from it too, then great! Suggestions and improvements welcome. 🙂

Advertisements

Configuring Arch Linux on a Thinkpad T420 August 21, 2012

Posted by Rich in laptop, linux.
Tags: , ,
2 comments

Configuring Arch Linux on a Thinkpad T420

This is a follow-on post to last time where I did a basic install of Arch linux on my Thinkpad T420.

The default Arch installation is very bare-bones, so there are several steps to getting it configured properly, such as adding a user, installing X11, and so on. Here are the T420-specific steps and notes I took to get it configured properly:

Basics

  • Install intel-ucode for latest CPU microcode, add to MODULES in /etc/rc.conf
  • While KMS is already enabled, we can enable it earlier in the boot process by adding i915 to MODULES in /etc/mkinitcpio.conf and re-generating the initcpio file
  • Increase the GRUB boot screen resolution by changing GRUB_GFXMODE in /etc/default/grub to 1024x768x32, then grub-mkconfig -o /boot/grub/grub.conf
  • Add repo-ck, install linux-ck-corex kernel, enable BFQ I/O scheduler by adding it to /etc/default/grub, and regenerate /boot/grub/grub.cfg to add the new kernel
  • Install ssmtp and forward system mail to my gmail account. Add the normal user to the mail group.

Multimedia

  • For alsa: install alsa-utils and use alsamixer to unmute and speaker-test to test sound, add alsa to DAEMONS in /etc/rc.conf
  • Disable PC speaker beep (didn’t happen until I unmuted ALSA) by blacklisting the pcspkr module
  • Bluetooth: install bluez, blueman. I finally got my pair of bluetooth headphones working by adding Enable=Socket and Enable=Source to /etc/bluetooth/audio.conf

GUI stuff

  • After installing Xorg, install the Xorg Intel driver: xf86-video-intel, and enable SNA, I also installed lib32-intel-dri for accelerating 32-bit applications. I found later that if I enable the "TearFree" option then xrandr doesn’t work for multiple displays properly except for mirroring.
  • Installed mplayer-vaapi, gstreamer-vaapi, and libva-driver-intel. Now running mplayer file.avi -vo vaapi gives hardware-accelerated decoding.
  • There’s a Thinkpad OSD available as tpb, but it’s really old and xfce (the DE I use) has a built-in OSD for brightness changes. So I won’t bother with it, but it’d be nice to have an OSD for volume changes someday.

Power management

  • Suspend-to-RAM (S3 sleep) works out of the box. I can even have mpd playing and suspend/resume is fine.
  • Hibernate (S5 sleep) does not work out-of-the-box, but it does after I add resume to /etc/mkinitcpio.conf between lvm2 and filesystems
  • Installed gnome-power-manager for the handy tool gnome-power-statistics
  • One feature that I like about Windows 7 is that after your laptop has been in sleep for some time, it wakes up and goes into hibernation. I found a script that does a similar thing using rtcwake.
  • I don’t think it takes a lot of power, but it’d be nice to disable the Ultrabay DVD drive on battery, since I rarely use it (especially on battery). laptop-mode-tools doesn’t seem to support this, but I found in /sys/devices/platform/dock.1 some good stuff. docked is normally set to 1, indicating that it’s docked. If I write 1 to undock then docked changes to 0. If I hit the eject button then it takes a couple of seconds longer to eject. So I think this shuts power off to the drive, and power is re-connected by hitting the eject button. This matches behavior under Windows 7. For now, I’m not going to bother with automatically shutting down the drive every time I’m on battery, but I could do this in the future.
  • Installed tp_smapi, pm-utils, acpi, powertop, acpid
  • The start_charge_thresh parameter for tm_smapi isn’t supported on the T420, but the AUR package tcacpi-bat fixes this. There are instructions here. Note that I need to specify battery 1, I can’t do battery 0 (both). As directed in the link, I add the lines to /etc/rc.local to set the thresholds upon boot-up, although it’s only needed whenever I remove/reinsert the battery.
  • Installed cpupower, and I don’t need to worry about any configuration stuff since the ondemand governor is automatically loaded for linux > 3.4
  • Installed laptop-mode-tools, sdparm
  • In configuring laptop-mode-tools, I changed lcd-brightness.conf so laptop-mode controls it and changed BRIGHTNESS_OUTPUT to /sys/class/backlight/acpi_video0/brightness, and set battery/AC values to 8/15 (out of 15 maximum levels).

Input

  • Installed synaptics as detailed here. One hiccup is that I like both edge scrolling AND two-finger scrolling, and the xfce settings only let you choose one or the other. Furthermore, if you enable both in /etc/X11/xorg.conf.d/10-synaptics.conf, that works fine until XFCE loads and overwrites your setting. So I added an XCE autostart script to call synclient VertEdgeScroll=1; synclient HorizEdgeScroll=1.
  • The power button is not detected with xfce4. It’s not a big deal.
  • I like to disable the touchpad (mainly the buttons, since it’s on the edge of the laptop and I accidentally click these if I’m resting the laptop on my stomach), so I follow these to enable the toggle button. Instead of using xbindkeys, I used the xfce4 keyboard handler to call a one-liner script in my home directory to toggle the touchpad. I call it by using Fn+F8, the trackpad enable/disable button.

Thinkpad-specific features

  • Installed hdapsd, hdaps-gl, xfce4-hdaps, set hdaps sensitivity to 35
  • Installed thinkfan and added a configuration similar to this one. I tweaked the thresholds a bit. Also, as indicated here, the thinkpad_acpi module needs an option to enable control of the fan speeds.
  • I added some kernel parameters here. I apparently don’t need the pcie_aspm=force argument since I’m running a kernel newer than 3.3. I haven’t compared power usage with and without these arguments, so I’m just keeping them for now.

Conclusion

I’d like to try switching to systemd in the near future, but these notes have gotten me up and running quite well.

Installing Arch Linux on a Thinkpad T420 with system encryption July 22, 2012

Posted by Rich in laptop, linux.
Tags: , , , ,
7 comments

I completed a basic install of Arch Linux over the weekend on my new Thinkpad T420. As mentioned in my previous post, I converted to UEFI mode and partitioned the hard drive in GPT format. I wanted to install Arch Linux for UEFI/GPT while using full system encryption using LUKS. Since it took a few tries to get all of the steps right, I’m posting what I did here in tutorial form.

Note: be wary of copying-and-pasting my commands for your configuration, since my drive letters and such won’t necessarily be typical.

The 2011-08-19 Arch installation media is rather out-of-date, so I elected to choose a more recent testing ISO. Word on the mailing lists indicates that they’ll release an updated official installation ISO soonish, but I don’t have the patience to wait for that. So, I used the one here and downloaded this iso. For BIOS systems, it’s easy to copy the ISO to a USB flash drive for installation purposes by doing something like dd if=archlinux.iso of=/dev/sdb bs=4M. A UEFI system requires a bit more than this. So, as per this README, I performed the following steps to get my USB media bootable for UEFI:

  1. Install gptfdisk on my desktop to get access to GPT partitioning utilities
  2. Create a GPT setup on the USB drive: sudo gdisk /dev/sdc, then o to create a blank GPT, n to create a new partition. I used the default size (whole disk) and set the type to ef00, an EFI system partition. I checked that it looks right with p to print the partition table, then w to write and quit.
  3. Create a FAT32 filesystem: sudo mkfs.vfat -F 32 -n ARCH_201207 /dev/sdc1 Note: It’s important that the EFI FAT32 partition is named ARCH_201207 otherwise the installer fails to boot.
  4. Mount the new filesystem: sudo mount /dev/sdc1 /media/sdc1 or pmount /dev/sdc1 if you use pmount
  5. Extract the ISO to the filesystem: bsdtar -x --exclude=isolinux/ --exclude=arch/boot/syslinux/ -f ~/Downloads/archlinux-2012.07.21_04-00-02-netinstall-dual.iso -C /media/sdc1
  6. Extract the contents of the EFI boot image: mcopy -s -i /media/sdc1/EFI/archiso/efiboot.img ::/EFI /media/sdc1/
  7. Remove the unneeded boot image: rm /media/sdc1/EFI/archiso/efiboot.img
  8. Unmount the USB: umount /media/sdc1 or pumount /dev/sdc1

And now the USB installation media is ready and we’re ready to boot.

Once booted into the Arch installation environment, the first thing to do is set up dm-crypt with LUKS. I’m using this page as my reference for this step. This is what I did:

  1. Create the boot partition: I used cgdisk /dev/sda to create a new partition. The boot partition doesn’t have to be big, so I opted for a 128 MiB partition with GPT partition type 8300, a Linux filesystem partition. GPT lets you use fancy partition names, so I opted for Arch boot partition. With the three previous partitions for Windows 7, this means that /dev/sda4 is my boot partition.

  2. Create the LUKS partition in the remaining free space of the disk: Still in cgdisk /dev/sda, I created a partition to fill the rest of the disk. I couldn’t find anything online to indicate what partition type/GUID to set, so I used 8e00, Linux LVM. I named the partition Arch Linux LUKS partition. This partition is now /dev/sda5.

  3. Securely erase the LUKS partition by filling it with random data: badblocks -c 10240 -wvst random /dev/sda5. This took a couple hours, but is much faster than using /dev/random or /dev/urandom as an input to dd.

  4. Setup dm-crypt/LUKS: It’s possible to use a keyfile and/or a passphrase. I opted for the passphrase only and the cipher suggested by the wiki page I’m following. The command is cryptsetup -c aes-xts-plain -y -s 512 luksFormat /dev/sda5

  5. Unlock the LUKS partition so we can install Arch on it: cryptsetup luksOpen /dev/sda5 arch-luks. Now the unlocked LUKS partition is accessed using device-mapper, /dev/mapper/arch-luks

  6. Now to create the LVM partitions. With a 238 GiB volume group, I opted for the following:

    • 15 GiB for the root partition
    • 8 GiB for the var partition
    • 4 GiB for the swap partition (I have 8 GiB RAM, so will probably never need swap, but I want hibernation support. Apparently the RAM is compressed before saving in the swap partition upon hibernation, so 40% of RAM should be sufficient). If I need to adjust, I can do so later.)
    • 200 GiB for the home partition
    • This leaves 11 GiB un-allocated. If/when I need it in the future to extend any of the three above partitions, then I can do so with some LVM and ext4 resizing magic.

    Here are the steps to create the LVM partitions:

    1. Initialize the LUKS partition to be an LVM physical volume: pvcreate /dev/mapper/arch-luks, check with pvdisplay
    2. Add this LVM physical volume to a volume group: vgcreate amonkira_lvm /dev/mapper/arch-luks, check with vgdisplay. I named the volume group amonkira_lvm, which is used later on.
    3. Create the logical volumes from the volume group: lvcreate -L 15G amonkira_lvm -n root, lvcreate -L 8G amonkira_lvm -n var, lvcreate -C y -L 4G amonkira_lvm -n swap, lvcreate -L 200G amonkira_lvm -n home. The LVM physical extents making up the swap partition are forced to be contiguous with -C y, whereas we aren’t so picky about the other volumes. We can check the logical volumes with lvdisplay. The logical volumes are accessed in /dev/mapper/<volume_group_name>-<volume_name> or /dev/<volume_group_name>/<volume_name>.
  7. Now we’re ready to format the partitions. I used ext4 for the non-boot partitions and added optional filesystem labels. I now have GPT partition names, LVM logical volume names, and filesystem labels.

    • Boot partition: This is not on LVM, so it’s mkfs.ext2 -L arch-boot /dev/sda4
    • / partition: mkfs.ext4 -L arch-root /dev/amonkira_lvm/root
    • /home partition: mkfs.ext4 -L arch-home /dev/amonkira_lvm/home
    • /var partition: mkfs.ext4 -L arch-var /dev/amonkira_lvm/var
    • Swap partition: mkswap -L arch-swap /dev/amonkira_lvm/swap

    I reduce or eliminate the number of reserved blocks for the filesystems (it’s 5% by default):

    • tune2fs -m 1.0 /dev/amonkira_lvm/root
    • tune2fs -m 1.0 /dev/amonkira_lvm/var
    • tune2fs -m 0.0 /dev/amonkira_lvm/home
  8. Now we have all of our partitions ready and can continue with the installation. Mount the partitions under /mnt for arch-install-scripts (these aren’t all strictly needed for installation, but it will make generating /etc/fstab easier later on):

    1. mount /dev/amonkira_lvm/root /mnt
    2. mkdir /mnt/var && mount /dev/amonkira_lvm/var /mnt/var
    3. mkdir /mnt/home && mount /dev/amonkira_lvm/home /mnt/home
    4. mkdir /mnt/boot && mount /dev/sda4 /mnt/boot
    5. swapon /dev/amonkira_lvm/swap
  9. Since this is a netinstall, I need to be connected to the Internet. I’ll bring up the wireless interface and connect to a WPA2 network using wpa_supplicant:

    1. My wireless drivers were working out-of-the-box, as confirmed with iwconfig
    2. Bring up the interface with ip link set wlan0 up
    3. Save old wpa_supplicant config: mv /etc/wpa_supplicant.conf /etc/wpa_supplicant.conf.original
    4. Add network name/password to new config: wpa_passphrase linksys "my_secret_passkey" > /etc/wpa_supplicant.conf
    5. Load wpa_supplicant to associate with the network: wpa_supplicant -B -Dwext -i wlan0 -c /etc/wpa_supplicant.conf
    6. Request an IP address: dhcpcd wlan0
    7. Test connection: ping -c 3 http://www.google.com
  10. Install the base system: pacstrap /mnt base base-devel

  11. Install the boot loader: I’m going to install GRUB2, even though I could use the EFISTUB method to bypass GRUB entirely. Maybe I’ll switch to EFISTUB booting another day.

    1. Mount the UEFI system partition: mkdir /mnt/boot/efi && mount -t vfat /dev/sda1 /mnt/boot/efi
    2. Install GRUB: pacstrap /mnt grub-efi-x86_64
  12. Configure the system

    1. Generate the /etc/fstab: genfstab -p /mnt >> /mnt/etc/fstab. I found I had to edit it since /dev/mapper/amonkira_lvm-swap didn’t show up.
    2. Secure /etc/fstab: I changed the following mount options to my devices for /mnt/etc/fstab:
      • <device> /boot ext2 defaults,nodev,nosuid,noexec,noatime 0 2
      • <device> / ext4 defaults,noatime 0 1
      • <device> /var ext4 defaults,nodev,nosuid,relatime 0 2
      • <device> /home ext4 defaults,nodev,nosuid,relatime 0 2
    3. chroot into the new system: arch-chroot /mnt
    4. Set up /etc/rc.conf and set USELVM to YES
    5. Set up /etc/locale.gen, then run locale-gen to generate locales
    6. Configure /etc/mkinitcpio.conf, add “encrypt” and “lvm2” to hooks (in that order) before “filesystem” entry, add “ext4” to MODULES; run mkinitcpio -p linux
    7. Configure GRUB
      1. Install GRUB UEFI application to the UEFI system partition and the GRUB modules to the boot partition:
        1. Outside the chroot (in another TTY, for example), run modprobe efivars
        2. grub-install --directory=/usr/lib/grub/x86_64-efi --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=arch_grub --boot-directory=/boot --recheck --debug
        3. mkdir -p /boot/grub/locale
        4. cp /usr/share/locale/en\@quot/LC_MESSAGES/grub.mo /boot/grub/locale/en.mo
      2. Add some command-line kernel arguments in /etc/default/grub: for GRUB_CMDLINE_LINUX, add root=/dev/mapper/amonkira_lvm-root for LVM and cryptdevice=/dev/sda5:arch-luks for LUKS and resume=/dev/mapper/amonkira_lvm-swap for hibernation
      3. I decided to add Windows 7 to the Grub menu. I followed this and added it to /etc/grub.d/40_custom.
      4. Create the GRUB config file: grub-mkconfig -o /boot/grub/grub.cfg
    8. Set root password: passwd
  13. Clean up

    1. exit out of the chrooted filesystem
    2. umount /mnt/{boot/efi,boot,var,home} && umount /mnt
  14. Reboot and hopefully everything works! It did for me. 🙂

Now a basic and minimal Arch installation is done. Next post will be the post-installation configuration steps.