Monday, October 21, 2013

Making Numpy work in PBS scripts on a cluster

I work on a computing cluster that uses the environmental modules (module from the command line) to simplify access to versioned software like Numpy.

To make Numpy work from a simple ssh session, I had to load modules in the proper order. For me this was:

module add python/2.7.3
module add atlas/3.8.3
module add suitesparse/20110224

module add numpy/1.5.1

I dropped this in my .bashrc so it's always on.

During interactive sessions, Numpy will work if you make sure your .bashrc gets sourced at the beginning of the session. I forget how I did that.

When submitting PBS scripts, I have no access to the module command. To figure out what was going on, I ran some diagnostic scripts with redirected output so I could see what was going on, e.g.:

./diagnostics.py &> out.txt

My out.txt file said something about how libpython2.7.so.1.0 could not be found. In an interactive session, I found out my Python path and my linked library paths:

echo $PYTHONPATH
echo $LD_LIBRARY_PATH

Then I took the output from those things and put them before the Numpy-including script in that bash script that the PBS script calls:

#! /bin/bash
echo Importing library stuff...
export PYTHONPATH=output_from_what_was_before
export LD_LIBRARY_PATH=output_from_what_was_before

Now I can run scripts that have Numpy in PBS!

Friday, July 19, 2013

summarize_taxa_through_plots give json error

When running Qiime's summarize_taxa_through_plots.py, I get an error No JSON object could be decoded. I fixed this by making sure that the input OTU table was in biom format and had "taxonomy" in place of the "consensus lineage" header output by make_otu_table.

running summarize_taxa_through_plots on cluster

I git-cloned Qiime into ~/src. I installed with --prefix=/home/username/local and then had to change the line in ~/.qiime_config to qiime_scripts_dir <tab> /home/username/local/bin.

In my ~/src I git-cloned qcli and installed it with --prefix=/home/username/local. I had to add /home/username/local/lib/python2.7/site-packages, /home/username/local/bin, and /home/username/local/lib to my $PYTHONPATH.

I also had to load the module matplotlib.

Thursday, July 18, 2013

changing default shell

To change from tcsh to bash:

chsh -p /bin/bash my_username

Maybe you need usermod rather than chsh. This modifies /etc/passwd.

Saturday, April 27, 2013

Ubuntu 12.04.02 on Dell Inspiron 1520

I installed from a USB stick. I kept a /home partition I had from a previous installation. I have 2GB RAM so I made a 2048 MB swap partition.

The ethernet card was recognized when running from the USB stick but not after booting into installed system (lspci listed it but ifconfig did not). I removed bcmw-kernel-source and rebooted, and the ethernet worked. I installed firmware-b43-installer, and rebooted. At this point I may have use Additional Drivers to install b43 drivers. Or I did something with b43-fwcutter. In any case, my wireless works now.

When calling suspend with the lid open, the screen goes to a terminal with a flashing cursor and then comes back to the login screen. When I close the lid, the machine honestly sleeps, so go figure.

Hibernate worked when using pm-suspend. I enabled hibernation according to standard documentation (here).

Thursday, April 18, 2013

OpenBox status bar

I wanted a simple status bar on OpenBox. I ended up using conky with the following .conkyrc:


# window
own_window yes
own_window_class Conky
own_window_type desktop
own_window_transparent yes
alignment top_middle

# stop flickering
double_buffer yes

# size
gap_x 0
gap_y 0
minimum_size 1265 20
#maximum_width 400
text_buffer_size 512

border_width 1
cpu_avg_samples 2
default_color white
default_outline_color white
default_shade_color white
draw_borders no
draw_graph_borders yes
draw_outline no
draw_shades no
use_xft yes
xftfont Inconsolata:size=12
net_avg_samples 2
no_buffers yes
out_to_console no
out_to_stderr no
extra_newline no
stippled_borders 0
update_interval 1.0
uppercase no
use_spacer none
show_graph_scale no
show_graph_range no

TEXT
${color grey}net:${if_existing /sys/class/net/eth0/operstate up}${color}eth0$else${if_existing /sys/class/net/wlan0/operstate up}$color${exec /home/my_name/scripts/network_stat.py name} ${exec /home/my_name/scripts/network_stat.py strength}${color grey}%${else}none$endif$endif $alignc${color grey}cpu:$color$cpu${color grey}%  ram:$color$memperc${color grey}%  bat:$color$battery_percent${color grey}%  vol:$color${execpi 1 /home/my_name/scripts/volume.sh}  ${color grey}${if_mpd_playing}${execpi 1 mpc | awk 'NR==2 {print $1}' | sed 's/[^a-Z]//g'}${else}not playing${endif} $alignr$alignc${time %a %d %b} $color${time %H:%M}

I made my own scripts network_stat.py to report the wireless connection and quality from iwconfig and and volume.sh to report the un/muted status and volume from amixer get Master.

I adjusted OpenBox using obconf to create a 20 px upper margin so that no program window overlays the conky panel.

Printing duplex pdfs

After I got printing on my Arch machine working, I had trouble getting pdf's to print duplex from lpr. I tried changing the Duplex option in lpoptions and the default settings for the printer in CUPS, all to no good result. I could get duplex pdf's printed using chrome or evince.

Instead I used pdftops to make a postscript file, which I can lpr in duplex just fine.

Mosh for a compute cluster

Mosh is an awesome tool for folks who repeatedly ssh to one machine. In an ideal world, you can keep stay logged in to the machine even as your computer’s internet connection turns on and off. Mosh does exactly this. I keep one terminal window open with mosh; whenever my computer reconnects to the internet, it automatically restarts the ssh session right where it left off, with no password entry or anything.

I wanted to use Mosh to connect to my computing cluster, but some dependencies were missing. It was a little confusing to get them set up, so I give the play-by-play here. First I got protobuf:

cd ~
mkdir src && cd src
wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar xvzf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./autogen.sh
./configure --prefix=/home/my_name/local
make
make install

Then I got mosh:

cd ~/src
git clone https://github.com/keithw/mosh
cd mosh
./autogen.sh
PKG_CONFIG_PATH=/home/my_name/local/lib/pkgconfig ./configure --prefix=/home/my_name/local
make
make install

If I don't specify the package config path, I get a message:
No package 'protobuf' found

Don't be so sure that your home folder is in /home/your_name on every cluster! Just do a quick pwd to make sure you know where you are. The package config path must be absolute.

Then I can use mosh to log into the cluster:
mosh cluster_name server='LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/my_name/local/lib /home/my_name/local/bin/mosh-server'

Without specifying the library path, I get an error:
error while loading shared libraries: libprotobuf.so.8: cannot open shared object file: No such file or directory

Tuesday, April 9, 2013

Mounting an SD card

My camera uses an SD card. To mount it, I needed dosfstools. I used dmesg | tail to find out which device the card was assigned to (/dev/mmblksomething). To mount, I needed to specify the filesystem (FAT16, so "vfat" from dosfstools) and the first partition on the device (/dev/mmblksomethingp1):

sudo mount /dev/mmblksomethingp1 /mnt/sd -t vfat

Tuesday, March 26, 2013

Wireless on Dell Inspiron 1520 with Arch

wicd worked fine on my home wifi network, but it failed when I was at work. Instead I got Network Manager. I use the nm-applet (which demands to be in a system tray, which I don't have) with trayer, which I call and kill when I need to access the wireless.

Getting broadcom on my old Dell Inspiron working was a pain. I ended up using b43, which I load explicity by making a file /etc/modules-load.d/load-b43.conf which just has one line:
b43

Monday, March 25, 2013

Printing on Arch

I wanted my Arch machine to print to a printer at my work. Under Lubuntu, I used the printing dialog and searched for the network printer name and specified in the setup for Mac and Windows printing.

On Arch, I needed cups (from libcups) and probably some other things, but I'm not sure (avahi, hplip). I also needed to give myself root privileges in order to access the CUPS interface. To change these privileges, I used sudo visudo then under "User privilege specification" I added a line:
my_name ALL=(ALL) ALL

Then I could get to the CUPS interface page using chrome: localhost:631. To add a printer I used Administration | Add printer. I used AppSocket and found the IP address of the printer by looking at its status screen.

I installed a driver for Xerox Phaser 6280DN by downloading an .rpm from Xerox's website, using rpmextract to get the .ppd.gz, gunzip to get .ppd, then put that in /usr/share/cups/model and selected it as the driver file during CUPS administration.