attempt_to_blog++

It’s unfortunate that this blog is turning into rants about scientific computing. Perhaps I am simply ethnically predisposed to doing science and kvetching.

Today’s rant is brought about by my collaborators’ request to make my paper figures “presentable”. The reason is simple — current figure drafts are straight up Mathematica output, relying mostly on defaults, and they suck. Let’s face it — figures produced by Mathematica can be good, but rarely great, and trying to fine-tune the appearance of frames and axes can be a daunting task.

The problem isn’t that Mathematica sucks at plots — so does most other software. The problem is that to my knowledge there are no reasonable open-source alternatives to something like Origin. So right now, if you are unsatisfied with Mathematica’s plotting capabilities, your options come down to this:

- get as far as you easily can with Mathematica, then switch to Illustrator and Photoshop. Pros: guaranteed to work. Cons: time-consuming, labor-intensive, and requires Illustrator, Photoshop, and Windows.

- import data into one of {R, Matlab, Python/Matplotlib} and hope it can do what you are after. Pros: you might get the output you want. Cons: you need to know R, Matlab, or Python.

What shocks me is that in this day and age, it seems that every few years, somebody sits down to write a plotting package, and ends up reinventing the wheel. So you have a bunch of wheels, none perfectly circular; there are all sorts of bumps and protrusions, all in different places. Within R, for instance, there are at least 3 different ways to create plots (base graphics, ggplot2, lattice), and more are being created (e.g. jjplot). However, despite all the man-hours spent, we still don’t have such basic things as TeX integration. In a software package that claims to be the premier graphics solution for a statistician/applied mathematician! The syntax to add formulas to plots is revolting. Even Excel can do better.

All right, rant over. I need to go generate some figures.

I come back from California sporting a tie-dyed t-shirt with a peace sign, good Chinese food, bubble tea and Yogurtland soft-serve still having the last of their nutrients being extracted by my system, and many conversations lingering in my mind.

I can still see the faces of my interlocutors and hear fragments of their remarks.  Yet when I think harder and try to place those fragments within some sort of a larger context — one that would paint a more complete picture of the person or convey a self-contained idea — I get lost.  I suppose it’s hard to get into people’s heads, figure out their beliefs, thoughts, and feelings, and then paint a verbal snapshot.  It’s doubly hard if you are an 83% I on Meyers-Briggs.  Perhaps next time I will try to listen as if I were a journalist seeking a coherent sound bite, or an author trying to develop a character.  Then again: the human psyche is an incredibly complex function, full of hidden variables, feedback loops, and nonlinear responses.  It takes great talent to accurately sample it and then translate it into words.  (Admittedly, if you are a TV journalist, you can always grab a couple of random points and draw a straight line.  If you are a Fox journalist, one point is sufficient.)  But that all is for next time.  Right now I’ll be content with whatever verbal and visual snapshots that are still imprinted in my mind.

“…The Western society has completely degraded.  Take a look at France — they are nothing but a banda pidorov.  I can’t wait to get out of this country.”  E. proceeded to light a cigarette and talk about the awesome slavic music + electronica parties he throws back in upstate NY.  Hanging out with him led to a sharp increase in beer consumption, appreciation for the wealth of practical scientific knowledge chemists seem to harbor, and a touch of jealosy for how easily he picks up girls.

“…Look at those two — they are like a pilot and the co-pilot.”  J. pointed at the young Russian prof and the old Russian prof sitting together at a desk in the front of the room.  The young one chaired the session.  The old one was there largely due to an overbearing feeling of self-importance.  “These guys are the assholes in the business class,” J. continued, looking at the front few rows.  “The rest are sleeping in coach.  If it were a real airplane someone would be groping a flight attendant in the back row.”  He looked at his watch and proceeded to another room, delivering  one of his trademark cynical oral presentations to an audience of 15 people, of which five were asleep and five were staring at their laptops.  The most impressive thing about J. is that he is a damn good scientist and is quite productive, all the while giving the impression that he doesn’t give a shit.

…this post will have to be abbreviated due to our impending arrival into St. Louis.  Who knows whether I’ll find time to paint any more snapshots — but I figure I’m doing quite well so far in terms of blog updates anyhow.

After spending a substantial amount of time tweaking my system, it now works as well as I want it to, and I am hereby making a public promise not to touch my config files for a month.  I want to be able to post a screenshot a month from now with a “last modified” timestamp corresponding to today.  (The only exception is M-x / smex stuff in .emacs; if the author of the package fixes the broken fuzzy matching, I’ll add it in.  No other changes are allowed, though!)

So what went into the final round of updates?..

First, there were more tweaks to .bashrc and .inputrc.  I turned on bash completion; here are some notes about this:

bash_completion is very handy, but has a few annoyances.  First, the bash startup time increased to 2+ seconds.  This was fixed by calling the completion functions dynamically — solution was found on http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=467231; dyncomp.sh script I saved locally.

To prevent cd/find from completing random useless things, put complete -d cd find in .bashrc

To prevent tilde from getting expanded: edit /etc/bash_completion; look for function named _expand(); comment out all code (but leave one dummy line e.g. foo=bar, o/w bash complains). If used dynamic completion replacement above, look for the function _expand as a sep. file in bash_completion.d

For .inputrc I added several customizations — set show-all-if-ambiguous on, history-search-{forward/backward}, etc. These things are handy; you might want to google for them if you don’t already use them. I realized a short time later that any keybindings involving meta (e.g. “\M-o”) failed to work in bash shell on Dreamhost.  The problem is a buggy version of readline that breaks in unicode locale and the solution is to replace “\M-” with “\e” (e.g. “\eo”).

Ok, now let’s get to the good stuff!  Here it is: any time I work in a terminal, I can open Windows Explorer in the current directory, or in any directory I specify.  I aliased this to a single key ‘e’.  It’s handy!  Conversely, if I am browsing a directory in Explorer, I can launch a terminal window in that directory using a single shortcut key (e.g. Win-T).  And finally, if I am in Emacs, I can launch either Windows Explorer or the terminal in the directory of the current buffer (or current dired buffer) with Win-E and Win-T respectively.  How does this magic work?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#!/usr/bin/bash
# Open explorer in the current directory or directory of the argument
#
# Unix/Windows path conversion: -- could've done it w/ cygpath in hindsight
# pwdWin=`pwd|perl -p -e 's/\/cygdrive\/(.)/\1:/; s/\//\\/g'`
# note that you have to double-escape special characters here
 
cygwin_root=/cygdrive/c/cygwin
 
if [ $# -eq 0 ]; then
    absPath="$(pwd -P)"
else
    absPath="$(realpath "$1")"
fi
 
# the following will always execute unless the path is printed as c:/...
# in which case we can proceed directly to replacement
if [ "${absPath:0:1}" = "/" ]; then # either have /cygrive/c/..., or:
	# special case: starts with /, e.g. / or /usr
    base_dir=$(echo $absPath | awk -F/ '{print $2}')
    # when absPath=="/", so basedir=="":
    if [ "$base_dir" = "" ] || [ "$base_dir" != "cygdrive" ]; then
	absPath=$cygwin_root$absPath
    fi
fi
winPath=$(echo $absPath|perl -p -e 's/\/cygdrive\/(.)/\1:/; s/\//\\/g')
explorer /e, "$winPath"

There are extra checks necessitated by idiosyncrasies of Cygwin paths. Ok, so this is standard bash scripting; no biggie. Now, how do we call terminal from Windows?.. This piece of magic requires several components.
The first is writing a program to launch the terminal (mrxvt in my case) in a particular directory. There are two ways to do this: VB script or a batch file. VB script is a more modern solution, and could look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Set WshShell = WScript.CreateObject("WScript.Shell")
WshShell.CurrentDirectory = "C:\home\Leo"
Set wshSystemEnv = wshShell.Environment( "PROCESS" )
wshSystemEnv( "DISPLAY" ) = "127.0.0.1:0.0"
'WScript.Echo "SYSTEM:  DISPLAY=" & wshSystemEnv( "DISPLAY" )
 
Dim StartInDir
If WScript.Arguments.Count > 0 Then
    StartInDir = CStr(WScript.Arguments.Item(0))
    Dim BashExeString
    BashExeString = chr(34) & "cd " & chr(34) & StartInDir & chr(34) & "; exec bash" & chr(34)
    Dim RunString
    RunString = "c:\cygwin\bin\run -p /usr/X11R6/bin /usr/local/bin/mrxvt -e /bin/bash --login -i -c " & BashExeString
    WshShell.Run RunString,0,false
Else
    WshShell.Run  "c:\cygwin\bin\run -p /usr/X11R6/bin /usr/local/bin/mrxvt -e /bin/bash --login -i",0,false
End If

The key aspect that makes this work is the execution of -c “cd [DIRECTORY]; exec bash;” command at the end of the chain run–mrxvt–bash. Run command, if you are curious, allows to launch commands that would otherwise spawn a terminal window. Exec bash is needed because otherwise bash exists after executing the -c string command.

Another way to write this is with a batch file:

@echo off
SET DISPLAY=127.0.0.1:0.0
SET CYGWIN_ROOT=\cygwin
SET RUN=%CYGWIN_ROOT%\bin\run -p /usr/X11R6/bin
SET PATH=%CYGWIN_ROOT%\bin;%CYGWIN_ROOT%\local\bin;%CYGWIN_ROOT%\usr\X11R6\bin;%PATH%
SET XAPPLRESDIR=/usr/X11R6/lib/X11/app-defaults
SET XCMSDB=/usr/X11R6/lib/X11/Xcms.txt
SET XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB
SET XNLSPATH=/usr/X11R6/lib/X11/locale
cd c:\home\leo\
%RUN%  /usr/local/bin/mrxvt -e /bin/bash --login -i -c "cd "%1"; exec bash"

It turns out that the batch file runs noticeably faster on my computer.

There is an alternative approach to specify the terminal’s start directory, in addition to the bash -c [string] command line option. You can set an environment variable and have .bashrc check for its existence and cd accordingly. This has the advantage of being a tiny bit faster than the preceding approach, but because all environment variables get cached, subsequent tabs of the terminal (mrxvt) will open in the directory in which the first tab started, which might not be the desirable behavior.

To get this working, put set STARTINGDIRECTORY=%1 into the batch file, use the run command
%RUN% /usr/local/bin/mrxvt -e /bin/bash --login -i, and put into .bashrc:

if [ ! -z "${STARTINGDIRECTORY}" ]; then # must strip literal quotation marks if any
    if [ "${STARTINGDIRECTORY:0:1}" = "\"" ]; then
	STARTINGDIRECTORY=$(echo "$STARTINGDIRECTORY"|sed 's/.\(.*\)./\1/')
    fi
    echo "starting in ${STARTINGDIRECTORY}"    
    cd "${STARTINGDIRECTORY}"
    unset STARTINGDIRECTORY
fi

Now the fun part: how do we invoke this for a particular directory as we browse it in Explorer?.. Autohotkey comes to our rescue! I found a piece of code on Stack Overflow to launch the windows command shell and tweaked it a little:

; Opens the command shell 'cmd' in the directory browsed in Explorer.
; Note: expecting to be run when the active window is Explorer.
;
OpenCmdInCurrent()
{
    WinGetText, full_path, A  ; This is required to get the full path of the file from the address bar

    ; Split on newline (`n)
    StringSplit, word_array, full_path, `n
    full_path = %word_array1%   ; Take the first element from the array

    ; Just in case - remove all carriage returns (`r)
    StringReplace, full_path, full_path, `r, , all  

    IfInString full_path, \
    {
        Run, c:\home\Leo\bin\mrxvt_win.bat "%full_path%"
    }
    else
    {
        Run, c:\home\Leo\bin\mrxvt_win.bat
    }
}

; ==== Win+T default ====

#t::
  Run, c:\home\Leo\bin\mrxvt_win.bat
Return

Notice how this hotkey is global — if we are in Explorer, it will give us a terminal started in the directory being browsed. Otherwise, it will just launch a terminal. But what about Emacs?.. We’ll do something sneaky: we’ll tell Autohotkey to send Emacs a different key combo for Win-T and use a different set of bindings in Emacs:

#IfWinActive ahk_class Emacs
#e::^f3 ;for explorer
#t::^f4 ;for terminal
#IfWinActive

And in Emacs, we write the following piece of lisp code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

(defun terminal-here ()   
  "Launch external terminal in the current buffer's directory or current dired
directory.  (Works by grabbing the directory name and passing as an argument to
a batch file.  Note the (toggle-read-only) workaround; the command will not run
in dired mode without it."
  (interactive)
  (let ((dir "") (diredp nil))
    (cond
     ((and (local-variable-p 'dired-directory) dired-directory)
      (setq dir dired-directory)
      (setq diredp t)
      (toggle-read-only)
)
     ((stringp (buffer-file-name))
      (setq dir (file-name-directory (buffer-file-name))))
      )
    (shell-command (concat "~/bin/mrxvt_win.bat \""dir"\" 2>/dev/null &") 
 (universal-argument))
    (if diredp (toggle-read-only))
))

(and there’s a similar function to launch Windows Explorer). Bind them to C-F3 and C-F4 (in reality Win-E and Win-T translated by Autohotkey) and we are all done, 5 scripting languages later! Who said administering a Windows box isn’t fun

I started playing around with running Mathematica inside Emacs.  Once I develop the set of Mathematica <-> Matlab interoperability scripts, this functionality will be invaluable.  An unexpected (pleasant) surprise was that Mathematica graphics is available at the console through Java — you just have to load <<JavaGraphics`.  The graphics performance is slow as molasses on my underpowered laptop, but it should be quite usable on a good workstation.

(Look, Ma!  No notebook!)

What’s that, by the way?.. Is this the Matlab beam propagation code snippet ported to Mathematica?  Why, yes it is — a direct translation done by Wolfram Research (although to be fair, it ended up being almost a direct translation of Matlab code).  This runs as fast as the Matlab version, but it’s also a bit of a contrived example because the beam propagation in homogeneous space is of the form exp(i k z), and this can be efficiently computed with an outer product (Transpose[Outer[Times, z, kz]]] in this case).  In more complicated cases, Matlab uses meshgrid(), and I am not sure that Mathematica could compete in terms of speed.  Even if it does, I don’t see a better way of doing things other than implement Matlab built-in constructs in Mathematica, which is kind of silly.  Thus, Matlab is still going to be the workhorse here.

Mathematica’s Manipulate[] functionality got me hooked on interactivity, so once I had the beam propagation code running smoothly, I decided to implement a primitive GUI for it — how hard could it be?..  Wasn’t hard, really, but somewhat time consuming and required lots of code.  I mean, order of 100 lines of code to do something that’s almost a one-liner in Mathematica.

While it is certainly lacking on the elegance part, it runs fast.  If I had to do it over again, though, I would implement the numerics in Matlab, dump it all into a datafile, read into Mathematica and Manipulate[] away.