Using R from Python - the best of both worlds

R is a programming environment containing every known statistical method; however, it's ugly and difficult to program in. Python has only a small number of statistical methods (which are available through SciPy), but it's elegant and easy to program in. Both have graphing libraries which leave a lot to be desired in terms of ease of use, but the graphs from Python's matplotlib are a lot more polished. So, I'd like to use R from Python...

In order to do this, you will need to install the RPy library (as well as Python and R, of course). I'm working on Windows, but I've used this previously on Linux. As an example, here are three ways to access the same hclust object from Python. They each illustrate different aspects of the Python/R interface:

from rpy import *

# START OF METHOD 1
hclust = r("""
    a <- read.table("cpOfSimMatrix.txt")
    mydist <- dist(1-a)
    hclust(mydist)
""")
r("rm(a)") # Ends with an error if you leave anything in memory
# END OF METHOD 1

# START OF METHOD 2
set_default_mode(NO_CONVERSION)
a = r.read_table("cpOfSimMatrix.txt")
mydist = r.dist(r["-"](1,a)) # Note the trick for '1-a' here
set_default_mode(BASIC_CONVERSION)
hclust = r.hclust(mydist) # Converts R object to Python dict
# END OF METHOD 2

# START OF METHOD 3 (here's one I created earlier)
r.load(".RData")
hclust = r('myHclust')
# END OF METHOD 3

For further examples, check out Peter Cock's excellent pages on combining R and Python.

Take a molecule for a spin with Avogadro

Avogadro 0.2 is an early stage release of a new open source molecular viewer and editor. It's available for Windows, Linux and MacOSX, so there's no excuse not to try it. I took it for a test drive for the first time a week or so ago (this was on Windows), and was very impressed. Sure, there are some rough edges, as you might expect with such a low version number, but it's already capable of matching the existing competition and is sure to become the molecular editor of choice with the next release.

Once it starts, open up a molecular structure file (for example, a PDB entry) and immediately choose the navigation tool (next to the Pencil in the Tools toolbar). The left mouse button rotates the molecule, the middle zooms, and the right translates. So far, so Rasmol.

However, it's the attention to detail in the user interface that makes Avogadro stand out from the crowd. If the initial click to rotate is on an atom, the molecule rotates around the atom, whereas otherwise the molecule rotates around its centroid. Something else you will notice is the eye-candy, which really aids clarity. When rotating, a set of curvy arrows indicate the degree of rotation. If you choose the Bond Centric Manipulation Tool (indicated by an icon containing the number 90) and click on a bond, you will see bond angles lovingly displayed as semi-transparent segments. Then if you click and drag on an atom adjacent to the bond, the visualisation of the dihedral angles as you alter them is pretty cool (see below).

So what else has Avogadro got? I didn't try the molecular builder, the force-field optimisation nor the export to POV-Ray, but they're all there. I'm more interested in what's to come. Avogadro uses a plugin architecture, which means that it will be easy to incorporate third-party add-ons. Along with forthcoming support for scripting languages, this will allow me to incorporate elements of cclib or GaussSum into Avogadro.

But that's all in the future. For now, I'm just wondering how many times I can rotate this dihedral angle until it falls off...

Preventing Spam on Mediawiki at SourceForge

Mail spam is annoying. But spam on a wiki is even more annoying. If you've spent your free time writing documentation, there's no way you want to see junk text or link spam inserted into the middle of your flowing prose. In an earlier article, I described how to install a Mediawiki wiki on SourceForge. After a while you will start to get your first spam. And so the battle begins...

Wikis don't look very impressive as a website, but for an open source project they are a great way of maintaining documentation. Nobody likes writing documentation, which is why it needs to be made as easy as possible. For example, if a user mails the project mailing list with a question about installing that software, it usually means that the documentation needs to be updated and this can be done in about a minute. (On the other hand, if a project has a frequently-asked questions page, it means that they couldn't be bothered improving the documentation.)

Fortunately, spam can simply be controlled using permissions (thanks to David Wild for some of these suggestions). To begin with, make sure you keep an eye on the Recent Changes feed on your wiki (use an RSS reader). (As an aside, if your Recent Changes feed is broken, a possible cause is that you have installed an extension and included a rouge blank line after the final "?>" - this happened to me.)

Next you need to edit the permissions in LocalSettings.php to disable anonymous edits and disable account creation except by users who already have accounts. Of course, the admin is always allowed to do whatever. I'll explain the rationale for this below, but first here are the settings:

# No anonymous editing allowed
$wgGroupPermissions['*']['edit'] = false;
$wgGroupPermissions['user']['edit'] = true;
$wgGroupPermissions['sysop']['edit'] = true;
# Only users with accounts can create accounts
$wgGroupPermissions['*']['createaccount'] = false;
$wgGroupPermissions['user']['createaccount'] = true;
$wgGroupPermissions['sysop']['createaccount'] = true

On one of the wikis I am involved with on SourceForge, we had already disabled anonymous edits. However, anyone could create an account. At some point the spammers upgraded their spam software and were then able to create accounts on the wiki. Since the RSS feed for recent changes was not working (for the reason described above), it was three days and about 1000 spam accounts later before I realised the problem. At that stage it was too late to implement the solution I described above. Instead, I created a new group 'human', added the 10 or so real accounts to that group and gave them permissions while simultaneously removing all permissions from the regular 'user' account.

# Only 'humans' can edit
$wgGroupPermissions['*']['edit'] = false;
$wgGroupPermissions['user']['edit'] = false;
$wgGroupPermissions['human']['edit'] = true;
$wgGroupPermissions['sysop']['edit'] = true;
# Only 'humans' can create accounts
$wgGroupPermissions['*']['createaccount'] = false;
$wgGroupPermissions['user']['createaccount'] = false;
$wgGroupPermissions['human']['createaccount'] = true;
$wgGroupPermissions['sysop']['createaccount'] = true;

Bye-bye spam. Of course, I still had to revert about 30 edits...:-/

Image credit: Spam wall by freezelight

ANN: cclib 0.8 released - parsers and algorithms for comp chem

On behalf of the cclib development team, I am pleased to announce that cclib 0.8 is now available for download.

cclib is an open source library, written in Python, for parsing and interpreting the results of computational chemistry packages. It currently parses output files from ADF, GAMESS (US), GAMESS-UK, Gaussian, Jaguar, Molpro and PC GAMESS. A paper is currently in press.

The main changes in this release are:
* the addition of a Molpro parser
* support for writing data to/from files as JSON
* API additions: charge, multiplicity, Natural Orbital coefficients
* New method: Lowdin population analysis
* use of Numpy instead of Numeric

Among other data, cclib extracts:
* coordinates and energies
* information about geometry optimization
* atomic orbital information
* molecular orbital information
* information on vibrational modes
* the results of a TD-DFT calculation

cclib also provides some calculation methods for interpreting the electronic properties of molecules using analyses such as:
* Mulliken and Lowdin population analyses
* Overlap population analysis
* Calculation of Mayer's bond orders.

For more information see our website, read the tutorial, or send us an email.

A few brief words about journal names

If you solved the cryptic crossword that is today's blog title, you will have realised that I'm talking about journal abbreviations. For some reason, almost all journals require you to abbreviate the names of the journals you cite, while simultaneously (and this is the bit that really gets me) not letting you know what abbreviations you should use: it is "J. Mol. Struc." or "J. Mol. Struct." or even "J. Mol. Struct. THEOCHEM"?

Perhaps you do as I do; in that last-minute tidy up of citations, I google "journal abbreviations" and try to find suitable abbreviations for the last few remaining articles. Or sometimes I spend a quiet afternoon searching through the list of citations in recent articles from that journal just to find how out to cite that particular journal!

But sometimes a little voice inside me cries "Why!" - why am I wasting the best years of my life solving the non-problem that is how to abbreviate the title of a particular journal? Don't journals want to be cited correctly? In this age of electronic linking of data, misspelling the journal abbreviation could mean that a citation to a particular journal would be missed. Since journals seem to think that Impact Factors are so important, couldn't they at least give some hint what abbreviation they think you should use?

So, on to my favourite part: "The Case Study", aka "The one that pushed me over the edge". I wanted to cite a paper from Nucleic Acids Research. This is commonly known (to me) as NAR. Indeed, the website, is nar.oxfordjournals.org. And the blurb on the front page says "Nucleic Acids Research (NAR) is a fully Open Access journal". Fantastic, NAR must be the accepted journal abbreviation. Job done.

Just to be sure, let's check the Instructions for Authors. Here they give an example of how they wish references to be formatted. The great thing is that the examples they give are from NAR:

1. Schmitt,E., Panvert,M., Blanquet,S. and Mechulam,Y. (1995) Transition state stabilisation by the 'high' motif of class I aminoacyl-tRNA synthetases: the case of Escherichia coli methionyl-tRNA synthetase. Nucleic Acids Res., 23, 4793-4798.

...but wait a second, that's "Nucleic Acids Res." not "NAR"!

Now our curiosity (aka annoyance) is piqued, so let's check out the current issue of Nucleic Acids Research. But what's that in the title of the page, "Nucl. Acids Res. -- Table of Contents...".

So NAR, or "Nucl. Acids Res." or "Nucleic Acids Res."? And don't even get me started on why journals require these abbreviations in the first place even if they are web-only...

Image credit: Raistrick's Index to Legal Citations and Abbreviations by ex_libris_gul (CC BY-NC-SA 2.0)

At last - merge, split and create PDFs with open source tools

I recently had to manipulate some PDFs, and I was pleasantly surprised that things had improved so much since the last time I had to do this, a couple of years ago.

Whatever you feel about PDFs (for example, you may believe they are a very effective way of destroying all scientific data in the literature), your opinion will suddenly take a nosedive the first time you have to, for example, extract a single page from a PDF, or merge two PDFs together. At this point, you will suddenly realise that Adobe own you, and that you will need to buy Adobe's software if you want to perform this trivial task.

But help is at hand in the form of Open Source software. I found that I was able to manipulate PDFs by using Open Source tools, and on Windows. The first thing to do is to install PDFCreator (GPL). Once it's installed, when you print from any application (for example, Word) you just choose the PDFCreator 'printer', and click OK. After a couple of seconds, a dialog box will pop up where you can just click "Save" (or you might want to adjust the page size to A4 or Letter), and it will make a PDF with the same name as your original document.

A couple of years ago, I used PDFCreator and it worked 96% of the time. That is, Word documents sometimes had strange symbols inserted instead of Greek letters or bullet points; also, extra spacing was sometimes inserted in lines with some text in superscript. This time, it worked perfectly. Well done, PDFCreator creators.

Scanned documents are now often provided as PDFs. I needed to merge some pages of one scanned document with my new PDF. For this, I needed Pdftk (GPL). The blurb says it all:

If PDF is electronic paper, then pdftk is an electronic staple-remover, hole-punch, binder, secret-decoder-ring, and X-Ray-glasses. Pdftk is a command-line tool for doing everyday things with PDF documents.

Here's an example commandline which is similar to the one I actually used (again I did this on Windows). It creates a combined PDF which consists of pages 1-7 from one.pdf, 1-5 from two.pdf, and ends with page 8 from one.pdf.

pdftk A=one.pdf B=two.pdf cat A1-7 B1-5 A8 output combined.pdf

If you ever need to deal with PDFs, hopefully these tools can help reduce the pain.

ANN: Frog donates code to OpenBabel for SMILES to 3D conversion

Recently, researchers at the French research institutes INSERM and CNRS developed an online service for converting SMILES string to 3D conformers: "FRee Online druG 3D conformation generator (Frog)". A description of this service was published in T. Bohme Leite, D. Gomes, M.A. Miteva, J. Chomilier, B.O. Villoutreix and P. Tufféry. Nucleic Acids Research, 2007, 35, W568-W572:

Frog is an on-line service aimed at generating 3D conformations for drug-like compounds starting from their 1D or 2D descriptions. Given the atomic constitution of the molecules and connectivity information, Frog can identify the different unambiguous isomers corresponding to each compound, and generate single or multiple low-to-medium energy 3D conformations, using an assembly process that does not presently consider ring flexibility. Tests show that Frog is able to generate bioactive conformations close to those observed in crystallographic complexes.

On behalf of the OpenBabel project, I am pleased to announce that Dr. Bruno Villoutreix (INSERM, University of Paris 5) and Dr. Pierre Tufféry (INSERM, University of Paris 7) have generously donated their code to OpenBabel. This code will be incorporated into OpenBabel under the GPL in the coming months, making fast and accurate SMILES-to-3D conformer generation available to the open source community for the first time.

The absence of an open source 3D conformer generation algorithm has increasingly become a problem in recent years due to the popularity of SMILES strings for the description of molecular information. Fortunately, this problem has now been solved. Thanks again to all those involved in the development and release of this code.

For further information on Frog, please contact the corresponding author of the Frog paper.

Image credit: gottcha78