While somewhat interesting, one question jumped out at me: what is the value of this paper? The system is specific to this one company so it’s not like I can access the code or the workflows. I can’t even buy this software. In this case, they do provide public access to some of their tools, so it’s not a totally “opaque” paper. But there are other examples where one cannot really even try out the tools described in the paper.

While I see the value of the paper from the authors point of view (spreading the word, publications being “currency”, etc.), such papers have always felt a little pointless to me, as a reader. What can I do after reading this paper? Is there anything I can follow-up on?

The program parses the OBO 1.2 formatted GO data file (either by directly downloading it or from a local file) and outputs a flat dictionary listing the term ID’s, names, namespace etc and a network representation of the GO hierarchy in ncol format. It uses a simple  (and relatively non-robust) class to represent the data as an undirected graph (not really correct), though it’d be easy to use something like igraph to start doing some real network analysis. It’s certainly not a comprehensive solution, but I thought I’d put it out there.

One of the problems that I have faced is that while one can perform a full text search for assays on PubChem, there is no form of annotations on the assays themselves. One effect of this is that it is difficult to link an assay to other biological resources (though for enzymatic assays, one can determine a Pubmed protein identifier). While working on my bioassay network project, I needed annotations and I didn’t want to do it manually.

Automated GO Annotations

While manual annotations would be the most rigorous, I wanted to see to what extent automated methods could provide useful information. A bit of searching led me to GOCat run by Patrick Ruch, which analyzes text and predicts a set of GO terms that could be associated with that text. Working with Patrick and his student Julien Gobeill, we were able to process the description field of each assay to generate a set of “predicted GO terms”. The tool identifies each term as being related to function, component or pathway. and also provides a score for each predicted term, allowing us to rank the terms associated with an assay.

Verification

The first thing to consider was correctness – are the assigned GO terms sensible? I manually verified about 30 assays and for many of them the top five terms that GOCat assigned were quite relevant (i.e.,  specific terms, that would be close to or actual leaf nodes in the GO hierarchy). In general, if one considered the top 15 or 20 terms, the relevancy was quite high. However, there were a number of assays for which the assigned terms were very general and thus, not very informative. It turns out that most of these arose from assays in which the description field was very general or very short, though there were some examples where there was a long description but the resultant predicted terms were still quite general and non-speciific. Of course, this process of verification is subjective, and one can be more or less strict as to whether predicted terms make sense for a given assay.

Overall, the results aren’t too bad, but this was a quick check that was included in an abstract submission.  More extensive verification (rigorous quality score, multiple people) are in the pipeline as we write up the paper.

Browsing

But as we proceed with verification, we are now able to link the bioassays with other biological resources via GO terms. After Patrick and Julien generated the predicted terms I put them into a Postgres database and whipped up a simple interface. It’s not complete but does let you view the terms associated with an assay to varying levels of detail (such as all terms, component-related terms etc).

Networks & Visualization

But the whole point of this exercise was to help me in my bioassay network project. More specifically I wanted to create a semantic similarity network of assays. That is, a network where the nodes are assays and two nodes are connected if the “semantic similarity” between two assays is greater than some cutoff. With the assigned GO terms, we can define a simplistic semantic similarity score by counting the number of terms in  common between the top 10 terms for two assays. With normalization this gives us a value between 0 and 1. Now, such a similarity score is quite arbitrary but provides some nice advantages as I noted in my presentation. For now it leads to some nice pictures, that I think also provide some  (high level) insight into the quality of the assigned terms. For example, using a cutoff of 0.9, we get a network that looks like Figure 1, where there are many disconnected components. Essentially, this is putting closely related assays together. This the large number of 2-node components appears to correspond to primary and secondary assay pairs. Wheres the larger clusters near the top tend to focus on groups of assays that have come from a specific center (such as the NCI DTP assays) or for a specific task (such as cytotoxicity assays). On going to lower cutoffs (Figures 2 and 3) we start seeing some interesting structures. As I noted, these are still just pretty pictures and I haven’t analysed them yet for meaning. But they do suggest these visualizations as a useful approach to exploring the assay collection compared with traditional full text search.

Great collaborators, fresh data, pretty pictures, intriguing analyses – aah life is good!

Figure 1. Semantic Similarity Network (Cutoff 0.9)

Figure 2. Semantic Similarity Network (Cutoff = 0.7)

Figure 3. Semantic Similarity Network (Cutoff = 0.6)

Infrastructure

The setup is a mod_python based REST service. Before describing the service, a little bit about the models themselves. The setup requires that you develop a model in R and then save it as a binary R file (via save). Right now you have to save the model in a variable called “model” – a bit restrictive but it might change in the future. You can build any type of model that has overloaded the predict method. Once you have that you need to edit a model manifest file that contains information on the author, description of the model and so on. More importantly, you have to specify the descriptors used in the model. This leads to a limitation – the descriptor calculation step of the service uses the CDK descriptor service and so the models must employ the CDK descriptors. While restrictive it’s not too bad, since the CDK has a wide variety of molecular descriptors. You can get more details about how models are deployed and the format of the manifest from the GitHub repository.

Usage

With the model file and the manifest details it’s pretty easy to setup a simple Python service that uses rpy2 to load the model, calculate descriptors for an input SMILES (Base64 encoded), get a prediction and return it. Thus, to get a list of available models, visit

http://rguha.ath.cx/~rguha/cicc/rest/predict/

This gives a plain text page with a list of model identifiers You can then use a model identifier to get the details of the model (as provided by the author of the model) by appending the identifier. An example would be

http://rguha.ath.cx/~rguha/cicc/rest/predict/Solubility2

Finally, to get a prediction from the above model, simply append a Base64 encoded SMILES string

http://rguha.ath.cx/~rguha/cicc/rest/predict/Solubility2/YzFjY2NjYzFDQ09DQw==

and you end up with a plain text represtation of the predicted value.

Caveats

Admittedly the current version of this service is a quick hack and has a number of restrictions. While any type of model can be deployed, something like a random forest model will require you to list many descriptors in the manifest file manually. In the future, this should probably be automated via an R function. While the manifest for a given model can contain an arbitrarily long description, it’s up to the developer to decide what goes in. Ideally, we’d serialize the model to PMML so that we could easily include details such as coefficients, training and validation statistics and so on. The use of PMML would allow easy inclusion in the manifest. On the other hand it’s relatively easy to extract this information from the model file, so it might simply require the construction of a different URL.

Another drawback is the fact that one gets a single return value. Now, it’s pretty easy to extract, say, confidence limits but this is dependent on the nature of the model. Providing more information in the return value would probably best be handled by generating PMML output.

The current format of the manifest file is pretty crude – ideally I’d use Dublin Core to represent provenance and support more details of the model (such as model type etc), thus avoiding the need to load the model file. Also, there is no schema for the format, which would be a useful addition. Some form of versioning information would also be useful.

One of the biggest performance bottlenecks is that the service deals with one SMILES string at a time. If you’re getting predictions for many molecules, this can become slow (since each prediction loads the model file). Ideally, the service would recognize a POST request and pull one or more SMILES from the fields in the request. This would allow predictions in bulk and make it much faster. Another advantage of use POST would be the ability to provide SDF (or any other multi-line) input.

Conclusions

Model deployment is now simple to achieve and Python is sweet!

http://rguha.ath.cx/~rguha/cicc/rest/depict/c1ccccc1

http://rguha.ath.cx/~rguha/cicc/rest/depict/C1CCCCC12CCCCC2

http://rguha.ath.cx/~rguha/cicc/rest/depict/CC(=O)OC1=CC=CC=C1C(=O)O

http://rguha.ath.cx/~rguha/cicc/rest/depict/c1ccccc1CC=CC%23N

Thanks to Gilleain and Egon for pointing me in the right direction. Anybody using this service should see the new depictions automatically

http://rguha.ath.cx/~rguha/cicc/rest/desc/descriptors/
org.openscience.cdk.qsar.descriptors.molecular.ALOGPDescriptor/c1ccccc1COCC

This type of URL is pretty handy to construct by hand. However, as Pat Walters pointed out in the comments to that post, SMILES containing ‘#’ will cause problems since that character is a URL fragment identifier. Furthermore, the presence of a ‘/’ in a SMILES string necessitates some processing in the service to recognize it as part of the SMILES, rather than a URL path separator. While the service could handle these (at the expense of messy code) it turned out that there were subtle bugs.

Based on Pats’ suggestion I converted the service to use base64 encoded SMILES, which let me simplify the code and remove the bugs. As a result, one cannot append the SMILES directly to the URL’s. Instead the above URL would be rewritten in the form

http://rguha.ath.cx/~rguha/cicc/rest/desc/descriptors/
org.openscience.cdk.qsar.descriptors.molecular.ALOGPDescriptor/YzFjY2NjYzFDT0ND

All the example URL’s described in my previous post that involve SMILES strings, should be rewritten using base64 encoded SMILES. So to get a document listing all descriptors for “c1ccccc1COCC” one would write

http://rguha.ath.cx/~rguha/cicc/rest/desc/descriptors/YzFjY2NjYzFDT0ND

and then follow the links therein.

While this makes it a little harder to directly write out these URL’s by hand, I expect that most uses of this service would be programmatic – in which case getting base64 encoded SMILES is trivial.

http://rguha.ath.cx/~rguha/cicc/rest/desc/descriptors/CC(=O)

you get a simple XML document containing a list of URL’s. Each URL represents a specific “resource”. In this context, the resource is the descriptor values for the given molecule. Thus by visiting

http://rguha.ath.cx/~rguha/cicc/rest/desc/descriptors/
org.openscience.cdk.qsar.descriptors.molecular.ALOGPDescriptor/CC(=O)C

one gets another simple XML document that lists the names and values of the AlogP descriptor. In this case, the CDK implementation evaluates AlogP, AlogP2 and molar refractivity – so there are actually three descriptor values. On the other hand something like the  molecular weight descriptor gives a single value. To just see the list of available descriptors visit

http://www.chembiogrid.org/cheminfo/rest/desc/descriptors

which gives an XML document containing a series of links. Visiting one of these links gives the “descriptor specification” – information on the vendor, version, reference to a descriptor ontology and so on.

(I should point out that the descriptors available in this service are from a pretty old version of the CDK. I really should update the descriptors to the 1.2.x versions)

Applications

This type of interface makes it easy to whip up various applications. One example is the PCA analysis of compound collections. Another one I put together today based on a conversation with Jean-Claude was a simple application to plot pairs of descriptor values for a collection of SMILES.

The app is pretty simple (and quite slow, since it uses synchronous GET’s to the descriptor service for each SMILES and has to make two calls for each SMILES – hey, it was a quick hack!). Currently, it’s a bit restrictive – if a descriptor calculates multiple values, it will only use the first value. To see how many values a molecular descriptor calculates, see the list here.

With a little more effort one could easily have a pretty nice online descriptor calculation application rivaling a standalone application such as the the CDK descriptor GUI

Also,if you struggle with nice CSS layouts, the CSS Layout Collection is a fantastic resource. And jQuery rocks.

]]> http://rguha.wordpress.com/2009/01/07/playing-with-rest-descriptor-services/feed/ 4 rguha dppss1