BlogGalleryAbout meContact
Jaganadh's bookshelf: read

Python Text Processing with NTLK 2.0 CookbookPython 2.6 Text Processing Beginners Guide

More of Jaganadh's books »
Jaganadh Gopinadhan's  book recommendations, reviews, quotes, book clubs, book trivia, book lists
Ubuntu GNU/Linux I am nerdier than 94% of all people. Are you a nerd? Click here to take the Nerd Test, get nerdy images and jokes, and write on the nerd forum! Python


Book Review: Python 2.6 Text Processing Beginner's Guide by Jeff McNei

Python 2.6 Text Processing Beginner's Guide by Jeff McNeil is one of the latest books by Packt Publishers. I received the review copy of this book before one and half months or so. Due to busy schedule I was not able to finish the review process. Finally I got enough time to review it. The book gives good insight to on different technical aspects and use of Python standards and third party libraries for text processing. It is filled with lots of examples and practical projects. I think I might have took almost one year to gather knowledge in  the topic discussed in this book, when I started my career in Natural Language Processing domain. I am giving a bit detailed review on the book here.

The first chapter of this book gives some practical and interesting exercises like implementing cypher, some basic tricks with HTML. It also discusses how to setup a Python virtual environment for working with the examples in the book. The section of setting virtual environment is nice an well written one. It gives a clear idea of how to setup virtual environments.

The second chapter deals with Python IO module. It narrates the basic file operations with Python. The use of context manager(with function) for for file processing is discussed in this chapter. I am suing Python for text processing for lat three to four years. But after reading only I found that there is something called "fileinput" in Python programming language for multiple file access. The chapter discuss how to access remote files and StringIO() module in Python. At the end of this chapter there is a discussion about IO in Python 3 too.

The third chapter is about Python String Services. It deals with string formatting, templating, modulo formatting etc. Every concept is explained with necessary mini projects which followed from chapter two. The chapter gives a comprehensive view on advanced string services in Python.

The fourth chapter is entitled as Text Processing Using the Standard Library. This chapter deals with topic like reading wnd writing csv files(csv file processing), playing with application config files(.ini files), and working with JSON. The examples are bit long one but worth practicing for better understanding.

The fifth chapter deals with one of the key aspect in text processing "Regular Expressions". The chapter teaches basics syntax of regular expression in Python. The chapter also discusses about advanced processing like regex grouping, look ashed and look behind assertion in regular expressions. The look behind operation in regular expression is the most tricky part in dealing with regex. I think only masters in regex can do it effectively ;-) .The chapter dscuss basics of Unicode regular expressions too. The chapter is filled with enough examples for each and every concept discussed.

The sixth chapter deals with Markup Languages. The chapter discusses about XMl and HTML processing with Pytho standard libraries. xml.dom.minido, SAX,laxm and BeautifulSoup packages are discussed with illustrative examples.

The seventh chapter is entitled as Creating Templates. "Templating involves the creation of text files, or templates, that contain special markup. When a specialized parser encounters this markup, it replaces it with a computed value". The templating concept was quite new to me. But I got a good insight on the topic from this chapter. The chapter discusses some libraries like "Makeo" for templating task.

The eight chapter deals with localization (l1on) and encoding. If you are working with non-English data this chapter is a must read for you.The chapter discuses about character encoding, Unicode processing and Python3 too. Apart from mere Python stuff this chapter gives a good insight about charter encoding too.

The ninth chapter Advanced Output Formats is quite useful if you are trying to create output in PDF, CSV orExcel format. This chapter discuss about ReportLab a PDF generation library in Python.The only disadvantage which I found in ReportLab is its lack of complete Unicode Support. The chapter also discusses about creating excel files with xlwt module. Finally the chapter deals with handling OpenDocument format woth ODFPy module too. I used to read excel file from Python. But after going through this book I am able to even write Excel output too.

The tenth chapter deals with Advanced Parsing and Grammars. This is one of the key skill which required for Python text processing peoples. Creating custom grammars for parsing specific data. Through out my career I spent lot of time to train Engineers to understand parsing and BNF grammar. This time I got a good pointer for my people to start with BNF and Python programming. Also this chapter discusses about some parsing module in NLTK my favorite Python library. Some advanced topics in PyParsing also discussed in this chapter.

The eleventh and last chapter is the most interesting one in the book. The chapter deals with Searching and Indexing. PyLucene is the bset known Searching Index library in Python. But it is a wrapper to the apache Lucene. But his chapter discusses about another Python tool Nucular. Practical examples for creating search index etc are given in this chapter. This is the first time I am using the Nucular tool. I feel it as a nice and easy one compared to PyLucene. But I dont think this is superior than Lucene. I will play more with this tool and will update it in another blog post.

There are two appendix . The first appendix gives pointers to Python resources. The next one is answer o the pop quiz in the chapters.

I will give 9 out of 10 for this book. If you are dealing with rigorous text processing this book is a must have reference for you.

Related Entries:
New book by Packt:'Python Text Processing with NLTK2.0 Cookbook'
Again Python programming in Malayalam
Python3 ZWJ and Malayalam; some doubts
Python3 is wonderful
HBase Administration Cookbook by Yifeng Jiang : Review
Comments (1)  Permalink

PyLucene in Action - Part I

PyLucene is a Python wrapper aroung the Java Lucene. The goal of this tool is use Lucene's text indexing and searching capabilities from Python. It is compatible with the latest version of Java Lucene. PyLucene is embeds a Java VM with Lucene into Python process. More details on PyLucene can be found at

In this blog post I am going to demonstrate how to build a search index and query a search index with PyLucene. You can see the installation instruction for PyLucene im my previous blog post ....
1) Creating an index with Pylucene

I am using the below given code to create an index with the PyLucene

===Code Python BEGIN ===========================================

#!/usr/bin/env python
import os,sys,glob
import lucene
from lucene import SimpleFSDirectory, System, File, Document, Field, \
StandardAnalyzer, IndexWriter, Version
Example of Indexing with PyLucene 3.0
def luceneIndexer(docdir,indir):
Index Documents from a dirrcory
INDEXIDR = indir
indexdir = SimpleFSDirectory(File(INDEXIDR))
analyzer = StandardAnalyzer(Version.LUCENE_30)
index_writer = IndexWriter(indexdir,analyzer,True,\
for tfile in glob.glob(os.path.join(DIRTOINDEX,'*.txt')):
print "Indexing: ", tfile
document = Document()
content = open(tfile,'r').read()
print "Done: ", tfile
print index_writer.numDocs()
==== Code Python END ============================================

You have to supply two parameter to the luceneIndexer().
a) A path to the directory to where the documents for indexing is stored
b) A path to the directory where the index can be saved

2) Querying an index with Pylucene

The below given code is for querying an index with PyLucene

======= Code Begin Python =======================================
#!/usr/bin/env python
import sys
import lucene
from lucene import SimpleFSDirectory, System, File, Document, Field,\
StandardAnalyzer, IndexSearcher, Version, QueryParser
PyLucene retriver simple example
INDEXDIR = "./MyIndex"
def luceneRetriver(query):
indir = SimpleFSDirectory(File(INDEXDIR))
lucene_analyzer = StandardAnalyzer(Version.LUCENE_30)
lucene_searcher = IndexSearcher(indir)
my_query = QueryParser(Version.LUCENE_30,"text",\
MAX = 1000
total_hits =,MAX)
print "Hits: ",total_hits.totalHits
for hit in total_hits.scoreDocs:
print "Hit Score: ",hit.score, "Hit Doc:",hit.doc, "Hit String:",hit.toString()
doc = lucene_searcher.doc(hit.doc)
print doc.get("text").encode("utf-8")
luceneRetriver("really cool restaurant")

In the code I have manually specified the index dir """INDEXDIR = "./MyIndex" """. Instead of this one can receive the index directory as a command line parameter (sys.argv) too.

When using the function luceneRetriver() you have to give a query as parameter.

The source code is available in bitbucket

Happy Hacking !!!!!!!!!!
Related Entries:
Using Yahoo! Term Extractor web service with Python
Python workshop at Kongu Engineering College, Perundurai
FOSS Workshop at PSR Engineering College Sivakasi
CSV to CouchDB data importing, a Python hack
Book Review: Python 2.6 Text Processing Beginner's Guide by Jeff McNei
Comments (2)  Permalink

Installing PyLucene 3.x in GNU\Linux

"Lucene is a high performance, scalable Information Retrieval (IR) library. It lets you add indexing and searching capabilities to your applications. Lucene is a
mature, free, open-source project implemented in Java; it’s a member of the popular Apache Jakarta family of projects, licensed under the liberal Apache Software License.

" PyLucene is a Python extension for accessing Java Lucene. Its goal is to allow you to use Lucene's text indexing and searching capabilities from Python. It is API compatible with the latest version of Java Lucene, version 2.9.0 as of October 13th, 2009."

I downloaded PyLucene 3.X version and tried to install. Initially I got some error. I was trying to follow the instructions given at There was a confusing statement to me "<edit to match your environment>" :-(. Where to match ?? I scrated my hed for 2 days. Finally I identified the place to edit. Insted of discussinf error I will try to explain how to install PyLucene in GNU\Linux.

Download PyLucene 3.0.0-1 source from .

Extract the source. Now you will be having 'pylucene-3.0.0-1' directory. Change to the directory. Then execute the command "pushd jcc". Now you will be in the jcc directory inside the PyLucene dir. Open the '' file. You have to make some changes in the file to install it properly. In there will be portion like this

    JDK = {
            'darwin': '/System/Library/Frameworks/JavaVM.framework/Versions/Current',
            'ipod': '/usr/include/gcc',
            'linux2': '/usr/lib/jvm/java-1.6.0-openjdk-',
            'sunos5': '/usr/jdk/instances/jdk1.6.0',
            'win32': 'o:/Java/jdk1.6.0_02',
If you are using GNU\Linux you have to change the line "'linux2': '/usr/lib/jvm/java-1.6.0-openjdk-'," . Here you have to specify the path to your Java installation. Here in the example '/usr/lib/jvm/java-1.6.0-openjdk-' is the Java installation path in my system. If you are using M$ windows you have to edit the line "'win32': 'o:/Java/jdk1.6.0_02',".

Now it is time to build and install JCC .
Be superuser and execute the command "python install".
If everything is ok JCC will be installed.

Then execute the command 'popd'. Now you are in the PyLucene source dir again.

In this directory you gave to some editing . Open the 'Makefile'. There will be a portain like this in the file.

"""    #Linux     (Ubuntu 8.10 64-bit, Python 2.5.2, OpenJDK 1.6, setuptools 0.6c9)
    #JCC=$(PYTHON) -m jcc --shared

Uncomment the five lines followed by "#Linux     (Ubuntu 8.10 64-bit, Python 2.5.2, OpenJDK 1.6, setuptools 0.6c9)".
Now the portain should look like
    #Linux     (Ubuntu 8.10 64-bit, Python 2.5.2, OpenJDK 1.6, setuptools 0.6c9)
    JCC=$(PYTHON) -m jcc --shared

If you are using Python 2.6 some more editing to be done. Change the line "JCC=$(PYTHON) -m jcc --shared" to "JCC=$(PYTHON) -m jcc.__main__ --shared".

Run make and make install command. If every setting is PyLucene will be installed in your system. To check your installation rum 'make test' command.

Now enjoy !!!

Happy Hacking!!!!!!!!!!!

Related Entries:
Using Yahoo! Term Extractor web service with Python
Python workshop at Kongu Engineering College, Perundurai
FOSS Workshop at PSR Engineering College Sivakasi
CSV to CouchDB data importing, a Python hack
Book Review: Python 2.6 Text Processing Beginner's Guide by Jeff McNei