tag:blogger.com,1999:blog-47736832456705894252024-03-12T23:10:05.961-04:00Full text search on Tryton ERPdfamoratohttp://www.blogger.com/profile/17788192456358770703noreply@blogger.comBlogger7125tag:blogger.com,1999:blog-4773683245670589425.post-72108202442480710882011-07-12T20:28:00.000-04:002011-07-12T20:28:55.660-04:00Draft Version of Tryton_Sphinx integrationI have made a draft version of the module to connect Tryton to Sphinx Search server.<br />
<br />
This module currently gets all the objects from the Tryton.pool.Pool and make any field that was originally selected by the author to be indexed (an searchable) on the database also searchable on Sphinx Search, with the benefit of full-text indexing on char and text fields, while making other fields available as attributes.<br />
<br />
I choose to implement this way in order to maintain compatibility with the current modules developed for Tryton 1.8, 2.0 and the future Tryton 2.2, in a manner that no module developer should make any additional effort in order to make their module compatible with the new search methodology if the Tryton.<br />
<br />
I have pushed my code to a github repository, along with the basic instruction on how to download, install, create the sphinx configuration, index the Tryton Pool ojects and run a searchd daemon in order to listen to incoming search queries.<br />
<br />
https://github.com/dfamorato/tryton_sphinxdfamoratohttp://www.blogger.com/profile/17788192456358770703noreply@blogger.com0tag:blogger.com,1999:blog-4773683245670589425.post-82413233447717234712011-06-20T12:23:00.001-04:002011-06-20T12:31:17.198-04:00Setting up sphinx.conf file for indexing tryton product dataI've successfully configured sphinx to index Tryton data from a postgres database using inheritance for different languages support<br />
<br />
Here is how I achieved this:<br />
<br />
** Again, I am assuming that you are using ubuntu 10.04 and you followed the instructions for compiling sphinx from source as my previous blog post indicated<br />
<br />
here is the sample file which should be on:<br />
<code><br />
/etc/sphinx/etc/sphinx.conf<br />
</code><br />
<br />
<script src="https://gist.github.com/1035944.js"> </script><br />
<br />
<br />
I will update later on how to start the indexer and the changes we need to do on the postgres side in order to enable the translated product indexing to work<br />
<br />
<br />
I've been very very busy this couple weeks with my finals and problems sets of my college.dfamoratohttp://www.blogger.com/profile/17788192456358770703noreply@blogger.com1tag:blogger.com,1999:blog-4773683245670589425.post-52700822005704656072011-06-07T15:58:00.001-04:002011-06-07T16:11:03.167-04:00How to install Sphinx Search with libstemmer support on Unbuntu 10.04I have made a little recipe to install Sphinx Search with libstemmer (for added number of languages) on Ubuntu 10.04 Lucid with postgres support and no MySQL support.<br /><br />Here is what needs to be done as the root user on your Ubuntu linux terminal:<br /><br />0-) Before we start, we need to make sure all Sphinx dependecies are met:<br /><code><br />aptitude install -y python-software-properties<br />apt-add-repository ppa:pitti/postgresql<br />aptitude update<br />aptitude install -y postgresql-server-dev-9.0 build-essential<br /></code><br /><br />1-) Get the current version of sphinx and uncompress it.<br /><code><br />cd /opt<br />wget http://sphinxsearch.com/files/sphinx-2.0.1-beta.tar.gz<br />tar -xvzf sphinx-2.0.1-beta.tar.gz<br /></code><br /><br />2-) Get libsnowball / libstemmer to add more languages to the stemming process and uncompress it<br /><code><br />wget http://snowball.tartarus.org/dist/libstemmer_c.tgz<br />tar -xvzf libstemmer_c.tgz'<br /></code><br /><br />3-) Copy all files from libstemmer to the sphinx directory so they can be compile together<br /><code><br />cp -fa libstemmer_c/* sphinx-2.0.1-beta/libstemmer_c/<br /></code><br />./configure --prefix=/etc/sphinx --without-mysql --with-pgsql --enable-id64 --with-libstemmer<br /><br /><br />5-) Make and install sphinx using 4 jobs to speed up compiling (most people have 4 core processors, so make it 4 jobs)<br /><code><br />make -j4 install<br /></code><br /><br />6-) Create simbolic links to sphinx executables so it can be accesses from anywhere<br /><code><br />cd /usr/local/bin<br />run('ln -s /etc/sphinx/bin/indexer') <br />ln -s /etc/sphinx/bin/indextool <br />ln -s /etc/sphinx/bin/search <br />ln -s /etc/sphinx/bin/searchd <br />ln -s /etc/sphinx/bin/spelldump<br /></code><br /><br /><br />Now, follow instruction on the next blog post in order to configure you sphinx instalationdfamoratohttp://www.blogger.com/profile/17788192456358770703noreply@blogger.com1tag:blogger.com,1999:blog-4773683245670589425.post-26651072866337382022011-05-24T10:05:00.000-04:002011-05-24T11:49:18.265-04:00Status upadate Week[2-3]Status upadate Week[1-2]<br /><br />I've learned a lot this couple weeks:<br /><br />Let's start with the questions that needed answer on my previous post:<br /><br />1-) There is a Python API for interfacing with Sphinx "searchd" daemon, but not much work has been done on it. No documentation is available for the python API.<br /><br />2-) The API does work with BETA version of sphinx and also the stable versions<br /><br />3-) Because of the number of enhancements made on the Stable Beta version, it’s recommended that we should implement using the sable beta version.<br /><br />I’ve bought a book called “Introduction to Search with Sphinx”, written by lead developer of Sphinx Search.<br /><br />So far, I’ve had a crash course on linguistics, morphology processing (cats = cat, mice = mouse, going = go/goes/went and so on), lemmatisation ( converting the word to its lemma/root ) stemming ( Intentionally trying to output the stem, even if it’s not necessarily a correct word.<br /><br />This impacts the effectiveness of the algorithm and results returned by Sphinx search server.<br /><br />It is also critical on indexing records of languages different than American English.<br /><br />Aside from the linguistics, I’ve learned the basic mechanics on how “Batch Indexing” and “Real-Time Indexing” needs to be done, the different components of the Sphinx search witch are:<br />1-) Data Source (may be a database, files, data through xml pipe or a combination of them)<br />2-) The indexer ( has the rules and the queries in order to know what should be indexed)<br />3-) “Searchd” ( The daemon which should receive the search queries from the application, look for possible matches on the index and return relevant results to the application)<br />4-) The application (This is where we are going to use the available Python API to interface with the “Searchd” daemon)<br /><br />I’ve also learned about basic configurations of the components above, how to aggregate multiple data sources, how do create multiple indexes (which i believe will be extremely helpful on indexing data in different languages on Tryton) <br /><br />In order to increase relevancy on more languages, it is necessary to compile sphinx search from source, with support for snowball “string processing language for creating stemming algorithms” ( http://snowball.tartarus.org/ ) . This will extend support beyond the default English, Russian and Czech languages to better relevance matching for French, Spanish, Portuguese, Italian, German, Dutch, Swedish, Norwegian, Danish, Finnish languages.<br /><br />I will update later on the process of indexing data.dfamoratohttp://www.blogger.com/profile/17788192456358770703noreply@blogger.com0tag:blogger.com,1999:blog-4773683245670589425.post-61462795666027394452011-05-07T15:23:00.000-04:002011-05-07T15:42:00.695-04:00Status upadate Week[0-1]To summarize, this week's work which was mostly spent on research.<br /><br /> * Tryton uses the attribute `select` with values 1 and 2 to indicate if the field has to be searchable or not.<br /> * Select =1 ends up creating an index in the current backend and also reflects as a valid field for simple search in the view.<br /> * Select = 2 does not create an index, but appears in the advanced search view.<br /><br />From the Sphinx point of view, the tryton fields with `select` attribute need to be sphinx attributes as well, which allows filtering and quick searching just like how it is done now. <br /><br />Furthermore the current python APIs for sphinx are as poorly documented as sphinx itself. It is yet to be confirmed if the python API also supports all the features of Sphinx server API. A good pythonic API may be needed to be written from scratch. <br /><br />After a discussion with Bertrand (my mentor), maybe the best alternative would be to integrate sphinx directly under the tryton application layer, which would be a sufficient level of abstraction considering that Tryton implements security at Model and record levels on the application rather than the database. This also makes it transparent to the different backends supported by Tryton (now and in future) which are currently Postgres, MySQL and SQLite.<br /><br />Questions that need answers this week:<br /><br /> 1. Do we really need to rewrite the Python API ? Is it dead or alive ?<br /> 2. Does the API work with the <1.0, 2.x releases of Sphinx ?<br /> 3. Sphinx seems to have both <1.0 and 2.X releases with a `STABLE BETA` tag. Which one do we consider stable enough to be used with Tryton.<br /><br />I will update again next weekdfamoratohttp://www.blogger.com/profile/17788192456358770703noreply@blogger.com1tag:blogger.com,1999:blog-4773683245670589425.post-78764945677219271092011-04-22T19:10:00.001-04:002011-04-22T19:10:44.919-04:00Project MilestonesThe goal is to have all major development and integration done by midterm evaluation.<br /><br />So far, the idea is to support for UTF-8 in full text searching, advanced syntax on search terms (Boolean operators, keyword matching), batch and real-time indexing, relevance ranking and non-text attributes support.<br /><br />This project can also be a very powerful tool to other ideas that are proposed for Tryton. I believe historical time-line, e-commerce integration and email integration will benefit a lot from this full text search capability. I plan to discuss constantly with the other students and mentors how to make this tool capable of indexing the modules that will be developed on this GSOC<br /><br />Start of Program (May 24)<br />· Analyze current implementation of searchable fields, the functions associated with search and how data is indexed;<br />· Evaluate design changes between versions in order to support all the 3 versions that Tryton supports (in May 24th, it will be 1.8, 2.0 and 2.1);<br />· Discuss with my mentor and the core development team what are the functionalities expected from this module;<br />· Put together all ideas, feature requests and setup milestones for each feature in a project management system publicly accessible;<br />· Discuss with other students implementing Email Integration and Historical Time-line;<br />· Specify the architecture design with mentor/core developers;<br />· Start Coding !<br />Midterm Evaluation (July 12)<br />· All major (critical) functions should be completed, with unit test / doctest and a complete set of English Documentation generated using sphinx documentation;<br />· Start performance testing and tuning;<br />· Start the Code Review process so this module can be distributed as Tryton Module;<br />· Develop the necessary changes for advance search GTK Interface;<br />· If core developers find it useful, I plan to “bundle” Sphinx Search Server with “NESO”, so new users can have a simple test environment to experience the potential of the full text search;· Discuss with mentor and core team the priority of the additional / optional functions and improvements in order to make sure I can complete them by the deadline <br />Final Evaluation (Aug 16)<br />· Coding for all functionalities in the roadmap are now completed, have complete unit tests, are reviewed by core team and fully documented;<br />· I plan to add a wiki page on the Tryton Code Snippets Wiki describing the basic usage, design patterns of this module, which will then link to a page containing all sphinx documentation;<br />· Any request from mentor and core team will be accommodated;dfamoratohttp://www.blogger.com/profile/17788192456358770703noreply@blogger.com1tag:blogger.com,1999:blog-4773683245670589425.post-77877998693910382822011-04-22T19:09:00.000-04:002011-04-22T19:10:19.886-04:00Project AbstractThis project proposes implementing full text search of records and attributes in Tryton using Sphinx Search Sever ( http://sphinxsearch.com/ ) in a pythonic implementation. The idea is to improve Tryton search capability to a more flexible, scalable and powerful option.<br /><br />I propose to use Sphinx Search Server because it is very fast (C++ language), can work in multiple OSes and has windows binaries (which might be included with NESO), it is very scalable, has proven success cases, we can implement in a pythonic way using SphinxAPI (which eliminates the need to install and support Apache Tomcat if we use Apache Solr), because it supports 2 of databases used by Tryton (PostgreSQL and Mysql), maybe can also support SQLlite3 through xmlpipe (if needed) and also supports NoSQL databases and future integrations (like MongoDB and CouchDB).<br /><br />My commitment is to submit fully functional code, using a wiki or project management system to keep track of ideas and milestones (Google Code or Assembla), using mercurial as a repository, coding in a pythonic standard, following PEP008 Style Guide, make unit tests or doc tests, with a full set of English documentation generated using Sphinx Document Generator ( http://sphinx.pocoo.org/ ).dfamoratohttp://www.blogger.com/profile/17788192456358770703noreply@blogger.com1