Slovenian stemmer for Sphinx

3
votes

I am searching stemming algorithm for Slovenian language that I can use with Sphinx search.

What I'm trying to achieve is for example when searching for 'jabolka', I also want results for documents containing 'jabolko', 'jabolki', 'jabolk', etc.

I found some references about existence of Slovenian stemmer, but I can't find where to download it, it's not even for sale anywhere...

Another option I've came across is using option wordforms in Sphinx source config (http://sphinxsearch.com/docs/manual-0.9.9.html#conf-wordforms), but building my own dictionary would be too difficult, so I'm wondering are there any publicly accessible dictionaries available already?

In case Slovenian stemmer is not available, can somebody suggest some other approach of achieving similar search results?

3
phpsearchfull-text-searchsphinxstemming
I have found Slovenian stemmer at tartarus snowball mail archive. It is in snowball format, but it seems incomplete. Also I don't know how to use it with Sphinx. – KoviNET

3 Answers

4
votes

I managed to compile slovenian stemmer in following steps:

  1. Download http://snowball.tartarus.org/dist/snowball_code.tgz (source code for snowball) and unpack it
  2. Download slovenian algorithm from http://snowball.tartarus.org/archives/snowball-discuss/0725.html and save it to unpacked project from step 1 in folder /algorithms/slovene. Name of the file has to be stem_ISO_8859_2.sbl
  3. Algorithm is in ISO encoding, so I converted it to UTF8 and saved it as stem_Unicode.sbl (you have to find utf char codes for slovenian special chars like ČŠŽĆ)

  4. Edit both of .txt files in /libstemmer folder and add entries for slovenian:

    slovene         UTF_8,ISO_8859_2        slovene,sl,slv
  5. Edit /GNUmakefile and add slovene (once to list of languages for utf and once for ISO_8859_2_algorithms)

  6. go to folder /libstemmer and run:

    ./mkmodules.pl modules.h src_c modules.txt ../mkinc.mak
./mkmodules.pl modules_utf8.h src_c  modules_utf8.txt ../mkinc_utf8.mak

    This will generate files needed for compiling later.

  7. run make (from root of unpacked files)

  8. If there were no errors during compile you should have /src_c folder and code for slovenian stemmer in them (next to others)

    stem_UTF_8_slovene.c
stem_ISO_8859_2_slovene.c
...

  9. Unpack latest sphinx and copy all files from your snowball project to sphinx /libstemmer_c folder (excluding libstemmer.o and GNUmakefile)

  10. compile sphinx:

    touch NEWS README AUTHORS ChangeLog
autoreconf --force --install
./configure --with-libstemmer
make
make install

  11. if all went fine you should have slovene stemmer for sphinx working, you just have to enable it in you sphinx index configuratiun (on my Debian it is in /usr/local/etc/sphinx.conf):

    charset_type = utf-8
morphology = libstemmer_slovene

Hope this helps someone, I had no prior experience with autoconf so it took me a while to figure this out.

This slovene stemmer is not officially released on http://snowball.tartarus.org, but from my tests it works good enough for my project.

1
votes

I'm not sure if this will do what you want, but I came across this reference to a tool called spelldump in the Sphinx documentation:

spelldump is one of the helper tools within the Sphinx package.

It is used to extract the contents of a dictionary file that uses ispell or MySpell format, which can help build word lists for wordforms - all of the possible forms are pre-built for you.

http://sphinxsearch.com/docs/current.html#ref-spelldump

It requires "a dictionary file that uses ispell or MySpell" - I found a reference to a Slovenian ispell dictionary file, which might be suitable.

0
votes

I was also trying to find a stemmer for the Slovene language but didn't come across any existing solutions.

I've built my own stemmer in Ruby using the never implemented Snowball version as inspiration.

It's available on Github as hajkr/slovene-stemmer. It's far from perfect, but it works for most cases.