Single index multi type - elasticsearch indexing via tire

Question

In my multi-tenant app (account based with number of users per account), how would I update index for a particular account when a user document is changed.

I have a separate index for each account, in which the mappings for each model (user and comments - just an example actual app has many models) are specified. In this case if any change has been done for user model or comment model, the index that has been created for the related account has to be updated. Is this possible? Please let me know if yes.

I guess this is the way I specify the mappings in my case. Correct me if I'm wrong.

Account Model:

include Tire::Model::Search

Tire.index('account_1') do
  create(
    :mappings => {
      :user => {
        :properties => {
          :name => { :type => :string, :boost => 10 },
          :company_name => { :type => :string, :boost => 5 }
        }
      },
      :comments => {
        :properties => {
          :description => { :type => :string, :boost => 5 }
        }
      }
    }
  )
end

The index is getting created correctly with both the mappings for account index. But, I don't see a way where I can update the index when any model specified in the mappings are changed.

Whenever a new user is added or if an user is updated the index created for the corresponding account has to be updated.

This has been debated over and over at Github and Stackoverflow... — karmi
Yes. That's true. But, I'm still looking for a solution. The call to update_index method as suggested by @karmi seems to work if I'm updating the data in the same model. i.e., Create an index for user model and make a call to update_index using after_save hook. Then the index is getting updated. But, I would like to create an index for each account with multi-type mapping (user and comments) and would like to update the account index on update of user or comment models. — Vamsi Krishna

karmi karmi · Accepted Answer · 2012-12-12T20:54:02

This question is cross-posted from Github issue Multiple model single index approach. Crossposting the answer here.

Let's say we have an Account class and we deal in articles entities.

In that case, our Account class would have following:

class Account
  #...

  # Set index name based on account ID
  #
  def articles
      Article.index_name "articles-#{self.id}"
      Article
  end
end

So, whenever we need to access articles for a particular account, either for searching or for indexing, we can simply do:

@account = Account.find( remember_token_or_something_like_that )

# Instead of `Article.search(...)`:
@account.articles.search { query { string 'something interesting' } }

# Instead of `Article.create(...)`:
@account.articles.create id: 'abc123', title: 'Another interesting article!', ...

Having a separate index per user/account works perfect in certain cases -- but definitely not well in cases where you'd have tens or hundreds of thousands of indices (or more). Having index aliases, with properly set up filters and routing, would perform much better in this case. We would slice the data not based on the tenant identity, but based on time.

Let's have a look at a second scenario, starting with a heavily simplified curl http://localhost:9200/_aliases?pretty output:

{
  "articles_2012-07-02" : {
    "aliases" : {
      "articles_plan_pro" : {
      }
    }
  },
  "articles_2012-07-09" : {
    "aliases" : {
      "articles_current" : {
      },
      "articles_shared" : {
      },
      "articles_plan_basic" : {
      },
      "articles_plan_pro" : {
      }
    }
  },
  "articles_2012-07-16" : {
    "aliases" : {
    }
  }
}

You can see that we have three indices, one per week. You can see there are two similar aliases: articles_plan_pro and articles_plan_basic -- obviously, accounts with the “pro” subscription can search two weeks back, but accounts with the “basic” subscription can search only this week.

Notice also, that the the articles_current alias points to, ehm, current week (I'm writing this on Thu 2012-07-12). The index for next week is just there, laying and waiting -- when the time comes, a background job (cron, Resque worker, custom script, ...) will update the aliases. There's a nifty example with aliases in “sliding window” scenario in the Tire integration test suite.

Let's not look on the articles_shared alias right now, let's look at what tricks we can play with this setup:

class Account
  # ...

  # Set index name based on account subscription
  #
  def articles
    if plan_code = self.subscription && self.subscription.plan_code
      Article.index_name "articles_plan_#{plan_code}"
    else
      Article.index_name "articles_shared"
    end
    return Article
  end
end

Again, we're setting up an index_name for the Article class, which holds our documents. When the current account has a valid subscription, we get the plan_code out of the subscription, and direct searches for this account into relevant index: “basic” or “pro”.

If the account has no subscription -- he's probably a “visitor” type -- , we direct the searches to the articles_shared alias. Using the interface is as simple as previously, eg. in ArticlesController:

@account  = Account.find( remember_token_or_something_like_that )
@articles = @account.articles.search { query { ... } }
# ...

We are not using the Article class as a gateway for indexing in this case; we have a separate indexing component, a Sinatra application serving as a light proxy to elasticsearch Bulk API, providing HTTP authentication, document validation (enforcing rules such as required properties or dates passed as UTC), and uses the bare Tire::Index#import and Tire::Index#store APIs.

These APIs talk to the articles_currentindex alias, which is periodically updated to the current week with said background process. In this way, we have decoupled all the logic for setting up index names in separate components of the application, so we don't need access to the Article or Account classes in the indexing proxy (it runs on a separate server), or any component of the application. Whichever component is indexing, indexes against articles_current alias; whichever component is searching, searches against whatever alias or index makes sense for the particular component.

Single index multi type - elasticsearch indexing via tire

2 Answers