How speed up program? (Lot of slow mysql queries)

Question

I have database with 300 000 rows, and I need filter some rows by algorithm.

protected boolean validateMatch(DbMatch m) throws MatchException, NotSupportedSportException{

    // expensive part
    List<DbMatch> hh = sd.getMatches(DateService.beforeDay(m.getStart()), m.getHt(), m.getCountry(),m.getSportID());
    List<DbMatch> ah = sd.getMatches(DateService.beforeDay(m.getStart()), m.getAt(), m.getCountry(),m.getSportID());
    ....

My hibernate dao function for load data from Mysql is used 2x times of init array size.

public List<DbMatch> getMatches(Date before,String team, String country,int sportID) throws NotSupportedSportException{
    //Match_soccer where date between :start and :end
    Criteria criteria = session.createCriteria(DbMatch.class);
    criteria.add(Restrictions.le("start",before));
    criteria.add(Restrictions.disjunction()
            .add(Restrictions.eq("ht", team))
            .add(Restrictions.eq("at", team)));

    criteria.add(Restrictions.eq("country",country));
    criteria.add(Restrictions.eq("sportID",sportID));
    criteria.addOrder(Order.desc("start") );
    return criteria.list();
}

Example how i try filter data

function List<DbMatch> filter(List<DbMatch> mSet){
   List<DbMatch> filtred = new ArrayList<>();
   for(DbMatch m:mSet){
      if(validateMatch(DbMatch m))filtred.add(m);
   }
}

(1)I tried different criteria settings and counted function times with stopwatch. My result is when I use filter(matches) matches size 1000 my program take 3 min 21 s 659 ms.

(2)I tried remove criteria.addOrder(Order.desc("start")); than program filtered after 3 min 12 s 811 ms.

(3)But if I remove criteria.addOrder(Order.desc("start")); and add criteria.setMaxResults(1); result was 22 s 311 ms.

Using last configs i can filter all my 300 000 record by 22,3 * 300 = 22300 s (~ 6h), but if use first function I should wait (~ 60 h).

If I want use criteria without order and limit i must be sure that my table is sorted by date on database because it is important get last match .

All data is stored on matches table.

Table indexes:

Table, Non_unique, Key_name, Seq_in_index, Column_name, Collation, Cardinality, Sub_part, Packed, Null, Index_type, Comment, Index_comment

matches, 0, PRIMARY, 1, mid, A, 220712, , , , BTREE, ,
matches, 0, UK_kcenwf4m58fssuccpknl1v25v, 1, beid, A, 220712, , , YES, BTREE, ,

UPDATED

After added ALTER TABLE matches ADD INDEX (sportID, country); now program time deacrised to 15s for 1000 matches. But if I not use order by and add limit need wait only 4s for 1000 mathces.

How I should act on this situation to improve program executions speed?

Sorry @npinti, but I not understand you question very weel :( — Edgaras Karka
Take a look here. It might be that you would need to make the optimization at a database level. — npinti
Make SHOW INDEX table_name in mysql prompt where table_name is the table from which you load data. — Mejmo

na-98 na-98 · Accepted Answer · 2015-10-15T10:36:57

Your first order of business is to figure out how long each component take to process the request.

Find out the SQL query generated by the ORM and run that manually in MySQL workbench and see how long it takes (non cached). You can also ask for it to explain the index usage.

If it's fast enough then it's your java code that's taking longer and you need to optimize your algorithm. You can use JConsole to dig further into that.

If you identify which component is taking longer you can post here with your analysis and we can make suggestions accordingly.

How speed up program? (Lot of slow mysql queries)

1 Answers