I have Created a app that crawl the website. here - http://crawler.javaignite.com/
I got all relative URL and having its meta keyword and description including title.
But now i want to store these data in some tables & i don't have idea about how search engine store data and search so help me create basic schema for this.
I have tried to create but need more improvement on this . basically i use only three tables
table : website_mst
website_id
website_url [root]
website_visitperiod // no of days to revisit
table: crawled_history
contains multiple data as how many times it crawled
crawled_id PK
website_id FK [ref. website_mst]
crawled_url
crawled_datetime
table: crawled_page
contains only meta and description
crawled_id FK [ref. crawled_history]
meta_title
meta_keyword
meta_description
total_viewed
updated_on
