We currently are not offering Customer Support on our Forum.
Please use our Store website for: Customer Contact.
 
 

Author Topic: New High Performance Forum Search Engine!  (Read 3335 times)

0 Members and 1 Guest are viewing this topic.

BigMike

  • Administrator
  • Offline Gold Turtle Award
  • *
  • Turtle Points: 2124
  • Male Posts: 18,289
  • Member since Apr '02
  • 511:1 Club
    • View Profile
    • Bone-Stock Plane-Jane 1981 Shortbed Pickup
New High Performance Forum Search Engine!
« on: Mar 16, 2009, 10:54:53 AM »
Hey guys

For quite some time now I've been longing to install a non MySQL search engine on the forum and I finally got it done!!

This forum's search feature is now powered by the ultra efficient Sphinx Search Engine :woohoo: :party: X 1,000!!

This Sphinx search engine runs as a separate deamon process and provides query results to the forum. A scheduled task retrieves the data from the database every hour and rebuilds the indexes. This engine is fulltext specialized and returns results often a thousand times faster than MySQL.

In the past the forum has been powered by the built-in MySQL query feature, but now, not only is the Sphinx about 1,000 times faster, ADDITIONALLY there is the following advantage that outweighs all:

When a user enters a search into the forum, MySQL begins opening and closing tables in the database while it searches and compiles results for you.

The problem with searching via MySQL is that whenever it is inside a table searching for content, if content inside that particular table suddenly changes midway through the search, it would return inaccurate or incorrect results. To prevent this from happening, MySQL "locks down" each table it is inside of while it is being searched...

...This is bad for any and everyone viewing any page anywhere on this forum because you have to effectively get in line or wait for tables to become available or "unlocked" before your internet browser can even start receiving the data to load the page.

So! The GREAT news is that the Sphinx search engine is out side of MySQL and does its own searching and indexing, so ZERO tables are locked down :clap: :clap2:

So I hope you will notice not only a drastic speed increase while searching but also an overall speed increase on any page anywhere through out the MCI forum :turtle:

Regards,
BigMike
« Last Edit: Mar 16, 2009, 11:29:11 AM by BigMike »
Check out our new Rock Crawling Videos!
2016 56-speed 580:1 Tacoma Rock Crawler   
1981 36-speed 511:1 3RZ-FE Rock Crawler
1987 6-speed Supercharged 4A-GZE MR2
Instagram: @SlowestTacoma
Things are only impossible until they are not.
"The worst of both worlds, the best of neither." -abnormaltoy
"An informed question. But difficult to answer. I am what you see." -Nanaki

BigMike [OP]

  • Administrator
  • Offline Gold Turtle Award
  • *
  • Turtle Points: 2124
  • Male Posts: 18,289
  • Member since Apr '02
  • 511:1 Club
    • View Profile
    • Bone-Stock Plane-Jane 1981 Shortbed Pickup
Re: New High Performance Forum Search Engine!
« Reply #1 on: Mar 16, 2009, 11:00:24 AM »
I forgot to mention that the only downside to this is that the index of search terms for the Sphinx engine is updated once every 60 mins.

Therefore, searches will only return results that are older than 1 hour.

BigMike
Check out our new Rock Crawling Videos!
2016 56-speed 580:1 Tacoma Rock Crawler   
1981 36-speed 511:1 3RZ-FE Rock Crawler
1987 6-speed Supercharged 4A-GZE MR2
Instagram: @SlowestTacoma
Things are only impossible until they are not.
"The worst of both worlds, the best of neither." -abnormaltoy
"An informed question. But difficult to answer. I am what you see." -Nanaki

*FFC*

  • Platinum Turtle Award
  • Internet Lassies
  • Offline Gold Turtle Award
  • *
  • Turtle Points: 11
  • Female Posts: 8,265
  • Member since May '04
  • ~*Moderator*~
    • View Profile
    • Buy me some coffee
Re: New High Performance Forum Search Engine!
« Reply #2 on: Mar 16, 2009, 11:05:57 AM »
:clap2: Way to go Mike!  :beerchug:
"Yesterday is history, tomorrow is a mystery, today is a gift - that's why they call it the present." - RW Emerson -

BigMike [OP]

  • Administrator
  • Offline Gold Turtle Award
  • *
  • Turtle Points: 2124
  • Male Posts: 18,289
  • Member since Apr '02
  • 511:1 Club
    • View Profile
    • Bone-Stock Plane-Jane 1981 Shortbed Pickup
Re: New High Performance Forum Search Engine!
« Reply #3 on: Oct 16, 2011, 07:09:48 PM »
Update on this guys :woohoo:

We ran this uberfast Sphinx search engine for only a short time, and then it stopped working taking out with it the original search method. Without going into detail, and getting to the point, I finally decided to repair and bring back the Sphinx engine which I just did moments ago. :beerchug:

So the search is back to its former glory and is remarkably fast! You may go ahead and start using it immediately :thumbs:

I forgot to mention that the only downside to this is that the index of search terms for the Sphinx engine is updated once every 60 mins.
Update to this, I am now using a merge instead of a replace method to update the search index. This means that the index can be updated with a fraction of server resources compared to before. I have therefore configured the index to be updated every 5 minutes :o :thumbs:

Enjoy!
BigMike
Check out our new Rock Crawling Videos!
2016 56-speed 580:1 Tacoma Rock Crawler   
1981 36-speed 511:1 3RZ-FE Rock Crawler
1987 6-speed Supercharged 4A-GZE MR2
Instagram: @SlowestTacoma
Things are only impossible until they are not.
"The worst of both worlds, the best of neither." -abnormaltoy
"An informed question. But difficult to answer. I am what you see." -Nanaki

GNasty 82

  • Offline Rock Master
  • ***
  • Turtle Points: 994
  • Male Posts: 206
  • Member since Nov '09
  • Get Nasty!!!
    • View Profile
Re: New High Performance Forum Search Engine!
« Reply #4 on: Oct 16, 2011, 09:32:18 PM »
 :thumbs: :thumbs: :thumbs: :thumbs: :thumbs:
82 Toyota SR5 Pickup
  Marlin Dual Ultimate
  Front 5.29 LC HP Elocker
  Rear 5.29 Grizzly Locker
  BFG 37" KM2s
  Custom rear frame and flatbed
  Propane Powered.

Build Thread:
http://board.marlincrawler.com/index.php?topic=64521.0

"The two enemies of the people are criminals and government, so let us tie the second down with the chains of the Constitution so the second will not become the legalized version of the first."-Thomas Jefferson

*** YELLER ***

  • BIGMIKE SAID I WAS A CLOWN SO .... GUESS THATS THAT
  • Offline Gold Turtle Award
  • *
  • Turtle Points: 314
  • Male Posts: 4,686
  • Member since Oct '05
  • USMC NO BETTER FRIEND NO WORSE ENEMY
    • View Profile
    • Buy me a soda
Re: New High Performance Forum Search Engine!
« Reply #5 on: Oct 20, 2011, 10:56:45 AM »
i dont get out much :rofl2:
MY ONLY REGRET IS THAT I HAVE NOTHING USEFUL TO OFFER THIS FORUM  :moon:
except BACON

 
 
 
 
 

Related Topics

1 Replies
1269 Views
Last post Dec 07, 2005, 05:29:35 PM
by crawlerdan
0 Replies
801 Views
Last post Mar 13, 2007, 05:34:51 PM
by Siringo
0 Replies
1568 Views
Last post Jun 01, 2008, 07:10:25 PM
by topkicktech
10 Replies
3238 Views
Last post Oct 28, 2008, 08:17:57 PM
by blackdiamond
1 Replies
2469 Views
Last post Oct 26, 2011, 03:44:37 PM
by CrawlerChick