search-engine

I've Rails apps, that record an IP-address from every request to specific URL, but in my IP database i've found facebook blok IP like 66.220.15.* and Google IP (i suggest it come from bot). Is there any formula to determine an IP from request was made by a robot or search engine spider ? Thanks Robots are required (by common sense / courtesy more than any kind of law) to send along a User-Agent with their request. You can check for this using request.env["HTTP_USER_AGENT"] and filter as you please. Since the well behaved bots at least typically include a reference URI in the UA string they

I'm looking to find alternatives to Solr from the Apache Software Foundation. For those that don't know, Solr is an enterprise search server. A client application uses a web-services like interface to submit documents for indexing and also to perform search queries. Solr has other features built in like caching and replication. I believe it was originally started by CNet and then open-sourced. I'm looking for other search servers out there that might be seen as the competition. Adam Carden I wrote a long post about my experiences and features of all the engines I listed below but I scrapped it

We have OCRed thousands of pages of newspaper articles. The newspaper, issue, date, page number and OCRed text of each page has been put into a mySQL database. We now want to build a Google-like search engine in PHP to find the pages given a query. It's got to be fast, and take no more than a second for any search. How should we do it? You can also try out SphinxSearch . Craigslist uses sphinx and it can connect to both mysql and postgresql. There are some interesting search engines for you to take a look at. I don't know what you mean by "Google like" so I'm just going to ignore that part.

I've always been interested in developing a web search engine. What's a good place to start? I've heard of Lucene, but I'm not a big Java guy. Any other good resources or open source projects? I understand it's a huge under-taking, but that's part of the appeal. I'm not looking to create the next Google, just something I can use to search a sub-set of sites that I might be interested in. SquareCog There are several parts to a search engine. Broadly speaking, in a hopelessly general manner (folks, feel free to edit if you feel you can add better descriptions, links, etc): The crawler . This is

Google is unfriendly to searching for verbatim strings with characters like $ and #. Is there a search engine that supports searching for verbatim strings? I'm aware of this question that was asked a while ago, but didn't get any good answers. I'm ready to put a bounty if I still can't get an answer. Of course it's entirely possible that there isn't one, but I want to be sure :) EDIT: Quotes don't work . Google strips symbols even when quoted. Example query: make "$<", for the $< variable in makefiles. Try http://www.symbolhound.com . This seems to solve your problem. SymbolHound is a search

I have been asked by our developers to give them an API key for Baidu maps so they can set up our on site store locator and I'm not really sure how to go about doing this. I tried to set up an account on Baidu but it asked for a chinese mobile number. Do I have to get one of these before I can get the key? And how easy is it to work out how to obtain the key once i've got an account? Can anyone advise on the best way to set this up? Thanks in advance! Update 2016: It now appears to be possible to use non-Chinese phone numbers to create Baidu accounts, see this page . Once you're logged into