scalability

I'm using MongoDB to handle timeseries, this is working fine as until now there is not too many data but I now need to identify what is needed to scale to a larger number of data. Today, there are +200k data received per day, each data received every couple of seconds, that is not huge but this should increase soon. The data collection used is far from beeing efficient as each piece of data (parentID, timestamp, value) creates a document. I've seen several approaches that uses a document that keeps the timeseries for a whole hour (with, for instance, an inner array that keeps data for each

In my latest project, I have to write a web service API to return data from a database. Techs are - SQL Server 08 R2 - WCF The data is mainly key-value pairs, e.g. for get user X's favourite colour. The load isn't massive, but not small-fry either - about 1,000 requests / sec at peak. My initial thoughts are to use Redis as the cache, meaning we don't hit SQL Server as often. However I've been trying to get some benchmarks etc out of this configuration, and it's not great, making me wonder if Redis will really offer any benefit for my problem! Architecture is - separate db server - WCF App

I want to build a highly scalable application where I intend to use Lucene as my search engine library. While browsing through the docs and faqs, I realize that it only allows one index writer to be open on a storage location by creating some write.lock in index directory. We can open multiple IndexReaders on that index. I am interested in building an architecture where there are number of indexers running on different machines/servers and multiple searcher answering various types of queries on the indexes created by these indexers. Both searchers and indexers will be running on different

Firebase has an interesting feature/nuisance where when you listen on a data ref, you get all the data that was ever added to that ref. So, for example, when you listen on 'child_added', you get a replay of all the children that were added to that ref from the beginning of time. We are writing a commenting system with a dataset that looks something like this: /comments /sites /sites/articles /users Sites have many articles and articles have many comments and users have many comments. We want to be able to track all the comments a user makes, so we feel it is wise to put comments in a separate

Below is a apache bench run for 10K requests with 50 concurrent threads. I need help understanding the results, does anything stand out in the results that might be pointing to something blocking and restricting more requests per second? I'm looking at the connection time section, and see 'waiting' and 'processing'. It shows the mean time for waiting is 208, and the mean time to connect is 0 and processing is 208..yet the total is 208. Can someone explain this to me as it doesn't make much sense to me. Connect time is time it took ab to establish connection with your server. you are probably