scalability

I am designing an application, one aspect of which is that it is supposed to be able to receive massive amounts of data into SQL database. I designed the database stricture as a single table with bigint identity, something like this one: CREATE TABLE MainTable ( _id bigint IDENTITY(1,1) NOT NULL PRIMARY KEY CLUSTERED, field1, field2, ... ) I will omit how am I intending to perform queries, since it is irrelevant to the question I have. I have written a prototype, which inserts data into this table using SqlBulkCopy. It seemed to work very well in the lab. I was able to insert tens of millions

After finishing school in Computer Science and entering the "real world" as a software engineer working on web applications, I've become overwhelmed by the amount of information to be learned about scaling web applications properly. Some topics/questions that have recently popped up for me: RDBMS's vs. unstructured data storage. Advantages and disadvantages of sharding for databases, search indexes, etc. Which network file systems scale? Which don't? The cost of HTTP and DB connections. Static content delivery, why not to store images in a database, etc. Why is it better to use one thread pool

As a programmer I make revolutionary findings every few years. I'm either ahead of the curve, or behind it by about π in the phase. One hard lesson I learned was that scaling OUT is not always better, quite often the biggest performance gains are when we regrouped and scaled up. What reasons to you have for scaling out vs. up? Price, performance, vision, projected usage? If so, how did this work for you? We once scaled out to several hundred nodes that would serialize and cache necessary data out to each node and run maths processes on the records. Many, many billions of records needed to be