cluster-computing

I'm working on scaling out my app to multiple servers, and one requirement is that a client is always communicating with the same server (too much live data is used to allow bouncing between servers efficiently). My current setup is a small server cluster (using Linode). I have a frontend node running HAProxy using "balance source" so that an IP is always pointed towards the same node. I'm noticing that "balance source" is not a very even distribution. With my current test setup (2 backend servers), one server often has 3-4x as many connections when using a sample size of 80-100 source IPs. Is

What is the best strategy to refactor a Singleton object to a cluster environment? We use Singleton to cache some custom information from Database. Its mostly read-only but gets refreshed when some particular event occurs. Now our application needs to be deployed in a Clustered environment. By definition, each JVM will have its own Singleton instance. So the cache may be out-of-sync between the JVM's when a refresh event occurs on a single node and its cache is refreshed. What is the best way to keep the cache's in sync? Thanks. Edit: The cache is mainly used to provide an autocomplete list

I have working (stock) script from node var cluster = require('cluster'); var http = require('http'); var numReqs = 0; if (cluster.isMaster) { // Fork workers. for (var i = 0; i < 2; i++) { var worker = cluster.fork(); worker.on('message', function(msg) { if (msg.cmd && msg.cmd == 'notifyRequest') { numReqs++; } }); } setInterval(function() { console.log("numReqs =", numReqs); }, 1000); } else { // Worker processes have a http server. http.Server(function(req, res) { res.writeHead(200); res.end("hello world\n"); // Send message to master process process.send({ cmd: 'notifyRequest' }); })

In many functions from scikit-learn implemented user-friendly parallelization. For example in sklearn.cross_validation.cross_val_score you just pass desired number of computational jobs in n_jobs argument. And for PC with multi-core processor it will work very nice. But if I want use such option in high performance cluster (with installed OpenMPI package and using SLURM for resource management) ? As I know sklearn uses joblib for parallelization, which uses multiprocessing . And, as I know (from this, for example, Python multiprocessing within mpi ) Python programs parallelized with

I am looking for a set of instructions to create a local multi-node Cassandra 2.x cluster on a Window 7 64 bit PC. It should preferably use CCM “Cassandra Cluster Manager” and allow management using DataStax OpsCenter I followed the instructions in “ Getting Started with Apache Cassandra on Windows the Easy Way ” but they are for a single node cluster. EDIT: I got stuck on deploying OpsCenter agents on each node using CCM, any ideas? Articles used for this tutorial: CCM 2.0 and Windows Cassandra Wiki - Windows Development Setting up a multi-node Cassandra cluster on a single Windows machine

I am configuring an Apache Spark cluster. When I run the cluster with 1 master and 3 slaves, I see this on the master monitor page: Memory 2.0 GB (512.0 MB Used) 2.0 GB (512.0 MB Used) 6.0 GB (512.0 MB Used) I want to increase the used memory for the workers but I could not find the right config for this. I have changed spark-env.sh as below: export SPARK_WORKER_MEMORY=6g export SPARK_MEM=6g export SPARK_DAEMON_MEMORY=6g export SPARK_JAVA_OPTS="-Dspark.executor.memory=6g" export JAVA_OPTS="-Xms6G -Xmx6G" But the used memory is still the same. What should I do to change used memory? When using