cluster-computing | 易学教程

AWS ECS Task Memory Hard and Soft Limits

阅读更多关于 AWS ECS Task Memory Hard and Soft Limits

问题 I'm confused about the purpose of having both hard and soft memory limits for ECS task definitions. IIRC the soft limit is how much memory the scheduler reserves on an instance for the task to run, and the hard limit is how much memory a container can use before it is murdered. My issue is that if the ECS scheduler allocates tasks to instances based on the soft limit, you could have a situation where a task that is using memory above the soft limit but below the hard limit could cause the

In a hadoop cluster, should hive be installed on all nodes?

阅读更多关于 In a hadoop cluster, should hive be installed on all nodes?

I am a newbie to Hadoop / Hive and I have just started reading the docs. There are lots of blogs on installing Hadoop in cluster mode. Also, I know that Hive runs on top of Hadoop. My question is: Hadoop is installed on all the cluster nodes. Should I also install Hive on all the cluster nodes or only on the master node? No, it is not something you install on worker nodes. Hive is a Hadoop client. Just run Hive according to the instructions you see at the Hive site . Hive is basically used for processing structured and semi-structured data in Hadoop. We can also perform Analysis of large

Share Sessions between tomcat instances (without using Sticky Sessions)

阅读更多关于 Share Sessions between tomcat instances (without using Sticky Sessions)

问题 I'm going to have 3 Tomcat servers and a Load Balancer that dispatches the requests without using 'sticky sessions'. I want to share sessions' data between the servers and I'm thinking in persisting them in DB. I'd like to use memcached as a layer in front of my DB to serve the requests faster and to don't put my db under heavy load. I'm thinking in providing my customized tomcat Manager that uses memcached before getting/persisting session data to DB as at the moment I don't see a

Java EE Application-scoped variables in a clustered environment (Websphere)?

阅读更多关于 Java EE Application-scoped variables in a clustered environment (Websphere)?

问题 Is there any easy way in a Java EE application (running on Websphere) to share an object in an application-wide scope across the entire cluster? Something maybe similar to Servlet Context parameters, but that is shared across the cluster. For example, in a cluster of servers "A" and "B", if a value is set on server A (key=value), that value should immediately (or nearly so) be available to requests on server B. (Note: Would like to avoid distributed caching solutions if possible. This really

kill all child_process when node process is killed

阅读更多关于 kill all child_process when node process is killed

How do i make sure all child_process are killed when the parent process is killed. I have something like the below one. Even when the node process is kill i see that FFMPEG continues to run and the out.avi is generated. How can i stop FFMPEG from running after the node process exits. var args = "ffmpeg -i in.avi out.avi" child_process.exec(args , function(err, stdout,stderr){}); child_process.exec(args , function(err, stdout,stderr){}); You need to listen for the process exit event and kill the child processes then. This should work for you: var args = "ffmpeg -i in.avi out.avi" var a = child

How to set up autoscaling RabbitMQ Cluster AWS

阅读更多关于 How to set up autoscaling RabbitMQ Cluster AWS

问题 I'm trying to move away from SQS to RabbitMQ for messaging service. I'm looking to build a stable high availability queuing service. For now I'm going with cluster. Current Implementation , I have three EC2 machines with RabbitMQ with management plugin installed in a AMI , and then I explicitly go to each of the machine and add sudo rabbitmqctl join_cluster rabbit@<hostnameOfParentMachine> With HA property set to all and the synchronization works. And a load balancer on top it with a DNS

How to fix symbol lookup error: undefined symbol errors in a cluster environment

阅读更多关于 How to fix symbol lookup error: undefined symbol errors in a cluster environment

问题 I'm working on some python code that extracts some image data from an ECW file using GDAL (http://www.gdal.org/) and its python bindings. GDAL was built from source to have ECW support. The program is run on a cluster server that I ssh into. I have tested the program through the ssh terminal and it runs fine. However, I would now like to submit a job to the cluster using qsub, but it reports the following: Traceback (most recent call last): File "./gdal-test.py", line 5, in <module> from

Apache Helix vs YARN

阅读更多关于 Apache Helix vs YARN

问题 What is the difference between Apache Helix and Hadoop YARN (MRv2). Does anyone have experience with both technologies? Can someone explain me the advantages/disadvantages of Helix over YARN and why the LinkedIn guys developed their own cluster management instead of using YARN? Thanks in advance Tobi 回答1: While Helix and YARN both provide capabilities to manage distributed applications, there are important differences between the two. YARN primarily provides resource management capabilities

Debugging Node.js processes with cluster.fork()

阅读更多关于 Debugging Node.js processes with cluster.fork()

问题 I've got some code that looks very much like the sample in the Cluster documentation at http://nodejs.org/docs/v0.6.0/api/cluster.html, to wit: var cluster = require('cluster'); var server = require('./mycustomserver'); var numCPUs = require('os').cpus().length; if (cluster.isMaster) { var i; // Master process for (i = 0; i < numCPUs; i++) { cluster.fork(); } cluster.on('death', function (worker) { console.log('Worker ' + worker.pid + ' died'); }); } else { // Worker process server

Running slurm script with multiple nodes, launch job steps with 1 task

阅读更多关于 Running slurm script with multiple nodes, launch job steps with 1 task

I am trying to launch a large number of job steps using a batch script. The different steps can be completely different programs and do need exactly one CPU each. First I tried doing this using the --multi-prog argument to srun . Unfortunately, when using all CPUs assigned to my job in this manner, performance degrades massively. The run time increases to almost its serialized value. By undersubscribing I could ameliorate this a little. I couldn't find anything online regarding this problem, so I assumed it to be a configuration problem of the cluster I am using. So I tried going a different