cluster-computing

问题 I am using Amazon EC2, and I have my master and development servers as one. And I have another instance for a single worker. I am new to this, but I have managed to make spark work in a standalone mode. Now I am trying cluster. the master and worker are active (I can see the webUI for them and they are functioning). I have Spark 2.0, and I have installed the latest Anaconda 4.1.1 which comes with Python 3.5.2. In both worker and master, if I go to pyspark and do os.version_info, I will get

问题 I'm using a Grid Engine cluster for running some OpenCV code. The code runs well when executed locally, but when submitted to the grid it's not working. I extracted here a minimal example. In the directory ~/code/ I have a file test.cpp containing the following code: #include <opencv2/core.hpp> #include <iterator> #include <string> #include <sys/types.h> #include <sys/stat.h> using namespace cv; using namespace std; int main(int ac, char** av) { /// Create a random matrix Mat M; /// Create a

问题 As per MarkLogic cluster recommendation, we need to configure it as per below link MarkLogic Cluster - Configure Forest with all documents Forest configuration is done as per MarkLogic on Azure Guide Page No. 28 i.e. Azure storage key has been set in Security -> Credentials -> Azure Data directory has been set as azure:// This is working fine and every forest on cluster host has been set in a different container within same azure Blob. Now i want to configure failover cluster by replicating

问题 I've gotten accustomed to doing R jobs on a cluster with 32 cores per node. I am now on a cluster with 16 cores per node. I'd like to maintain (or improve) performance by using more than one node (as I had been doing) at a time. As can be seen from my dummy sell script and dummy function (below), parallelization on a single node is really easy. Is it similarly easy to extend this to multiple nodes? If so, how would I modify my scripts? R script: library(plyr) library(doMC) registerDoMC(16)

问题 When trying to connect to a remote dev cluster using the following sample code: var proxy = ActorProxy.Create<IActor1_NoS>(ActorId.NewId(), "fabric:/applicationname"); I get the following error: System.Fabric.FabricException : The supplied address was invalid Note that this code works fine when ran locally from the dev cluster machine. The Dev cluster manifest file has been modified to listen on the machine IP address. The remote machine is a Windows 7. All Service Fabric assemblies were

问题 java and mpj express are installed in /opt in compute node. JAVA_HOME , MPJ_HOME and PATH are set already via bashrc. error when running mpjboot machines : bash: java: command not found java is working already in both machines mpjboot: #!/bin/sh if [ $# -ne 1 ]; then echo "Usage: mpjboot <machines_file>"; exit 127 fi java -jar $MPJ_HOME/lib/daemonmanager.jar -boot -m "$@" 回答1: which Linux Distribution are you using? Try placing MPJ_HOME and JAVA_HOME at top of .bashrc. It fixes this problem

问题 I am now working on a node.js project based on cluster. I got stuck on the logging. After doing some research, I worked out a solution. here is it. i don't know if it is a good idea. The idea is like this. only master process can wirte to the log file, if the current process is a worker, then it send a log message to the master and then write to the log file while the master can directly write to the log file. this can avoid multiple process open and write to a same file. var util = require(

问题 I'm looking for a distributed message queue that will support millions of queues, with each queue handling tens of messages per second. The messages will be small (tens of bytes), and I don't expect the queues to get very long--on the order of tens of messages per queue at maximum, but when the system is humming along, the queues should stay fairly empty. I'm not sure how many nodes to expect in the cluster--probably depends on the specific solution, but if I had to guess, I would say ten

问题 I've run into a weird problem with /usr/bin/env ... I designed a simple script to show the problem. The script is in Ruby, but the same happens with a similar script in Python. Here is the script: #!/usr/bin/env ruby p ARGV And another one without /usr/bin/env : #!/data/software/ruby-1.9.2-p180/bin/ruby p ARGV As you see it should just print script arguments. An it works flawlessly on the head node: [gusev@scyld test]$ which ruby /data/software/ruby-1.9.2-p180/bin/ruby [gusev@scyld test]$ .

问题 Correct me if I am wrong, but it isn't possible to start multiple http-servers on the same port. Based on this it is interesting the NodeJS cluster may fork. Of cause I know there is the master what is passing the request to one of the forked workers. What worker is managed by operating system or cluster.schedulingPolicy= "rr" for "round robin". The point is: Every worker needs its own memory, so you need x-times much memory where x is the number of workers . But if I like to run different