cluster-computing | 易学教程

How to pass local variable to remote using ssh and bash script?

阅读更多关于 How to pass local variable to remote using ssh and bash script?

问题 ssh remotecluster 'bash -s' << EOF > export TEST="sdfsd" > echo $TEST > EOF This prints nothing. Also it still does not work even if I store the variable into file and copy it to remote. TEST="sdfsdf" echo $TEST > temp.par scp temp.par remotecluster ssh remotecluster 'bash -s' << EOF > export test2=`cat temp.par` > echo $test2 > EOF Still prints nothing. So my question is how to pass local variable to the remote machine as a variable ? Answers have been give in this 回答1: The variable

Using Quartz in a clustered environment

阅读更多关于 Using Quartz in a clustered environment

问题 I'm looking to use the quartz scheduler in my application because I have a clustered environment and want to guarantee that only one instance of my job runs each hour. My question is...Do I have to use a JDBC job store or some sort of "outside" storage of job data to guarantee that only once instance in my cluster runs the job at any given hour or is there more magic to Quartz that I am aware of? 回答1: Yes, you need to use the JDBC-JobStore, or else the TerracottaJobStore to enable a mechanism

Jenkins Master/Slave configuration

阅读更多关于 Jenkins Master/Slave configuration

I've been reading about Jenkins master/slave configurations but I still have some questions: Is it so that the slave Jenkins is not actually installed and started up the way master Jenkins is? I assumed I would install one master Jenkins and another slave Jenkins in the same way, and then master Jenkins would control the slave e.g. through SSH? So I cannot view the slave Jenkins through a GUI? The reason why I have thought about adding a slave Jenkins on another VM is because the VM contains our application servers (many test environments). Deploying and starting/stopping application servers

How to make RabbitMQ scalable?

阅读更多关于 How to make RabbitMQ scalable?

I tried to test RabbitMQ, but I found that rabbitmq has some problems: if I created a cluster of 3 nodes, I can't publish/delivered more than 6000/s. in other hand, if I worked with one single node, I can publish/delivery until 25000/s. which means, more that I add nodes, more performance is deteriorating. but from this article : https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine they can publish more than 1 million, so how they can do that? I want to make RabbitMQ process more than 1 million messages per second I resolved the

Compute dissimilarity matrix for large data

阅读更多关于 Compute dissimilarity matrix for large data

问题 I'm trying to compute a dissimilarity matrix based on a big data frame with both numerical and categorical features. When I run the daisy function from the cluster package I get the error message: Error: cannot allocate vector of size X. In my case X is about 800 GB. Any idea how I can deal with this problem? Additionally it would be also great if someone could help me to run the function in parallel cores. Below you can find the function that computes the dissimilarity matrix on the iris

Jenkins Master/Slave configuration

阅读更多关于 Jenkins Master/Slave configuration

问题 I've been reading about Jenkins master/slave configurations but I still have some questions: Is it so that the slave Jenkins is not actually installed and started up the way master Jenkins is? I assumed I would install one master Jenkins and another slave Jenkins in the same way, and then master Jenkins would control the slave e.g. through SSH? So I cannot view the slave Jenkins through a GUI? The reason why I have thought about adding a slave Jenkins on another VM is because the VM contains

Can MySQL Cluster handle a terabyte database

阅读更多关于 Can MySQL Cluster handle a terabyte database

I have to look into solutions for providing a MySQL database that can handle data volumes in the terabyte range and be highly available (five nines). Each database row is likely to have a timestamp and up to 30 float values. The expected workload is up to 2500 inserts/sec. Queries are likely to be less frequent but could be large (maybe involving 100Gb of data) though probably only involving single tables. I have been looking at MySQL Cluster given that is their HA offering. Due to the volume of data I would need to make use of disk based storage. Realistically I think only the timestamps

Starting remote processes in a Windows network

阅读更多关于 Starting remote processes in a Windows network

I have several slave machines and a master machine which together run a distributed application. Processes on each slave machine have to have a GUI and network access (I think it would be called an interactive process then). For ease of use it would be nice if the master machine could start/stop the processes on those slave machines. My first idea was to use WMI and the Win32_Process class to start a remote process but upon further investigation it was reveiled that processes started this way are non-interactive and isolated, and thus cannot have any GUI. A note says that one can use Win32

How to set up doSNOW and SOCK cluster with Torque/MOAB scheduler?

阅读更多关于 How to set up doSNOW and SOCK cluster with Torque/MOAB scheduler?

问题 In continuation of this question (https://stackoverflow.com/questions/17222942/allow-foreach-workers-to-register-and-distribute-sub-tasks-to-other-workers), what is a best practice to connect doSNOW and SOCK cluster to Torque/MOAB scheduler in order to avoid processor affinity in an inner parallel loop that handles some part of the code of an outer parallel loop? From the Steve's answer to that question, the baseline code without intraction with the scheduler could be: library(doSNOW) hosts <

Node.js multi-server cluster: how to share object in several nodes cluster

阅读更多关于 Node.js multi-server cluster: how to share object in several nodes cluster

I want to create a cluster of node.js server in order to support high concurrency, for a chat rooms application. I need to be able to share information between all nodes. I am trying to find out what would be the best way to keep all the servers in-sync. I want as much flexibility as possible in the shared object, as I plan to add more features in the future. So far, I have 2 solutions in mind: Subscribe to NoSQL key (for example redis publish-subscribe ) Nodes update each other using sockets . Which is better? Any other ideas? Redis is nice because it's independent of your node app and fairly