cluster-computing

Wait for all jobs of a user to finish before submitting subsequent jobs to a PBS cluster

由 谁说胖子不能爱 提交于 2019-12-06 02:44:48
问题 I am trying to adjust some bash scripts to make them run on a (pbs) cluster. The individual tasks are performed by several script thats are started by a main script. So far this main scripts starts multiple scripts in background (by appending & ) making them run in parallel on one multi core machine. I want to substitute these calls by qsub s to distribute load accross the cluster nodes. However, some jobs depend on others to be finished before they can start. So far, this was achieved by

Using cluster in a Node module

由 被刻印的时光 ゝ 提交于 2019-12-06 01:12:40
问题 UPDATE: Even if this particular scenario is not realistic, as per comments, I'm still interested in how one could write a module that makes use of clustering without rerunning the parent process each time. I'm trying to write a Node.js module called mass-request that speeds up large numbers of HTTP requests by distributing them to child processes. My hope is that, on the outside, it work like this. var mr = require("mass-request"), scraper = mr(); for (var i = 0; i < my_urls_to_visit.length;

Installing Rmpi on LAM/MPI cluster

由 a 夏天 提交于 2019-12-06 00:46:34
问题 I'm trying to install Rmpi package on a LAM MPI cluster machine. Previously I had been compiling and testing some stuff ( mpi4py and small C++ programs) so I'm sure the MPI itself works. However installing Rmpi package fails when linking libraries. My main suspect is a call to gcc instead of mpicc in makefile (I'm trying to find the line in configuration to change this but so far could not locate it). Does somebody have experience with installing Rmpi on LAM, and how did you manage that?

Whirr: Cannot connect to Hadoop cluster on EC2 after lauch-cluster

由 放肆的年华 提交于 2019-12-06 00:29:50
I am new to Whirr and I'm trying to setup a Hadoop cluster on EC2 with Whirr,I have followed the tutorial on Cloudera https://ccp.cloudera.com/display/CDHDOC/Whirr+Installation Before install Whirr, I install Hadoop (0.20.2-cdh3u3), then install Whirr (0.5.0-cdh3u3). Here's my cluster config file whirr.cluster-name=large-cluster whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker whirr.provider=aws-ec2 whirr.identity=XXXXXXXXXXXXXXX whirr.credential=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx whirr.private-key-file=${sys:user.home}/.ssh/id_rsa whirr

Failover support for a DB

由 只谈情不闲聊 提交于 2019-12-06 00:12:08
We are currently evaluating failover support in different databases. We were earlier using HSQLDB but it seems that it does not have clustering/replication support. Our requirement is simply to have two database servers, one being only for synchronous backup but if the primary server is down, then the secondary should automatically start acting as the primary server. Has anyone evaluated MySQL, PostgreSQL or any other DB server for such a use case? Edit: We had thought of using MySQL cluster but it now seems that it is under GPL license which we won't be able to work with. Could anyone please

Hadoop+HBase cluster on windows: winutils not found

由 孤街浪徒 提交于 2019-12-05 21:56:17
I'm trying to set up a fully-distributed 4-node dev cluster with Hadoop 2.20 and HBase 0.98 on Windows. I've built Hadoop on Windows successfully, and more recently, also build HBase on Windows. We have successfully ran the wordcount example from the Hadoop installation guide, as well as a custom WebHDFS job. As HBase fully-distributed on Windows isn't supported yet, I'm running HBase under cygwin. When trying to start hbase from my master (./bin/start-hbase.sh), I get the following error: 2014-04-17 16:22:08,599 ERROR [main] util.Shell: Failed to locate the winutils binary in the hadoop

Multi-node Hadoop cluster with Docker

由 ♀尐吖头ヾ 提交于 2019-12-05 20:50:07
问题 I am in planning phase of a multi-node Hadoop cluster in a Docker based environment. So it should be based on a lightweight easy to use virtualized system. Current architecture (regarding to documentation) contains 1 master and 3 slave nodes. This host machine uses HDFS filesystem and KVM for virtualization. The whole cloud is managed by Cloudera Manager . There are several Hadoop modules installed on this cluster. There is also a NodeJS data upload service. This time I should make

How to connect embedded solr with each other by sharding

由 一曲冷凌霜 提交于 2019-12-05 20:26:23
I have been using sharding with multiple basic solr server for clustering. I also used one embedded solr server (Solrj Java API) with many basic solr servers and connecting them by sharding as embedded solr server is the caller of them. I used the code line below for this purpose. SolrQuery query = new SolrQuery(); query.set("shards", "solr1URL,solr2URL,..."); Now, I have many embedded solr servers running on different computers and they are unaware of each others. I want to communicate them with each other by sharding. Is it possible? if yes how? if not what are the other options that you can

Socket.io, cluster, express and sync events

由 爱⌒轻易说出口 提交于 2019-12-05 19:55:52
问题 I have a big problem sice 1 week. I try to convert my node.JS project actually run on single core to multi core with cluster. With websockets, at this moment, i have no problems for events but, for xhr-polling or jsonp-polling, i have big problems with socket.io on cluster mode. this is my server configuration : 00-generic.js 'use strict'; var http = require('http'), os = require('os'), cluster = require('cluster'); module.exports = function(done) { var app = this.express, port = process.env

JBoss 4.2.2 nodes start to cluster then suspect each other

由 北慕城南 提交于 2019-12-05 19:36:48
问题 I have a website running with JBoss 4.2.2 on an existing Red Hat server. I'm setting up a second server so as to have a clustered pair (which will then be load-balanced). However, I can't get them to cluster successfully. The existing server starts up JBoss with: run.sh -c default -b 0.0.0.0 (I know the 'default' configuration doesn't support clustering out of the box - I'm using a modified version of it which includes clustering support.) When I start the second JBoss instance with the same
订阅 cluster-computing