distributed

How is Git Distributed Source Code Management?

送分小仙女□ 提交于 2019-12-05 05:16:33
I am a Git newbee with UNIX SCCS and Microsoft Visual SourceSafe experience. I’m just learning Git and it seems to have a huge and painful learning curve. I’ve already seen Git blow away all the data files I hadn’t committed, which concerns me. (How a utility can delete data files without warning is beyond me). Linus Torvalds, in his video on Git, claims that Git is distributed, touts the benefits of distribution, but other than everyone having a copy (clone) of the source, doesn’t really explain how distribution works. How does Git help distribution? How does Git help recover lost files? How

Is there an off-the-shelf clock synchronization solution for Java?

折月煮酒 提交于 2019-12-05 04:01:38
We have a large high-performance software system which consists of multiple interacting Java processes (not EJBs). Each process can be on the same machine or on a different machine. Certain events are generated in one process, and are then propagated in different ways to other processes for further processing and so on. For benchmarking purposes, we need to create a log of when each event passed through a "checkpoint", eventually combine these logs to obtain a timeline of how each event propagated through the system and with what latency (of course, process switching and IPC adds latency,

When to use the XCode Distributed Build Feature

99封情书 提交于 2019-12-05 02:48:19
I work in a small iPhone development team, in our office, we have at least 4 copies of XCode running on the network at any one time. Contemplating getting everyone to have it running. We're networked together using a standard WIFI Switch, so network speed and latency isn't as good as wired network... Just wondering, is there any real time gain to be had on using distributed builds? Once it passes the relevant data back and forth over the network. At least for relatively small projects. it depends on your project, its dependencies, and the amount of data that must be transferred. 15-20 seconds

GRPC causes training to pause in individual worker (distributed tensorflow, synchronised)

陌路散爱 提交于 2019-12-04 22:48:58
I am trying to train model in synchronous distributed fashion for data parallelism. There are 4 gpus in my machine. Each gpu should should run a worker to train on separate non-overlapping subset of the data (between graph replication). The main data file is separated into 16 smaller TFRecord files. Each worker is supposed to process 4 different files. The problem is that training freezes independently and at different times in each worker process. They freeze at some point. One of the 'ps' reports following error related to grpc: 2017-09-21 16:45:55.606842: I tensorflow/core/distributed

What is the right way to do model parallelism in tensorflow?

半城伤御伤魂 提交于 2019-12-04 18:51:50
I have multiple 4GB GPU nodes so I want them to run huge model in parallel. I hope just splitting layers into several pieces with appropriate device scopes just enables model parallelism but it turns out that it doesn't reduce memory footprint for master node(task 0). (10 nodes configuration - master: 20g, followers:2g, 1 node configuration - master: 6~7g) Suspicious one is that gradients are not distributed because I didn't setup right device scope for them. my model is available on github.( https://github.com/nakosung/tensorflow-wavenet/tree/model_parallel_2 ) device placement log is here:

Parallel debuggers

强颜欢笑 提交于 2019-12-04 18:09:18
问题 I am trying to decide which parallel debugger to use. So far I found not many open sources ones so my choices are: https://www.arm.com/products/development-tools/server-and-hpc/forge/ddt http://www.roguewave.com/products/totalview-family/totalview.aspx Which one do you recommend? Are there anything else worthwhile? 回答1: A colleague wrote a few years ago a short technical report comparing the two. Moral of the story: they're comparable, and the fact that there's competition in the market has

Solr/Lucene分布式搜索,Solr Integrate katta step3

自闭症网瘾萝莉.ら 提交于 2019-12-04 16:26:46
前面的两篇介绍了安装katta及ZooKeeper,后边来介绍katta的Node. 我们回到step1 后边提到的solr-katta-plugin项目,源码导入后会出现很多的错误,在项目中继承了solr-core,和solrj中的类尝试着把访问修饰private改为protected. 如:solr-core org.apache.solr.handler.component.SearchHandler类中的shardHandlerFactory成员变量 protected ShardHandlerFactory shardHandlerFactory = new HttpShardHandlerFactory(); 同时借鉴 https://issues.apache.org/jira/browse/SOLR-1395 Tomliu的做法,把 the bugs is : 1. solr's ShardDoc.java, ShardFieldSortedHitQueue line 210 : final float f1 = e1.score == null ? 0.00f : e1.score; final float f2 = e2.score == null ? 0.00f : e2.score; 等等.直到项目基本错误解决. 下载solr并且copy其中的apache

Erlang: How to view output of io:format/2 calls in processes spawned on remote nodes

牧云@^-^@ 提交于 2019-12-04 13:24:28
问题 I am working on a decentralized Erlang application. I am currently working on a single PC and creating multiple nodes by initializing erl with the -sname flag. When I spawn a process using spawn/4 on its home node, I can see output generated by calls io:format/2 within that process in its home erl instance. When I spawn a process remotely by using spawn/4 in combination with register_name , output of io:format/2 is sometimes redirected back to the erl instance where the remote spawn/4 call

Java RMI : What is the role of the stub-skeleton that are generated by the rmic compiler

拈花ヽ惹草 提交于 2019-12-04 13:19:03
问题 I am currently learning Java RMI (Remote Method Invocation), and I followed the tutorial provided by Oracle on it´s website. I have a particular question however: What is the use of the stub-skeleton generated by rmic? Do I really need it? 回答1: The Stub/Skeleton hides the communication details away from the developer. The Stub is the class that implements the remote interface. It serves as a client-side placeholder for the remote object. The stub communicates with the server-side skeleton.

How does raft handle committing entries from previous one?

北战南征 提交于 2019-12-04 12:06:15
问题 In raft paper section 5.4.2 If a leader crashes before committing an entry, future leaders will attempt to finish replicating the entry. However, a leader cannot immediately conclude that an entry from a previous term is committed once it is stored on a majority of servers. There could be a situation where an old log entry is stored on a majority of servers, yet can still be overwritten by a future leader. The author mentioned that to avoid the situation above To eliminate problems like the