mesos

Spark集群三种部署模式的区别

由 社会主义新天地 提交于 2019-12-15 02:42:53
目前Apache Spark支持三种分布式部署方式，分别是standalone、spark on mesos和 spark on YARN，其中，第一种类似于MapReduce 1.0所采用的模式，内部实现了容错性和资源管理，后两种则是未来发展的趋势，部分容错性和资源管理交由统一的资源管理系统完成：让Spark运行在一个通用的资源管理系统之上，这样可以与其他计算框架，比如MapReduce，公用一个集群资源，最大的好处是降低运维成本和提高资源利用率（资源按需分配）。本文将介绍这三种部署方式，并比较其优缺点。 Standalone模式 即独立模式，自带完整的服务，可单独部署到一个集群中，无需依赖任何其他资源管理系统。从一定程度上说，该模式是其他两种的基础。借鉴Spark开发模式，我们可以得到一种开发新型计算框架的一般思路：先设计出它的standalone模式，为了快速开发，起初不需要考虑服务（比如master/slave）的容错性，之后再开发相应的wrapper，将stanlone模式下的服务原封不动的部署到资源管理系统yarn或者mesos上，由资源管理系统负责服务本身的容错。目前Spark在standalone模式下是没有任何单点故障问题的，这是借助zookeeper实现的，思想类似于Hbase master单点故障解决方案。将Spark

Github上的几个C++开源项目

由 可紊 提交于 2019-12-14 03:29:47
Github上的几个C++开源项目 http://blog.csdn.net/fyifei0558/article/details/47001677 http://www.zhihu.com/question/28341521 有人吐槽怎么下手FreeNOS，上文档 FreeNOS: FreeNOS Documentation －－更新－－ 把平时关注的一些c++相关的github的project列下，欢迎补充。 1. FreeNOS lordsergioinspa/FreeNOS · GitHub 这是我在大学的时候，接触完整学习的第一个开源的C++项目，当时简直是，各种惊叹，里面有太多值得学习的地方。 这是一个用C++实现的微内核的 操作系统 ，各种宏内核中的服务作为一个独立的services在微内核中，基于消息的通信方式，这点其实跟mac内核中的mach那部分机制相似。 除了是一个操作系统的实现，另外从中也能很好的学习到OOP的设计方式，整个代码风格特别好，完全基于面相对象，还有一些常见的设计模式，在接触了这个开源项目之后，才了解，代码风格，注释，doxygen，scons，设计模式。 对于直接入手就啃 Linux kernel的同学，个人觉得会门槛有点高，身体还不是特别棒的时候，直接攀登珠穆拉玛还是有点吃力的，可以先找个黄山爬爬。 2. Chromium 向左向右

UnknownHostException with Mesos + Spark and custom Jar

由 时间秒杀一切 提交于 2019-12-14 02:36:21
问题 I am receiving an UnknownHostException when running a custom jar with Spark on Mesos. The issue does not happen when running spark-shell. My spark-env.sh contains the following: export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so export HADOOP_CONF_DIR=/hadoop-2.7.1/etc/hadoop/ My spark-defaults.conf contains the following: spark.master mesos://zk://172.31.0.81:2181,172.31.16.81:2181,172.31.32.81:2181/mesos spark.mesos.executor.home /spark-1.5.0-bin-hadoop2.6/ These settings are on

Docker mesosphere/chronos container fails immediately after launch

由 允我心安 提交于 2019-12-13 18:25:31
问题 I am trying to launch Chronos in Docker, using mesosphere/chronos image. From command line Running following command to run the image doesn't work fine. docker run -p 8081:8081 -t mesosphere/chronos:latest /usr/bin/chronos --master zk://<master-hostname>:2181/mesos --zk_hosts <master-hostname>:2181 --http_port 8081 (I am trying with a single ZK node and a single Mesos Master node) It shows following messages soon after a few seconds. And no docker container of Chronos runs. /usr/bin/chronos:

A single Kubernetes Cluster running on multiple clouds

由 纵然是瞬间 提交于 2019-12-13 01:53:50
问题 Is it in any way possible to configure a Kubernetes Cluster that utilizes ressources from multiple IaaS providers at the same time e.g. a cluster running partially on GCE and AWS? Or a Kubernetes Cluster running on your bare metal and an IaaS provider? Maybe in combination with some other tools like Mesos? Are there any other tools like Kubernetes that provide this capability? If it's not possbile with Kubernetes, what would one have to do in order to provide that feature? Any help or

How does Spark handle exceptions for a spark streaming job?

由 泄露秘密 提交于 2019-12-12 21:21:42
问题 This question might seem pretty large as is, however I have two specific situations which are better put together than separately. To start with, I'm reading data from Kafka into a dstream using the spark-streaming-kafka API. Assume I have one of the following two situations: // something goes wrong on the driver dstream.transform { rdd => throw new Exception } // something goes wrong on the executors dstream.transform { rdd => rdd.foreachPartition { partition => throw new Exception } } This

Missing java system properties when running spark-streaming on Mesos cluster

由 和自甴很熟 提交于 2019-12-12 05:13:48
问题 I submit a spark app to mesos cluster(running in cluster mode), and pass java system property through "--drive-java-options=-Dkey=value -Dkey=value" , however these system properties are not available at runtime, seems they are not set. --conf "spark.driver.extraJavaOptions=-Dkey=value" doesn't work either More details: the command is bin/spark-submit --master mesos://10.3.101.119:7077 --deploy-mode cluster --class ${classname} --driver-java-options "-Dconfiguration.http=http://10.3.101.119

Spark to MongoDB via Mesos

由 北城以北 提交于 2019-12-12 04:59:13
问题 I am trying to connect Apache Spark to MongoDB using Mesos. Here is my architecture: - MongoDB: MongoDB Cluster of 2 shards, 1 config server and 1 query server. Mesos: 1 Mesos Master, 4 Mesos slaves Now I have installed Spark on just 1 node. There is not much information available on this out there. I just wanted to pose a few questions: - As per what I understand, I can connect Spark to MongoDB via mesos. In other words, I end up using MongoDB as a storage layer. Do I really Need Hadoop? Is

what is a better way to run docker under chronos?

由 梦想与她 提交于 2019-12-12 04:32:39
问题 I have a dockerized spring-boot app, which needs to be scheduled in mesos / chronos (DC/OS). There are 2 places, which (I think) can be used to accomplish Chronos scheduling: 1) by using the "command" tag 2) by using the "container" tag An example of Chronos command tag would be: { "name": "my-dockerized-app", "command": "docker login -u my_username -p my_password -e dev-my_user@my_company.com;docker run -e id=123 my_owner/my_dockerized_app_image:latest", "shell": true, "epsilon": "PT60S",

finding active framework current resource usage in mesos

由 落爺英雄遲暮 提交于 2019-12-12 02:59:40
问题 Which HTTP endpoint will help me to find all the active frameworks current resource utilization? We want this information because we want to dynamically scale Mesos cluster and our algorithm needs information regarding what resources each active framework is using. 回答1: I think to focus on the frameworks is not really what you would want to to. What you're after is probably the Mesos Slave utilization, which can be requested via calling http://{mesos-master}:5050/master/state-summary In the
订阅 mesos