mesos | 易学教程

Spark mesos cluster mode is slower than local mode

阅读更多关于 Spark mesos cluster mode is slower than local mode

I submit the same jar to run by using both local mode and mesos cluster mode. And found for some exactly same stages, local mode only takes several milliseconds to finish however cluster mode will take seconds! listed is one example: stage 659 local mode: 659 Streaming job from [output operation 1, batch time 17:45:50] map at KafkaHelper.scala:35 +details 2016/03/22 17:46:31 11 ms mesos cluster mode: 659 Streaming job from [output operation 1, batch time 18:01:20] map at KafkaHelper.scala:35 +details 2016/03/22 18:09:33 3 s And I found from spark UI that mesos cluster mode will consistently

Mesos Framework

阅读更多关于 Mesos Framework

问题 I want to distribute Docker containers on a cluster by using java and define dependencies, so that when one container is finished I can parse the produced output in my java code. There are many mesos frameworks that can achieve that and I'm not sure which one to take: Marathon, Singularity, Chronos, Aurora. What I have learned so far: Marathon has a really nice java client api but is used for long term tasks (don't know if that is a problem, because my tasks will not run that long) and I dont

mesos+docker+marathon

阅读更多关于 mesos+docker+marathon

最近工作不饱和啊，加上正好有空闲的机器，就想着搭个mesos+docker+marathon玩玩。上干货：环境Centos6.7，内核版本3.18 先安装mesos的依赖包 wget http://mirror.centos.org/centos/6/os/x86_64/Packages/apr-devel-1.3.9-5.el6_2.x86_64.rpm sudo rpm ihv apr-devel-1.3.9-5.el6_2.x86_64.rpm sudo yum install libcurl-devel apr-util-devel cyrus-sasl cyrus-sasl-devel cyrus-sasl-md5 subversion-devel 安装maven，配置PATH保证能找到mvn命令即可 BTW，linux下的依赖包一般都可以在https://pkgs.org/这个网站里找到下载mesos-1.0.0： wget http://archive.apache.org/dist/mesos/1.0.0/mesos-1.0.0.tar.gz tar zxvf mesos-1.0.0.tar.gz cd mesos-1.0.0 # 配置安装，编译之前要保证g++版本不要过低，过低请先升级gcc版本： ./configure --prefix=/usr/local

How should a .dockercfg file be hosted in a Mesosphere-on-AWS setup so that only Mesosphere can use it?

阅读更多关于 How should a .dockercfg file be hosted in a Mesosphere-on-AWS setup so that only Mesosphere can use it?

问题 We have set up a test cluster with Mesosphere on AWS, in a private VPC. We have some Docker images which are public, which are easy enough to deploy. However most of our services are private images, hosted on the Docker Hub private plan, and require authentication to access. Mesosphere is capable of private registry authentication, but it achieves this in a not-exactly-ideal way: a HTTPS URI to a .dockercfg file needs to be specified in all Mesos/Marathon task definitions. As the title

How to measure the CPU utilization of Each container in Mesos?

阅读更多关于 How to measure the CPU utilization of Each container in Mesos?

I have many apps running on containers in Mesos, managed via marathon. I have given CPU allocation for each app while deploying via marathon like 1, .5 etc. But the CPU allocation in marathon, does not mean that its 1 CPU or half CPU. It simply means that its time sharing ratio. Also each container gets to access all the CPUs on its Host. Now, I want to measure the CPU efficiency of each Container on Mesos slaves, so that I can reduce or increase the CPU allocation in for each App in Marathon. I just want to make resource utilisation even more efficient. I could use https://github.com/bobrik

Mesos DCOS doesn't install Kafka

阅读更多关于 Mesos DCOS doesn't install Kafka

I'm trying to install Kafka on Mesos. Installation seems to have succeeded. vagrant@DevNode:/dcos$ dcos package install kafka This will install Apache Kafka DCOS Service. Continue installing? [yes/no] yes Installing Marathon app for package [kafka] version [0.9.4.0] Installing CLI subcommand for package [kafka] version [0.9.4.0] New command available: dcos kafka The Apache Kafka DCOS Service is installed: docs - https://github.com/mesos/kafka issues - https://github.com/mesos/kafka/issues vagrant@DevNode:/dcos$ dcos package list NAME VERSION APP COMMAND DESCRIPTION kafka 0.9.4.0 /kafka kafka

How to run Hadoop on a Mesos cluster?

阅读更多关于 How to run Hadoop on a Mesos cluster?

问题 I am trying to set up a Apache Mesos Cluster and run Hadoop-Job on it. The documentation here is not at my level, so that i am not able to understand and maybe someone here can explain me: First should I set up a working Hadoop Cluster? Or first set up a Mesos cluster? where do I point the slaves? in Hadoop-slaves file or registered Mesos slaves should only be used? 回答1: The goal of Mesos is to run an abstraction for your cluster, where Hadoop would just be 1 service among others. In order

How to understand CPU allocation in Mesos?

阅读更多关于 How to understand CPU allocation in Mesos?

问题 I am reading Building Applications on Mesos, and come across the following statements: cpus This resource expresses how many CPU cores are available. Tasks may use fractional parts of a CPU—this is possible because Mesos slaves use CPU shares, rather than reserving specific CPUs. This means that, if you have 1.5 cpus reserved, your processes will be allowed to use a total of 1.5 seconds of CPU time each second. That could mean that, within a single executor, two processes each get 750

Spark shell connect to Mesos hangs: No credentials provided. Attempting to register without authentication

阅读更多关于 Spark shell connect to Mesos hangs: No credentials provided. Attempting to register without authentication

I installed Mesos in an OpenStack environment using these instructions from Mesosphere: https://open.mesosphere.com/getting-started/datacenter/install/ . I ran the verification test as described and it was successful. UI for both Mesos and Marathon are working as expected. When I run the Spark shell from my laptop I cannot connect. The shell hangs with the output below. I don't see anything in the Mesos master or slave logs that would indicate an error, so am not sure what to investigate next. Any help would be appreciated. TOMWATER-M-60SN:bin tomwater$ ./spark-shell --master mesos://zk://10

Understanding resource allocation for spark jobs on mesos

阅读更多关于 Understanding resource allocation for spark jobs on mesos

问题 I'm working on a project in Spark, and recently switched from using Spark Standalone to Mesos for cluster management. I now find myself confused about how to allocate resources when submitting a job under the new system. In standalone mode, I was using something like this (following some recommendations from this Cloudera blog post: /opt/spark/bin/spark-submit --executor-memory 16G --executor-cores 8 --total-executor-cores 240 myscript.py This is on a cluster where each machine has 16 cores