executor

Spark cluster full of heartbeat timeouts, executors exiting on their own

匿名 (未验证) 提交于 2019-12-03 02:49:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: My Apache Spark cluster is running an application that is giving me lots of executor timeouts: 10:23:30,761 ERROR ~ Lost executor 5 on slave2.cluster: Executor heartbeat timed out after 177005 ms 10:23:30,806 ERROR ~ Lost executor 1 on slave4.cluster: Executor heartbeat timed out after 176991 ms 10:23:30,812 ERROR ~ Lost executor 4 on slave6.cluster: Executor heartbeat timed out after 176981 ms 10:23:30,816 ERROR ~ Lost executor 6 on slave3.cluster: Executor heartbeat timed out after 176984 ms 10:23:30,820 ERROR ~ Lost executor 0 on slave5

AsyncTask execute() or executeOnExecutor()?

匿名 (未验证) 提交于 2019-12-03 02:46:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: What's the difference between using execute() and executeOnExecuter() ? How does execute() execute tasks by default? (in serial or in parallel?) What should be used for new SDKs >16? Is it a good practice to use parallel execution ( THREAD_POOL_EXECUTOR ) for tasks rather than serial even if it doesn't matter for the application or does that depends on the number of AsyncTask s that will be executed? 回答1: How does .execute execute tasks by default (in serial or in parallel). Before API level 11: parallel. API level 11 and up: serial. which

How to test Android UI using IdlingResource when using Retrofit network requests

匿名 (未验证) 提交于 2019-12-03 02:45:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I am writing integration tests that perform actions in the UI which start network calls using Retrofit . I know I need to implement a CountingIdlingResource , but I want to do it the correct way (and not reinvent the wheel if it has already been done). Has anyone implemented an IdlingResource in their app's Espresso test suite to wait while network requests execute? More info here . 回答1: The most straightforward solution for this: is to basically swap out Retrofit's Thread-pool executor with an AsyncTask one (as recommended by the very

Kubernetes executor on Gitlab - ERROR: Job failed (system failure): Post *api/v1/namespaces/gitlab/pods: x509: certificate signed by unknown authority

匿名 (未验证) 提交于 2019-12-03 02:36:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I'm trying to set up Kubernetes executor for Gitlab but I get this error: ERROR: Job failed (system failure): Post https://api.kubernetes.de/api/v1/namespaces/gitlab/pods : x509: certificate signed by unknown authority This is my configmap.yml: apiVersion: v1 kind: ConfigMap metadata: name: gitlab-runner namespace: gitlab data: config.toml: | concurrent = 4 [[runners]] name = "Kubernetes Runner" url = "http://########/ci" token = "############" executor = "kubernetes" [runners.kubernetes] host = "https://api.kubernetes.de" namespace =

Executor and Daemon in Java

匿名 (未验证) 提交于 2019-12-03 02:14:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I have a MyThread object which I instantiate when my app is loaded through the server, I mark it as a Daemon thread and then call start() on it. The thread is meant to sit and wait for information from a queue as long as the application is active. My problem/question is this: Currently MyThread is extending Thread because I mark it as Daemon and I read about how it's more prefferable to implement Runnable and to use Executors. So what I wanted to ask is if MyThread will implement Runnable instead of extending Thread (and of course will be

Apache Spark: The number of cores vs. the number of executors

匿名 (未验证) 提交于 2019-12-03 01:58:03
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I'm trying to understand the relationship of the number of cores and the number of executors when running a Spark job on YARN. The test environment is as follows: Number of data nodes: 3 Data node machine spec: CPU: Core i7-4790 (# of cores: 4, # of threads: 8) RAM: 32GB (8GB x 4) HDD: 8TB (2TB x 4) Network: 1Gb Spark version: 1.0.0 Hadoop version: 2.4.0 (Hortonworks HDP 2.1) Spark job flow: sc.textFile -> filter -> map -> filter -> mapToPair -> reduceByKey -> map -> saveAsTextFile Input data Type: single text file Size: 165GB Number of

What is the “task” in Storm parallelism

匿名 (未验证) 提交于 2019-12-03 01:09:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I'm trying to learn twitter storm by following the great article " Understanding the parallelism of a Storm topology " However I'm a bit confused by the concept of "task". Is a task an running instance of the component(spout or bolt) ? A executor having multiple tasks actually is saying the same component is executed for multiple times by the executor, am I correct ? Moreover in a general parallelism sense, Storm will spawn a dedicated thread(executor) for a spout or bolt, but what is contributed to the parallelism by an executor(thread)

How to install CUDA 8.0 in the latest version of Tensorflow (1.0) in AWS p2.xlarge instance, AMI ami-edb11e8d and nvidia drivers up to date (375.39)

匿名 (未验证) 提交于 2019-12-03 01:09:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I have upgraded to Tensorflow version 1.0 and installed CUDA 8.0 with the cudnn 5.1 version and the nvidia drivers up to date 375.39. My NVIDIA hardware is the one that is on Amazon Web Services using the p2.xlarge instance, a Tesla K-80. My OS is Linux 64-bit. I get the next error message every time I use the command: tf.Session() [ec2-user@ip-172-31-7-96 CUDA]$ python Python 2.7.12 (default, Sep 1 2016, 22:14:00) [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>

Elegantly implementing queue length indicators to ExecutorServices

泪湿孤枕 提交于 2019-12-03 01:04:15
Why, oh why doesn't java.util.concurrent provide a queue length indicators for its ExecutorService s? Recently I found myself doing something like this: ExecutorService queue = Executors.newSingleThreadExecutor(); AtomicInteger queueLength = new AtomicInteger(); ... public void addTaskToQueue(Runnable runnable) { if (queueLength.get() < MAX_QUEUE_LENGTH) { queueLength.incrementAndGet(); // Increment queue when submitting task. queue.submit(new Runnable() { public void run() { runnable.run(); queueLength.decrementAndGet(); // Decrement queue when task done. } }); } else { // Trigger error: too

Spark can no longer execute jobs. Executors fail to create directory

匿名 (未验证) 提交于 2019-12-03 00:47:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: We've had a small spark cluster running for a month now that's been successfully executing jobs or let me start up a spark-shell to the cluster. It doesn't matter if I submit a job to the cluster or connect to it using the shell, the error is always the same. root@~]$ $SPARK_HOME/bin/spark-shell Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/11/10 20:43:01 INFO spark.SecurityManager: Changing view acls to: root, 14/11/10 20:43:01 INFO spark.SecurityManager: Changing modify acls to: root, 14/11/10 20:43:01