luigi

类脑丨我们如何控制自己的身体？

由 白昼怎懂夜的黑 提交于 2020-08-04 18:07:28
撰文丨米·戈（中国科学院大学博士研究生） 编辑丨qingning 排版丨夏獭 作者语 大脑这个结构精妙又交错纵横的器官一直以来是研究者们关心的研究对象，通过研究让我们的大脑来理解大脑，这一旅程想必十分有趣。而随着神经科学研究的不断深入，与之并行的是类脑科学的日新月异，除了想办法理解脑，研究者们还致力于如何模拟脑。为了向读者介绍如何从类脑科学的角度出发来理解大脑的工作机制，笔者计划通过“类脑”专栏和读者一起感受大脑的“机械”之美。 封面：19世纪机械娃娃工艺品草图 得益于当时工匠的奇技淫巧，勒内·笛卡尔（René Descartes）在1662年出版的L’homme《论人》中描写了这样一段经历：他在青年时期曾在以机械雕像闻名的圣杰曼·昂·雷（Saint-Germain-en-Laye）皇家花园散步，在园中看到一个栩栩如生的机器人跟他打招呼。仔细追问之下，工匠给他展示了机器人的内部结构。这个机器人由液压控制，打开对应的阀门，就会有水注入，让机器人做出相应的动作。 机械剧场 由亚历山大的Hero发明，该图片收录在其编写的Hero's Pneumatics的意大利译本中。 这个事件启发了 笛卡尔 ，他认为“通过物质过程来解释生命是有可能的”。基于这个观点，他提出了动物精气理论，认为动物没有心灵，其行为用机器原理就可以解释。动物体内能产生一种叫做“动物精气”（Animal

Airflow/Luigi for AWS EMR automatic cluster creation and pyspark deployment

由 ℡╲_俬逩灬. 提交于 2020-06-13 05:36:48
问题 I am new to airflow automation, i dont now if it is possible to do this with apache airflow(or luigi etc) or should i just make a long bash file to do this. I want to build dag for this Create/clone a cluster on AWS EMR Install python requirements Install pyspark related libararies Get latest code from github Submit spark job Terminate cluster on finish for individual steps, i can make .sh files like below(not sure if it is good to do this or not) but dont know how to do it in airflow 1)

Airflow/Luigi for AWS EMR automatic cluster creation and pyspark deployment

由 主宰稳场 提交于 2020-06-13 05:36:30
问题 I am new to airflow automation, i dont now if it is possible to do this with apache airflow(or luigi etc) or should i just make a long bash file to do this. I want to build dag for this Create/clone a cluster on AWS EMR Install python requirements Install pyspark related libararies Get latest code from github Submit spark job Terminate cluster on finish for individual steps, i can make .sh files like below(not sure if it is good to do this or not) but dont know how to do it in airflow 1)

When a new file arrives in S3, trigger luigi task

由 纵饮孤独 提交于 2019-12-25 06:29:04
问题 I have a bucket with new objects getting added at random intervals with keys based on their time of creation. For example: 's3://my-bucket/mass/%s/%s/%s/%s/%s_%s.csv' % (time.strftime('%Y'), time.strftime('%m'), time.strftime('%d'), time.strftime('%H'), name, the_time) In fact, these are the outputs of Scrapy crawls. I want to trigger a task that matches these crawls to a master .csv product catalog file I have (call it "product_catalog.csv"), which also gets updated regularly. Right now, I

How to run parallel instances of a Luigi Pipeline : Pid set already running

由 ﹥＞﹥吖頭↗ 提交于 2019-12-24 07:29:13
问题 I have a simple pipeline. I want to start it once with the Id 2381, then while the first job is running I want to start a second run with the Id 231. The first run completes as expected. The second run returns this response Pid(s) set([10362]) already running Process finished with exit code 0 I am starting the runs like this run one: luigi.run( cmdline_args=["--id='newId13822'", "--TaskTwo-id=2381"], main_task_cls=TaskTwo() ) run two: luigi.run( cmdline_args=["--id='newId1322'", "--TaskTwo-id

Luigi - Overriding Task requires/input

由 扶醉桌前 提交于 2019-12-23 11:01:03
问题 I am using luigi to execute a chain of tasks, like so: class Task1(luigi.Task): stuff = luigi.Parameter() def output(self): return luigi.LocalTarget('test.json') def run(self): with self.output().open('w') as f: f.write(stuff) class Task2(luigi.Task): stuff = luigi.Parameter() def requires(self): return Task1(stuff=self.stuff) def output(self): return luigi.LocalTarget('something-else.json') def run(self): with self.output().open('w') as f: f.write(stuff) This works exactly as desired when I

Luigi - Overriding Task requires/input

由 二次信任 提交于 2019-12-23 11:00:39
问题 I am using luigi to execute a chain of tasks, like so: class Task1(luigi.Task): stuff = luigi.Parameter() def output(self): return luigi.LocalTarget('test.json') def run(self): with self.output().open('w') as f: f.write(stuff) class Task2(luigi.Task): stuff = luigi.Parameter() def requires(self): return Task1(stuff=self.stuff) def output(self): return luigi.LocalTarget('something-else.json') def run(self): with self.output().open('w') as f: f.write(stuff) This works exactly as desired when I

Persist Completed Pipeline in Luigi Visualiser

由 久未见 提交于 2019-12-23 10:04:22
问题 I'm starting to port a nightly data pipeline from a visual ETL tool to Luigi, and I really enjoy that there is a visualiser to see the status of jobs. However, I've noticed that a few minutes after the last job (named MasterEnd ) completes, all of the nodes disappear from the graph except for MasterEnd . This is a little inconvenient, as I'd like to see that everything is complete for the day/past days. Further, if in the visualiser I go directly to the last job's URL, it can't find any

S3 file to local using luigi raises UnicodeDecodeError

由 北战南征 提交于 2019-12-23 02:40:59
问题 I am copying a pdf file to local, using the following piece of code: with self.input_target().open('r') as r: with self.output_target().open('w') as w: for line in r: w.write(line) Which is based in this question (kind of) But when I execute that code I get the following: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 11: invalid continuation byte I tried this other approach, without good results: with self.input_target().open('r') as r, self.output_target().open('w') as

Suggestion for scheduling tool(s) for building hadoop based data pipelines

由 青春壹個敷衍的年華 提交于 2019-12-22 17:54:56
问题 Between Apache Oozie, Spotify/Luigi and airbnb/airflow, what are the pros and cons for each of them? I have used oozie and airflow in the past for building a data ingestion pipeline using PIG and Hive. Currently, I am in the process of building a pipeline that looks at logs and extracts out useful events and puts them on redshift. I found that airflow was much easier to use/test/setup. It has a much cooler UI and lets users perform actions from the UI itself, which is not the case with Oozie.
订阅 luigi