papermill

Bash script with multiple papermill commands does not fail on notebook errors

由 无人久伴 提交于 2021-02-16 20:55:25
问题 I have a refresh_data.sh file which contains multiple papermill commands, for example: papermill notebook_1.ipynb output_1.ipynb -p start "2017-12-01" -p date "2017-12-31" papermill notebook_2.ipynb output_2.ipynb -p start "2018-01-01" -p date "2018-01-31" If I get an error while it is running the first notebook, the process continues executing the second one. In other words, an error in one of the notebooks doesn't "break" the overall script. As far as I remember with normal python scripts

Bash script with multiple papermill commands does not fail on notebook errors

由 断了今生、忘了曾经 提交于 2021-02-16 20:55:12
问题 I have a refresh_data.sh file which contains multiple papermill commands, for example: papermill notebook_1.ipynb output_1.ipynb -p start "2017-12-01" -p date "2017-12-31" papermill notebook_2.ipynb output_2.ipynb -p start "2018-01-01" -p date "2018-01-31" If I get an error while it is running the first notebook, the process continues executing the second one. In other words, an error in one of the notebooks doesn't "break" the overall script. As far as I remember with normal python scripts

Connect to remote python kernel from python code

由 纵饮孤独 提交于 2020-02-24 17:31:25
问题 I have been using PaperMill for executing my python notebook periodically. To execute compute intensive notebook, I need to connect to remote kernel running in my EMR cluster. In case of Jupyter notebook I can do that by starting jupyter server with jupyter notebook --gateway-url=http://my-gateway-server:8888 and I am able to execute my code on remote kernel. But how do I let my local python code(through PaperMill) to use remote kernel? What changes do what to make in Kernel Manager to

Connect to remote python kernel from python code

由 寵の児 提交于 2020-02-24 17:29:57
问题 I have been using PaperMill for executing my python notebook periodically. To execute compute intensive notebook, I need to connect to remote kernel running in my EMR cluster. In case of Jupyter notebook I can do that by starting jupyter server with jupyter notebook --gateway-url=http://my-gateway-server:8888 and I am able to execute my code on remote kernel. But how do I let my local python code(through PaperMill) to use remote kernel? What changes do what to make in Kernel Manager to

Connect to remote python kernel from python code

由 两盒软妹~` 提交于 2020-02-24 17:29:06
问题 I have been using PaperMill for executing my python notebook periodically. To execute compute intensive notebook, I need to connect to remote kernel running in my EMR cluster. In case of Jupyter notebook I can do that by starting jupyter server with jupyter notebook --gateway-url=http://my-gateway-server:8888 and I am able to execute my code on remote kernel. But how do I let my local python code(through PaperMill) to use remote kernel? What changes do what to make in Kernel Manager to

Passing command line arguments to argv in jupyter/ipython notebook

由 让人想犯罪 __ 提交于 2019-12-17 06:37:32
问题 I'm wondering if it's possible to populate sys.argv (or some other structure) with command line arguments in a jupyter/ipython notebook, similar to how it's done through a python script. For instance, if I were to run a python script as follows: python test.py False Then sys.argv would contain the argument False . But if I run a jupyter notebook in a similar manner: jupyter notebook test.ipynb False Then the command line argument gets lost. Is there any way to access this argument from within

How do I set parameters for each notebook on JuPyterHub running on K8s

由 倾然丶 夕夏残阳落幕 提交于 2019-12-11 15:57:43
问题 I want to set some parameters as defined here(https://github.com/nteract/papermill#python-version-support). The catch is, I want to be able to do this via UI. I have a JHub installed on my cluster and while opening it, I want certain parameters to be set by default. Also, when I pass the parameters via papermill(the above script gets saved somewhere and then I will run it via papermill), I want the latter to override the former. I tried looking into several topics in pure JuPyter notebooks

how to run a python jupyter notebook daily automatically

由 て烟熏妆下的殇ゞ 提交于 2019-12-03 12:10:44
问题 I have a code in a python jupyter notebook but i need to run this every day so I would like to know if there's a way to set this, I really appreciate it 回答1: It's better to combine with airflow if you want to have higher quality. I packaged them in a docker image, https://github.com/michaelchanwahyan/datalab. It is done by modifing an open source package nbparameterize and integrating the passing arguments such as execution_date. Graph can be generated on the fly The output can be updated and

how to run a python jupyter notebook daily automatically

由 坚强是说给别人听的谎言 提交于 2019-12-03 03:34:10
I have a code in a python jupyter notebook but i need to run this every day so I would like to know if there's a way to set this, I really appreciate it It's better to combine with airflow if you want to have higher quality. I packaged them in a docker image, https://github.com/michaelchanwahyan/datalab . It is done by modifing an open source package nbparameterize and integrating the passing arguments such as execution_date. Graph can be generated on the fly The output can be updated and saved within inside the notebook. When it is executed the notebook will be read and inject the parameters

Passing command line arguments to argv in jupyter/ipython notebook

由 跟風遠走 提交于 2019-11-27 13:27:45
I'm wondering if it's possible to populate sys.argv (or some other structure) with command line arguments in a jupyter/ipython notebook, similar to how it's done through a python script. For instance, if I were to run a python script as follows: python test.py False Then sys.argv would contain the argument False . But if I run a jupyter notebook in a similar manner: jupyter notebook test.ipynb False Then the command line argument gets lost. Is there any way to access this argument from within the notebook itself? After a lot of looking around I found very cumbersome, custom libraries, but
订阅 papermill