Copy file from dbfs in cluster-scoped init script

问题

I want to try out cluster scoped init scripts on a Azure Databricks cluster. I'm struggling to see which commands are available.

Basically, I've got a file on dbfs that I want to copy to a local directory /tmp/config when the cluster spins up.

So I created a very simple bash script:

#!/bin/bash
mkdir - p /tmp/config
databricks fs cp dbfs:/path/to/myFile.conf /tmp/config

Spinning up the cluster fails with "Cluster terminated. Reason: Init Script Failure". Looking at the log on dbfs, I see the error

bash: line 1: databricks: command not found

OK, so databricks as a command is not available. That's the command I use on the local bash to copy files from and to dbfs.

What other commands are available to copy a file from dbfs? And more general: Which commands are actually available?

回答1:

By default, Databricks CLI is not installed on the databricks cluster. That's the reason you see this error message bash: line 1: databricks: command not found.

To achieve this, you should use dbutils commands as shown below.

dbutils.fs.mkdirs("/tmp/config")
dbutils.fs.mv("/configuration/proxy.conf", "/tmp/config")

Reference: Databricks Utilities

Hope this helps.

来源：https://stackoverflow.com/questions/58728321/copy-file-from-dbfs-in-cluster-scoped-init-script