UDF(Java): permission denied at HDFS

不问归期 提交于 2019-12-12 05:16:48

问题


I write an hive-UDTF to resolve ip address via loading a .dat file on HDFS,but meet an error:

java.io.FileNotFoundException: hdfs:/THE_IP_ADDRESS:9000/tmp/ip_20170204.dat (Permission denied) 

But actually, both the dfs directory /tmp and the .data file have full access: 777, and I cannot modify the config to disable dfs permission.

The line in my UDTF to read the file:

IP.load("hdfs://THE_IP_ADDRESS:9000/tmp/ip_20170204.dat");

and the static method .load():

public static void load(String filename) {
        ipFile = new File(filename);
        load();
        if (enableFileWatch) {
            watch();
        }
    }


private static void load() {
    lastModifyTime = ipFile.lastModified();
    FileInputStream fin = null;
    lock.lock();
    try {
        dataBuffer = ByteBuffer.allocate(Long.valueOf(ipFile.length()).intValue());
        fin = new FileInputStream(ipFile);
        int readBytesLength;
        byte[] chunk = new byte[4096];
        while (fin.available() > 0) {
            readBytesLength = fin.read(chunk);
            dataBuffer.put(chunk, 0, readBytesLength);
        }
        dataBuffer.position(0);
        int indexLength = dataBuffer.getInt();
        byte[] indexBytes = new byte[indexLength];
        dataBuffer.get(indexBytes, 0, indexLength - 4);
        indexBuffer = ByteBuffer.wrap(indexBytes);
        indexBuffer.order(ByteOrder.LITTLE_ENDIAN);
        offset = indexLength;

        int loop = 0;
        while (loop++ < 256) {
            index[loop - 1] = indexBuffer.getInt();
        }
        indexBuffer.order(ByteOrder.BIG_ENDIAN);
    } catch (IOException ioe) {
        ioe.printStackTrace();
    } finally {
        try {
            if (fin != null) {
                fin.close();
            }
        } catch (IOException e){
            e.printStackTrace();
        }
        lock.unlock();
    }
}

来源:https://stackoverflow.com/questions/43799688/udfjava-permission-denied-at-hdfs

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!