Are multiple files stored in a single block?

前端 未结 4 971
轻奢々
轻奢々 2020-12-28 11:27

When I store many small files into HDFS, will they get stored in a single block?

In my opinion, these small files should get stored into a single block according to

4条回答
  •  天涯浪人
    2020-12-28 11:40

    Yes. when you store large number of small files, they get stored in a single block until the block has equal space to accommodate. But the inefficiency comes because for each of these small files, there will be an indexing entry(filename,block,offset) gets created in the namenode for each small file. This wastes up the memory reserved for metadata in the namenode if we have many small files instead of small number of very large files.

提交回复
热议问题