How to change sqoop metastore?

后端 未结 3 1915
甜味超标
甜味超标 2020-12-16 08:23

I am using sqoop 1.4.2 version. I am trying to change the sqoop metastore from default hsqldb to mysql.

I have configured following properties in sqoop-site.xml file

3条回答
  •  佛祖请我去吃肉
    2020-12-16 08:31

    Public service announcement: Sqoop Metastore on other DBs may fail

    We have been able to get PostgreSQL and MySQL working as targets for the Sqoop Metastore on Sqoop 1, replacing the HyperSQL database. There's a little setup and seeding of the database needed, but from then on, it seemed fine.

    However, we are seeing cases when we are running many sqoop jobs, updating the metastore concurrently -- sqoop 1.4.6 has no code to trap and handle cases where metastore updates for incremental updates fail due to concurrency issues. In particular, Sqoop _will complete it's import successfully but not update the metastore with the most recently imported values. This will cause the next incremental run will import duplicate data. Sqoop will return a non-zero return code, but data in either Hadoop or the metastore need to be synced afterward in order for data to be correct.

    We're not sure there is a solution, but this is an expansion of @SandeerKumar's answer. This may be an issue with HyperSQL as well, but it would be much less likely because HSQL is in memory, so faster.

提交回复
热议问题