Apache spark Hive, executable JAR with maven shade

Question

I\'m building apache-spark application with Apache Spark Hive. So far everything was ok - I\'ve been running tests and whole application in Intellij IDEA and all tests toget

Answer 1

Here is a link to a newer plugin.xml: https://pastebin.com/8b2deeCL build out of these dependencies:

• datanucleus-core-4.1.6
• datanucleus-rdbms-4.1.7
• datanucleus-api-jdo-4.2.1

according to the answer by @anebril .

Answer 2

I have found solution to my problem. Solution is described in answer to original question for datanucleus in executable jar: https://stackoverflow.com/a/27030103/6390361

1. Edit MANIFEST.MF to pretend that it is datanucleus OSGi bundle. This can be done by adding Bundle-SymbolicName and Premain-Class entries from datanucleus-core manifest.

2. Create file plugin.xml in your classpath (resource folder) and use root tag from datanucleus-core project.

3. Put all extension-point tags from datanucleus-core and datanucleus-rdbms at the beginning of the plugin tag. All extension-points from RDBMS projects has to be prefixed with store.rdbms. This is very important because datanucleus uses fully classified IDs including part from root plugin tag.

4. Merge all extension tags from projects datanucleus-core, datanucleus-rdbms and datanucleus-api-jdo and put them behind all extension points. Be careful some extensions are present in more projects so you need to merge content of extensions with same IDs.

Manifest entries

Bundle-SymbolicName: org.datanucleus;singleton:=true
Premain-Class: org.datanucleus.enhancer.DataNucleusClassFileTransformer

plugin.xml

File plugin.xml is too big to be pasted here but you should be able to merge it by hand. Following code contains all RDBMS extension points with fixed IDs.

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.2"?>    
<plugin id="org.datanucleus" name="DataNucleus Core" provider-name="DataNucleus">

    <!-- Extension points from datanucleus-core -->
    <extension-point id="api_adapter" name="Api Adapter" schema="schema/apiadapter.exsd"/>
    ...

    <!-- extension points from datanucleus-rdbms - fixed IDs -->
    <extension-point id="store.rdbms.connectionprovider" name="Connection Provider" schema="schema/connectionprovider.exsd"/>
    <extension-point id="store.rdbms.connectionpool" name="ConnectionPool" schema="schema/connectionpool.exsd"/>
    <extension-point id="store.rdbms.sql_expression" name="SQL Expressions" schema="schema/sql_expression.exsd"/>
    <extension-point id="store.rdbms.sql_method" name="SQL Methods" schema="schema/sql_method.exsd"/>
    <extension-point id="store.rdbms.sql_operation" name="SQL Expressions" schema="schema/sql_operation.exsd"/>
    <extension-point id="store.rdbms.sql_tablenamer" name="SQL Table Namer" schema="schema/sql_tablenamer.exsd"/>
    <extension-point id="store.rdbms.rdbms_mapping" name="RDBMS Mapping" schema="schema/rdbms_mapping.exsd"/>

    <!-- Merged extensions from datanucleus-core, datanucleus-rdbms and datanucleus-api-jdo -->
    <extension point="org.datanucleus.persistence_properties">...</extension>
    ...
</plugin>

maven-shade-plugin

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <manifestEntries>
                            <Main-Class>${main.class}</Main-Class>
                            <Premain-Class>org.datanucleus.enhancer.DataNucleusClassFileTransformer</Premain-Class>
                            <Bundle-SymbolicName>org.datanucleus;singleton:=true</Bundle-SymbolicName>
                        </manifestEntries>
                    </transformer>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>reference.conf</resource>
                    </transformer>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                </transformers>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
        </execution>
    </executions>
</plugin>