Running only Java 8 tests and nothing else. We added the short commit hash to the file name so that we can distinguish individual packages built for SNAPSHOT versions. The debian package can then be found under assembly/target. This can be created by specifying the following: mvn -Pdeb -DskipTests clean package The Maven build includes support for building a Debian package containing the assembly ‘fat-jar’, PySpark, and the necessary scripts and configuration files.
Intellij jar red full#
Thus, the full flow for running continuous-compilation of the core submodule may look more like:īuilding Spark with IntelliJ IDEA or Eclipseįor help in setting up IntelliJ IDEA or Eclipse for Spark development, and troubleshooting, refer to the Specific submodules to work this is because submodules that depend on other submodules do so via
Intellij jar red install#
* you’ll typically need to run mvn install from the project root for compilation within * it only scans the paths src/main and src/test (seeįrom within certain submodules that have that structure. However, this has not been testedĮxtensively. We use the scala-maven-plugin which supports incremental and continuous compilation. The ScalaTest plugin also supports running only a specific test suite as follows: mvn -Dhadoop.version=. Mvn -Pyarn -Phadoop-2.3 -Phive -Phive-thriftserver test The following is an example of a correct (build, test) sequence: mvn -Pyarn -Phadoop-2.3 -DskipTests -Phive -Phive-thriftserver clean package Some of the tests require Spark to be packaged first, so always run mvn package with -DskipTests the first time. Tests are run by default via the ScalaTest Maven plugin. Specifically, Spark’s external Kafka library and JDBC component are not yet Scala 2.11 support in Spark is experimental and does not support a few features. Mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 property: dev/change-version-to-2.11.sh Mvn -Pyarn -Phadoop-2.4 -Dhadoop.version =2.4.0 -Phive -Phive-0.12.0 -Phive-thriftserver -DskipTests clean package Building for Scala 2.11 # Apache Hadoop 2.4.X with Hive 12 support Mvn -Pyarn -Phadoop-2.4 -Dhadoop.version =2.4.0 -Phive -Phive-thriftserver -DskipTests clean package # Apache Hadoop 2.4.X with Hive 13 support To enable Hive integration for Spark SQL along with its JDBC server and CLI,Īdd the -Phive and Phive-thriftserver profiles to your existing build options.īy default Spark will build with Hive 0.13.1 bindings. Mvn -Pyarn-alpha -Phadoop-2.3 -Dhadoop.version =2.3.0 -Dyarn.version =0.23.7 -DskipTests clean package Building With Hive and JDBC Support Versions of Hadoop after 2.5.X may or may not work with the -Phadoop-2.4 profile (they were Mvn -Pyarn -Phadoop-2.4 -Dhadoop.version =VERSION -DskipTests clean package Mvn -Pyarn -Phadoop-2.3 -Dhadoop.version =2.3.0 -DskipTests clean package Mvn -Pyarn -Phadoop-2.2 -Dhadoop.version =2.2.0 -DskipTests clean package Mvn -Pyarn-alpha -Phadoop-0.23 -Dhadoop.version =0.23.7 -DskipTests clean package Mvn -Pyarn-alpha -Dhadoop.version =2.0.0-cdh4.2.0 -DskipTests clean package Mvn -Pyarn-alpha -Dhadoop.version =2.0.5-alpha -DskipTests clean package Note that certain build profiles are required for particular Hadoop versions: Hadoop versionįor Apache Hadoop versions 1.x, Cloudera CDH “mr1” distributions, and other Hadoop versions without YARN, use: If unset, Spark will build against Hadoop 1.0.4 by default. You can do this through the “hadoop.version” property. Specifying the Hadoop Versionīecause HDFS is not protocol-compatible across versions, if you want to read from HDFS, you’ll need to build Spark against the specific HDFS version in your environment. Note: For Java 8 and above this step is not required. You can fix this by setting the MAVEN_OPTS variable as discussed before. Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.10/classes. If you don’t run this, you may see errors like the following: Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.10/classes. Export MAVEN_OPTS = "-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"