Thursday, July 15, 2010

Spring Batch Deployment Example

Deployment has always been a tricky part of Batch Processing. Unlike a Web application, there is no standardized deployment, and there are a variety of environments that could be deployed to. You could actually deploy a Batch application within a web container, or as a standalone java app to be started by one of many available schedulers. For this reason, everyone's environment is different, and there can be no one example that someone can use as a starting point. However, I have done enough standalone deployments for Linux using Bash to share a simple example.

The job itself matters little for the purposes of this post. A simple one tasklet job suffice:

   <job id="sampleJob" job-repository="jobRepository">
<step id="simpleStep">
<tasklet ref="tasklet" />
</step>
</job>

<beans:bean id="jobRepository"
class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"/>

<beans:bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />

<beans:bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<beans:property name="jobRepository" ref="jobRepository" />
</beans:bean>

<beans:bean id="tasklet" class="net.lucasward.sample.SampleTasklet" />


(namespace removed for readability)

The Tasklet simply prints 'Hello World'

public class SampleTasklet implements Tasklet {

public RepeatStatus execute(StepContribution contribution,
ChunkContext chunkContext) throws Exception {
System.out.println("Hello World");
return FINISHED;
}
}


Simple enough, right? Now for the hard part. If we want a scheduler to be able to launch this from the command line, what do we do? The first problem to solve is, what does our deployment look like? In my mind, there's three necessary components:


  1. The jars themselves, which need to be on the classpath
  2. A Script that can be called
  3. The xml and/or .properties files that will be used for configuration


Personally, I prefer to separate the XML files from the jars, to allow for tweaks to the Job. This is especially useful for Spring Batch job definitions, although may be less so for normal spring configuration files. My preferred layout is to have three directories: bin, lib, and resources. Obviously, the scripts go in /bin, jars in /lib, and xml/properties in /resources. It doesn't really matter how you break yours up, but it's the format I'll be using. In order to create this layout, I'll use Maven and the assembly plugin:


<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2-beta-2</version>
<configuration>
<descriptors>
<descriptor>src/main/assembly/descriptor.xml</descriptor>
</descriptors>
</configuration>
<executions>
<execution>
<id>make-distribution</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>


And the Descriptor:

<assembly>
<id>distribution</id>
<formats>
<format>tar.gz</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>src/main/scripts</directory>
<outputDirectory>bin</outputDirectory>
<useDefaultExcludes>true</useDefaultExcludes>
</fileSet>
<fileSet>
<directory>src/main/resources</directory>
<outputDirectory>resources</outputDirectory>
<useDefaultExcludes>true</useDefaultExcludes>
<filtered>true</filtered>
</fileSet>
</fileSets>

<dependencySets>
<dependencySet>
<outputDirectory>lib</outputDirectory>
</dependencySet>
</dependencySets>
</assembly>


The format I'm using is tar.gz, since I'm targeting linux, but there are many more available.

The two fileset entries describes both the bin and resources directory. (I haven't talked about the script yet, but I will below). The 'DependencySet' reference is to transfer all the dependencies that Maven is managing into the lib directory, including the created jar itself. When you run 'mvn install' there will be two artifacts created: The normal jar, and another file with the same name, ending in -distribution.tar.gz. In the case of my example that is: batch-deploy-sample-1.0-SNAPSHOT.jar and batch-deploy-sample-1.0-SNAPSHOT-distribution.tar.gz. In my example, unzipping gave me the following:

./bin:
sampleJob.sh

./lib:
aopalliance-1.0.jar spring-batch-core-2.1.1.RELEASE.jar spring-tx-2.5.6.jar
batch-deploy-sample-1.0-SNAPSHOT.jar spring-batch-infrastructure-2.1.1.RELEASE.jar stax-1.2.0.jar
commons-logging-1.1.1.jar spring-beans-2.5.6.jar stax-api-1.0.1.jar
jettison-1.1.jar spring-context-2.5.6.jar xpp3_min-1.1.4c.jar
spring-aop-2.5.6.jar spring-core-2.5.6.jar xstream-1.3.jar

./resources:
jobs

./resources/jobs:
sampleJob.xml


All we need now is a simple script to actually run the job:

#!/bin/bash

CP=resources/

LIB=lib/*
for f in $LIB
do
CP=$CP:$f
done

java -cp $CP org.springframework.batch.core.launch.support.CommandLineJobRunner \
jobs/sampleJob.xml sampleJob


I'm probably not going to win any awards for my bash scripting skills anytime soon, but it gets the job done and isn't quite as archaic as a more concise version would be. Essentially, I'm creating a classpath from all the jar files in /lib by looping through the files and separating them with a colon. Once that is done, I can start a java process, using the CommandLineJobRunner as the Main method. As described in the documentation, all I need to pass to the job runner is the xml file to run the job, and the job name. (It's worth noting that normally you would also need JobParameters, but since I'm using the MapRepository, it isn't necessary)

You can download the running example from my github account: batch-deploy-sample.

15 comments:

David Tam said...

Thanks, this is very useful for someone totally new to Spring Batch, but when I run the job.. I get this..

ClassPathXmlApplicationContext [INFO] Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@4ca31e1b: display name [org.springframework.context.support.ClassPathXmlApplicationContext@4ca31e1b]; startup date [Fri Oct 22 10:28:52 CDT 2010]; root of context hierarchy
2010-10-22 10:28:52 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jobs/sampleJob.xml]
2010-10-22 10:28:53 CommandLineJobRunner [ERROR] Job Terminated in error:
org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from class path resource [jobs/sampleJob.xml]; nested exception is java.lang.IllegalArgumentException: 'beanName' must not be empty

Its probably something simple I'm missing.

Lucas Ward said...

I'm sorry for the crazy late response, I've been extremely busy with client work for the last few weeks. Are you still having this issue?

Dil.............................................:) said...

I want to stop the running spring batch job, Please let me know how can i do that .. i wil b follooing u r blog fr ans

Philippe said...
This comment has been removed by the author.
Philippe said...

Thanks a lot ! It's exactly what I was looking for and it works like a charm...

Philippe said...

I slightly corrected the shell script so that it can be launched from anywhere, rather than having to cd to the bin directory.
Here is what I did :

# Set the classpath
SCRIPTDIR="$( cd "$( dirname "$0" )" && pwd )"
CLASSPATH=.

for file in `ls $SCRIPTDIR/../lib`
do
export CLASSPATH=$CLASSPATH:$SCRIPTDIR/../lib/$file
done
export CLASSPATH=$CLASSPATH:$SCRIPTDIR:$SCRIPTDIR/../resources

# Launch the conversion job
java org.springframework.batch.core.launch.support.CommandLineJobRunner ...

Lucas Ward said...

Interesting. I'll definitely add your code for the script directory. I'm not a big fan of using the CLASSPATH variable, so I think I'll stick to just passing in the classpath to the command line via -cp

sudhakar said...

Administrator@heart_of_devil /cygdrive/c/aqq/asd/bin
$ ./sampleJob.sh
java.lang.NoClassDefFoundError: org/springframework/batch/core/launch/support/CommandLineJobRunner
Caused by: java.lang.ClassNotFoundException: org.springframework.batch.core.launch.support.CommandLineJobRunner
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.springframework.batch.core.launch.support.CommandLineJobRunner. Program will exit.
Exception in thread "main"

alexduff said...

Good stuff, Lucas. Thanks for writing this up.

host tony said...

What's even more comfortable when you are entertaining after a day of hard work.
friv 3

Mariya Philip said...

Batch processing is an important process in java. That means execution of a series of programs on a computer without manual intervention. like this how to write an assignment is also very important topic in essay writing

John Gibson said...

A proper guidance in all works will always makes students happy. That is why there are so many online writing services launched in interenet. They wil help students tio prepare a good scdemic papers in affordable price. The http://bestukdissertation.deviantart.com/ is one one of the site i used. Also http://dissertationeditingservice.bravesites.com/ http://dissertationwritinghelp1.soup.io/ and http://bestukdissertation.inube.com/ are reliable services.

duyen kull said...

The war between humans, orcs and elves continues tank trouble . Lead your race through a series of epic battles, using your crossbow to fend off foes and sending out units to destroy castleshappy wheels . Researching and upgrading wisely will be crucial to your success! There are 5 ages total and each one will bring you new units to train to fight in the war for you cause.earn to die game Whatever you do, don’t neglect your home base because you cannot repair it and once it is destroyed, you lose! Age of War is the first game of the series and really sets the tone for the Age of War games earn to die for you. Also try out the Age of Defense series as it is pretty similar.
In this game, you start at the cavern men’s age, then evolvetank trouble game! There is a total of 5 ages, each with its units and turrets. Take control of 16 different units and 15 different turrets to defend your base and destroy your enemy.
The goal of the game also differs depending on the level. In most levels the goal is to reach a finish line or to collect tokens. Many levels feature alternate or nonexistent goals for the player. The game controls are shown just under gold mine. Movement mechanisms primarily include acceleration and tilting controls.
It consists of a total of 17 levels and the challenge you face in each level increases as you go up. unfair mario The game basically has a red ball that has to be moved across the various obstacles in its path to the goal.

dong dong23 said...

hermes bag
burberry outlet
nike air max
canada goose outlet
ugg outlet
jordan shoes
canada goose jackets
coach factory outlet
abercrombie outlet
burberry outlet online
polo ralph lauren outlet
nfl jerseys wholesale
pandora jewelry
celine handbags
golden state warriors jerseys
christian louboutin sale
uggs sale
tiffany outlet
clarisonic
abercrombie kids
louis vuitton handbags
ugg sale
hermes handbags
coach outlet
louis vuitton handbags
true religion uk
nike roshe run
coach outlet online
polo ralph shirts
coach outlet store online
ralph lauren
polo shirts
nike outlet
los angeles clippers jerseys
michael kors handbags
cheap uggs
rolex watches
jordans
rolex submariner watches
seattle seahawks jerseys
20168.15wengdongdong

eric yao said...

Coach Outlet http://www.coachstoreoutlet.us.org

Christian Louboutin Shoes http://www.christianlouboutinshoe.eu.com

Valentino Shoes http://www.valentinoshoes.eu.com

Michael Kors Outlet http://www.michaelkorsoutlet.us.org

Coach Factory Outlet http://www.coachfactoryoutlet.us.org

Coach Outlet Online http://www.coachoutletonlinestore.us.org

Coach Purses http://www.coachpurse.us.com

Kate Spade Outlet http://www.katespadefactoryoutlet.us.com

Toms Shoes http://www.toms-shoes.name

Hermes Belts http://www.hermesbelts.us.org

Louis Vuitton http://www.louisvuitton-outletstores.us.com

Fendi Handbags http://www.fendibelt.us.com

Giuseppe Shoes http://www.giuseppezanotti.eu.com

Michael Kors Outlet http://www.michael-kors-outlet.us.com

Stephen Curry Shoes http://www.stephencurryshoes.us.com

Salomon Shoes http://www.salomonshoes.eu.com

North Face Outlet http://www.northfacesoutlet.in.net

Coach Outlet http://www.outletcoach.in.net

North Face Outlet http://www.northfacesoutlet.us.com

Burberry Outlet http://www.burberryoutletstore.us.org

North Face Outlet http://www.north-faceoutlets.in.net

North Face Jackets http://www.outletnorthface.in.net

Skechers Shoes http://www.skechersshoes.us.com

Toms Outlethttp://www.tomsoutletstore.us.org

North Face Outlet http://www.northfacestoreoutlet.us.org

Nike Air Max http://www.nikeoutlet.us.org

Nike Hoodies http://www.nikehoodies.us.com

Marc Jacobs Handbags http://www.marcjacobshandbags.name

Marc Jacobs Outlet http://www.marcjacobsoutletstore.us.com

Jimmy Choo Shoes http://www.jimmy-chooshoes.us.com

Jimmy Choos http://www.jimmychoos.in.net

Burberry Belt http://www.burberrybelt.us.com

Louis Vuitton Belt http://www.louisvuitton-belt.com

Salvatore Ferragamo http://www.ferragamo-belt.us.com

Marc Jacobs Handbags http://marcjacobshandbagsoutlet.blog.com

Lululemon Outlet http://www.lululemonsoutlet.us.com

True Religion Outlet http://www.truereligion-outlets.in.net

Tommy Hilfiger http://www.tommyhilfigeroutlet.us.com

Michael Kors Outlet http://www.outletmichaelkors.eu.com

Coach Outlet http://www.outletcoach.eu.com

Red Bottoms http://www.redbottom.in.net

Kevin Durant Shoes http://www.kevindurantshoes.us.org

New Balance Outlet http://www.newbalanceoutlet.in.net

Adidas Outlet http://www.adidasoutlet.in.net

Coach Outlet Online http://www.coachoutlet-online.eu.com

Stephen Curry Jersey http://www.stephencurryjersey.us.com

Vans Outlet http://www.vansoutlet.us.com

Ralph Lauren Outlet http://www.ralphlauren-outletstore.in.net

True Religion Outlet http://www.true-religionoutlets.in.net

ED Hardy Outlet http://www.edhardyoutlet.in.net

North Face Outlet http://www.northfaceoutlet.eu.com

UGG Outlet http://www.uggoutlet.us.org

UGG Outlet http://www.outletugg.in.net

North Face Outlet http://www.northface--outlet.us.com

Ugg Boots Sale http://www.uggbootssale.in.net

UGGS For Women http://www.uggsforwomen.us.com

Skechers Go Walk http://www.skechersgowalk.us.com

Adidas Yeezy Boost http://www.adidasyeezyboost.in.net

Adidas Yeezy http://www.adidasyeezy.us.com

Adidas NMD http://www.adidas-nmd.us.org

Coach Outlet http://www.coachoutletinc.us.com