Since we're on a major migration process of this website, some component documents here are out of sync right now. In the meantime you may want to look at the early version of the new website
https://camel.apache.org/staging/
We would very much like to receive any feedback on the new site, please join the discussion on the Camel user mailing list.
Data Format AppendixData FormatCamel supports a pluggable DataFormat to allow messages to be marshalled to and from binary or text formats to support a kind of Message Translator. The following data formats are currently supported:
And related is the following:
UnmarshallingIf you receive a message from one of the Camel Components such as File, HTTP or JMS you often want to unmarshal the payload into some bean so that you can process it using some Bean Integration or perform Predicate evaluation and so forth. To do this use the unmarshal word in the DSL in Java or the Xml Configuration. For example DataFormat jaxb = new JaxbDataFormat("com.acme.model"); from("activemq:My.Queue"). unmarshal(jaxb). to("mqseries:Another.Queue"); The above uses a named DataFormat of jaxb which is configured with a number of Java package names. You can if you prefer use a named reference to a data format which can then be defined in your Registry such as via your Spring XML file. You can also use the DSL itself to define the data format as you use it. For example the following uses Java serialization to unmarshal a binary file then send it as an ObjectMessage to ActiveMQ from("file://foo/bar"). unmarshal().serialization(). to("activemq:Some.Queue"); MarshallingMarshalling is the opposite of unmarshalling, where a bean is marshalled into some binary or textual format for transmission over some transport via a Camel Component. Marshalling is used in the same way as unmarshalling above; in the DSL you can use a DataFormat instance, you can configure the DataFormat dynamically using the DSL or you can refer to a named instance of the format in the Registry. The following example unmarshals via serialization then marshals using a named JAXB data format to perform a kind of Message Translator from("file://foo/bar"). unmarshal().serialization(). marshal("jaxb"). to("activemq:Some.Queue"); Using Spring XMLThis example shows how to configure the data type just once and reuse it on multiple routes Error rendering macro 'code': Invalid value specified for parameter 'java.lang.NullPointerException'<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring"> <dataFormats> <jaxb id="myJaxb" prettyPrint="true" contextPath="org.apache.camel.example"/> </dataFormats> <route> <from uri="direct:start"/> <marshal><custom ref="myJaxb"/></marshal> <to uri="direct:marshalled"/> </route> <route> <from uri="direct:marshalled"/> <unmarshal><custom ref="myJaxb"/></unmarshal> <to uri="mock:result"/> </route> </camelContext> You can also define reusable data formats as Spring beans <bean id="myJaxb" class="org.apache.camel.model.dataformat.JaxbDataFormat"> <property name="prettyPrint" value="true"/> <property name="contextPath" value="org.apache.camel.example"/> </bean> SerializationSerialization is a Data Format which uses the standard Java Serialization mechanism to unmarshal a binary payload into Java objects or to marshal Java objects into a binary blob. from("file://foo/bar"). unmarshal().serialization(). to("activemq:Some.Queue"); DependenciesThis data format is provided in camel-core so no additional dependencies is needed. JAXBJAXB is a Data Format which uses the JAXB2 XML marshalling standard which is included in Java 6 to unmarshal an XML payload into Java objects or to marshal Java objects into an XML payload. Using the Java DSLFor example the following uses a named DataFormat of jaxb which is configured with a number of Java package names to initialize the JAXBContext. You can if you prefer use a named reference to a data format which can then be defined in your Registry such as via your Spring XML file. e.g. Using Spring XMLThe following example shows how to use JAXB to unmarshal using Spring configuring the jaxb data type It is possible to use this data format with more than one context path. You can specify context path using Partial marshalling/unmarshallingThis feature is new to Camel 2.2.0. FragmentThis feature is new to Camel 2.8.0. Ignoring the NonXML CharacterThis feature is new to Camel 2.2.0.
This feature has been tested with Woodstox 3.2.9 and Sun JDK 1.6 StAX implementation. New for Camel 2.12.1 The following example shows using the Spring DSL and also enabling Camel's NonXML filtering: Working with the ObjectFactoryIf you use XJC to create the java class from the schema, you will get an ObjectFactory for you JAXB context. Since the ObjectFactory uses JAXBElement to hold the reference of the schema and element instance value, jaxbDataformat will ignore the JAXBElement by default and you will get the element instance value instead of the JAXBElement object form the unmarshaled message body. Setting encodingYou can set the encoding option to use when marshalling. Its the In this Spring DSL we have defined to use Controlling namespace prefix mappingAvailable as of Camel 2.11 When marshalling using JAXB or SOAP then the JAXB implementation will automatic assign namespace prefixes, such as ns2, ns3, ns4 etc. To control this mapping, Camel allows you to refer to a map which contains the desired mapping. Notice this requires having JAXB-RI 2.1 or better (from SUN) on the classpath, as the mapping functionality is dependent on the implementation of JAXB, whether its supported. For example in Spring XML we can define a Map with the mapping. In the mapping file below, we map SOAP to use soap as prefix. While our custom namespace "http://www.mycompany.com/foo/2" is not using any prefix. To use this in JAXB or SOAP you refer to this map, using the Schema validationAvailable as of Camel 2.11 The JAXB Data Format supports validation by marshalling and unmarshalling from/to XML. Your can use the prefix classpath:, file:* or *http: to specify how the resource should by resolved. You can separate multiple schema files by using the ',' character. Camel 2.11.0 and 2.11.1 has a known issue by validation multiple Using the Java DSL, you can configure it in the following way: You can do the same using the XML DSL: Camel will create and pool the underling Schema LocationAvailable as of Camel 2.14 The JAXB Data Format supports to specify the SchemaLocation when marshaling the XML. Using the Java DSL, you can configure it in the following way: You can do the same using the XML DSL: Marshal data that is already XMLAvailable as of Camel 2.14.1 The JAXB marshaller requires that the message body is JAXB compatible, eg its a JAXBElement, eg a java instance that has JAXB annotations, or extend JAXBElement. There can be situations where the message body is already in XML, eg from a String type. There is a new option mustBeJAXBElement you can set to false, to relax this check, so the JAXB marshaller only attempts to marshal JAXBElements (javax.xml.bind.JAXBIntrospector#isElement returns true). And in those situations the marshaller fallbacks to marshal the message body as-is.XmlRootElement objectsAvailable as of Camel 2.17.2 The JAXB Data Format option objectFactory has a default value equals to false. This is related to a performance degrading. For more information look at the issue CAMEL-10043 For the marshalling of non-XmlRootElement JaxB objects you'll need to call JaxbDataFormat#setObjectFactory(true) DependenciesTo use JAXB in your camel routes you need to add the a dependency on camel-jaxb which implements this data format. If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions). XmlBeansXmlBeans is a Data Format which uses the XmlBeans library to unmarshal an XML payload into Java objects or to marshal Java objects into an XML payload. from("activemq:My.Queue"). unmarshal().xmlBeans(). to("mqseries:Another.Queue"); DependenciesTo use XmlBeans in your camel routes you need to add the dependency on camel-xmlbeans which implements this data format. If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions). <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-xmlbeans</artifactId> <version>x.x.x</version> <!-- use the same version as your Camel core version --> </dependency> XStreamXStream is a Data Format which uses the XStream library to marshal and unmarshal Java objects to and from XML. To use XStream in your camel routes you need to add the a dependency on camel-xstream which implements this data format. Maven users will need to add the following dependency to their Using the Java DSLIf you would like to configure the XMLInputFactory and XMLOutputFactoryThe XStream library uses the The Factory is discovered using this algorithm: How to set the XML encoding in Xstream DataFormat?From Camel 2.2.0, you can set the encoding of XML in Xstream DataFormat by setting the Exchange's property with the key Setting the type permissions of Xstream DataFormatIn Camel, one can always use its own processing step in the route to filter and block certain XML documents to be routed to the XStream's unmarhall step. From Camel 2.16.1, 2.15.5, you can set XStream's type permissions to automatically allow or deny the instantiation of certain types. The default type permissions setting used by Camel denies all types except for those from java.lang and java.util packages. This setting can be changed by setting System property org.apache.camel.xstream.permissions. Its value is a string of comma-separated permission terms, each representing a type being allowed or denied, depending on whether the term is prefixed with '+' (note '+' may be omitted) or with '-', respectively. Each term may contain a wildcard character '*'. For example, value "-*,java.lang.*,java.util.*" indicates denying all types except for java.lang.* and java.util.* classes. Setting this value to an empty string "" reverts to the default XStream's type permissions handling which denies certain blacklisted classes and allow others. The type permissions setting can be extended at an individual XStream DataFormat instance by setting its type permissions property. CSVThe CSV Data Format uses Apache Commons CSV to handle CSV payloads (Comma Separated Values) such as those exported/imported by Excel. As of Camel 2.15.0, it now uses the Apache Commons CSV 1.1 which is based on a completely different set of options. Available options until Camel 2.15
Available options as of Camel 2.15
Marshalling a Map to CSVThe component allows you to marshal a Java Map (or any other message type that can be converted in a Map) into a CSV payload.
Unmarshalling a CSV message into a Java ListUnmarshalling will transform a CSV messsage into a Java List with CSV file lines (containing another List with all the field values). An example: we have a CSV file with names of persons, their IQ and their current activity. Jack Dalton, 115, mad at Averell Joe Dalton, 105, calming Joe William Dalton, 105, keeping Joe from killing Averell Averell Dalton, 80, playing with Rantanplan Lucky Luke, 120, capturing the Daltons We can now use the CSV component to unmarshal this file: from("file:src/test/resources/?fileName=daltons.csv&noop=true") .unmarshal().csv() .to("mock:daltons"); The resulting message will contain a List<List<String>> data = (List<List<String>>) exchange.getIn().getBody(); for (List<String> line : data) { LOG.debug(String.format("%s has an IQ of %s and is currently %s", line.get(0), line.get(1), line.get(2))); } Marshalling a List<Map> to CSVAvailable as of Camel 2.1 If you have multiple rows of data you want to be marshalled into CSV format you can now store the message payload as a File Poller of CSV, then unmarshalingGiven a bean which can handle the incoming data... MyCsvHandler.java // Some comments here public void doHandleCsvData(List<List<String>> csvData) { // do magic here } ... your route then looks as follows <route> <!-- poll every 10 seconds --> <from uri="file:///some/path/to/pickup/csvfiles?delete=true&consumer.delay=10000" /> <unmarshal><csv /></unmarshal> <to uri="bean:myCsvHandler?method=doHandleCsvData" /> </route> Marshaling with a pipe as delimiter
Using autogenColumns, configRef and strategyRef attributes inside XML DSLAvailable as of Camel 2.9.2 / 2.10 and deleted for Camel 2.15 You can customize the CSV Data Format to make use of your own <route> <from uri="direct:start" /> <marshal> <!-- make use of a strategy other than the default one which is 'org.apache.commons.csv.CSVStrategy.DEFAULT_STRATEGY' --> <csv autogenColumns="false" delimiter="|" configRef="csvConfig" strategyRef="excelStrategy" /> </marshal> <convertBodyTo type="java.lang.String" /> <to uri="mock:result" /> </route> <bean id="csvConfig" class="org.apache.commons.csv.writer.CSVConfig"> <property name="fields"> <list> <bean class="org.apache.commons.csv.writer.CSVField"> <property name="name" value="orderId" /> </bean> <bean class="org.apache.commons.csv.writer.CSVField"> <property name="name" value="amount" /> </bean> </list> </property> </bean> <bean id="excelStrategy" class="org.springframework.beans.factory.config.FieldRetrievingFactoryBean"> <property name="staticField" value="org.apache.commons.csv.CSVStrategy.EXCEL_STRATEGY" /> </bean> Using skipFirstLine option while unmarshalingAvailable as of Camel 2.10 and deleted for Camel 2.15 You can instruct the CSV Data Format to skip the first line which contains the CSV headers. Using the Spring/XML DSL: <route> <from uri="direct:start" /> <unmarshal> <csv skipFirstLine="true" /> </unmarshal> <to uri="bean:myCsvHandler?method=doHandleCsv" /> </route> Or the Java DSL: CsvDataFormat csv = new CsvDataFormat(); csv.setSkipFirstLine(true); from("direct:start") .unmarshal(csv) .to("bean:myCsvHandler?method=doHandleCsv"); Unmarshaling with a pipe as delimiterUsing the Spring/XML DSL: <route> <from uri="direct:start" /> <unmarshal> <csv delimiter="|" /> </unmarshal> <to uri="bean:myCsvHandler?method=doHandleCsv" /> </route> Or the Java DSL: CsvDataFormat csv = new CsvDataFormat(); CSVStrategy strategy = CSVStrategy.DEFAULT_STRATEGY; strategy.setDelimiter('|'); csv.setStrategy(strategy); from("direct:start") .unmarshal(csv) .to("bean:myCsvHandler?method=doHandleCsv"); CsvDataFormat csv = new CsvDataFormat(); csv.setDelimiter("|"); from("direct:start") .unmarshal(csv) .to("bean:myCsvHandler?method=doHandleCsv"); CsvDataFormat csv = new CsvDataFormat(); CSVConfig csvConfig = new CSVConfig(); csvConfig.setDelimiter(";"); csv.setConfig(csvConfig); from("direct:start") .unmarshal(csv) .to("bean:myCsvHandler?method=doHandleCsv"); Issue in CSVConfig It looks like that CSVConfig csvConfig = new CSVConfig(); csvConfig.setDelimiter(';'); doesn't work. You have to set the delimiter as a String! DependenciesTo use CSV in your Camel routes you need to add a dependency on camel-csv, which implements this data format. If you use Maven you can just add the following to your pom.xml, substituting the version number for the latest and greatest release (see the download page for the latest versions). <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-csv</artifactId> <version>x.x.x</version> </dependency> The String Data Format is a textual based format that supports encoding.
Options
MarshalIn this example we marshal the file content to String object in UTF-8 encoding. from("file://data.csv").marshal().string("UTF-8").to("jms://myqueue"); UnmarshalIn this example we unmarshal the payload from the JMS queue to a String object using UTF-8 encoding, before its processed by the newOrder processor. from("jms://queue/order").unmarshal().string("UTF-8").processRef("newOrder"); DependenciesThis data format is provided in camel-core so no additional dependencies is needed. HL7 DataFormatThe HL7 component ships with a HL7 data format that can be used to marshal or unmarshal HL7 model objects.
To use the data format, simply instantiate an instance and invoke the marshal or unmarshal operation in the route builder: DataFormat hl7 = new HL7DataFormat(); ... from("direct:hl7in").marshal(hl7).to("jms:queue:hl7out"); In the sample above, the HL7 is marshalled from a HAPI Message object to a byte stream and put on a JMS queue. DataFormat hl7 = new HL7DataFormat(); ... from("jms:queue:hl7out").unmarshal(hl7).to("patientLookupService"); Here we unmarshal the byte stream into a HAPI Message object that is passed to our patient lookup service. Serializable messages As of HAPI 2.0 (used by Camel 2.11), the HL7v2 model classes are fully serializable. So you can put HL7v2 messages directly into a JMS queue (i.e. without calling Segment separators As of Camel 2.11, Charset As of Camel 2.14.1, both
There is a shorthand syntax in Camel for well-known data formats that are commonly used. from("direct:hl7in").marshal().hl7().to("jms:queue:hl7out"); from("jms:queue:hl7out").unmarshal().hl7().to("patientLookupService");
EDI DataFormatWe encourage end users to look at the Smooks which supports EDI and Camel natively. Flatpack DataFormatThe Flatpack component ships with the Flatpack data format that can be used to format between fixed width or delimited text messages to a
Notice: The Flatpack library does currently not support header and trailers for the marshal operation. OptionsThe data format has the following options:
UsageTo use the data format, simply instantiate an instance and invoke the marshal or unmarshal operation in the route builder: FlatpackDataFormat fp = new FlatpackDataFormat(); fp.setDefinition(new ClassPathResource("INVENTORY-Delimited.pzmap.xml")); ... from("file:order/in").unmarshal(df).to("seda:queue:neworder"); The sample above will read files from the FlatpackDataFormat df = new FlatpackDataFormat(); df.setDefinition(new ClassPathResource("PEOPLE-FixedLength.pzmap.xml")); df.setFixed(true); df.setIgnoreFirstRecord(false); from("seda:people").marshal(df).convertBodyTo(String.class).to("jms:queue:people"); In the code above we marshal the data from a Object representation as a DependenciesTo use Flatpack in your camel routes you need to add the a dependency on camel-flatpack which implements this data format. If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions). <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-flatpack</artifactId> <version>x.x.x</version> </dependency> JSONJSON is a Data Format to marshal and unmarshal Java objects to and from JSON. For JSON to object marshalling, Camel provides integration with three popular JSON libraries:
Every library requires adding the special camel component (see "Dependency..." paragraphs further down). By default Camel uses the XStream library. As of Camel 2.10, Camel supports direct, bi-directional JSON <=> XML conversions via the camel-xmljson data format, which is documented separately. Using JSON Data Format With the |
Option | Default | Description |
---|---|---|
compressionLevel | null | To specify a specific compression Level use |
Marshal
In this example we marshal a regular text/XML payload to a compressed payload employing zip compression Deflater.BEST_COMPRESSION
and send it an ActiveMQ queue called MY_QUEUE.
from("direct:start").marshal().zip(Deflater.BEST_COMPRESSION).to("activemq:queue:MY_QUEUE");
Alternatively if you would like to use the default setting you could send it as
from("direct:start").marshal().zip().to("activemq:queue:MY_QUEUE");
Unmarshal
In this example we unmarshal a zipped payload from an ActiveMQ queue called MY_QUEUE to its original format, and forward it for processing to the UnZippedMessageProcessor. Note that the compression Level employed during the marshalling should be identical to the one employed during unmarshalling to avoid errors.
from("activemq:queue:MY_QUEUE").unmarshal().zip().process(new UnZippedMessageProcessor());
Dependencies
This data format is provided in camel-core so no additional dependencies are needed.
TidyMarkup
TidyMarkup is a Data Format that uses the TagSoup to tidy up HTML. It can be used to parse ugly HTML and return it as pretty wellformed HTML.
Camel eats our own -dog food- soap
TidyMarkup only supports the unmarshal operation as we really don't want to turn well formed HTML into ugly HTML
Java DSL Example
An example where the consumer provides some HTML
from("file://site/inbox").unmarshal().tidyMarkup().to("file://site/blogs");
Spring XML Example
The following example shows how to use TidyMarkup to unmarshal using Spring
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring"> <route> <from uri="file://site/inbox"/> <unmarshal> <tidyMarkup/> </unmarshal> <to uri="file://site/blogs"/> </route> </camelContext>
Dependencies
To use TidyMarkup in your camel routes you need to add the a dependency on camel-tagsoup which implements this data format.
If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).
<dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-tagsoup</artifactId> <version>x.x.x</version> </dependency>
Bindy
The goal of this component is to allow the parsing/binding of non-structured data (or to be more precise non-XML data)
to/from Java Beans that have binding mappings defined with annotations. Using Bindy, you can bind data from sources such as :
- CSV records,
- Fixed-length records,
- FIX messages,
- or almost any other non-structured data
to one or many Plain Old Java Object (POJO). Bindy converts the data according to the type of the java property. POJOs can be linked together with one-to-many relationships available in some cases. Moreover, for data type like Date, Double, Float, Integer, Short, Long and BigDecimal, you can provide the pattern to apply during the formatting of the property.
For the BigDecimal numbers, you can also define the precision and the decimal or grouping separators.
Type | Format Type | Pattern example | Link |
---|---|---|---|
Date | DateFormat | "dd-MM-yyyy" | http://java.sun.com/j2se/1.5.0/docs/api/java/text/SimpleDateFormat.html |
Decimal* | Decimalformat | "##.###.###" | http://java.sun.com/j2se/1.5.0/docs/api/java/text/DecimalFormat.html |
Decimal* = Double, Integer, Float, Short, Long
Format supported
This first release only support comma separated values fields and key value pair fields (e.g. : FIX messages).
To work with camel-bindy, you must first define your model in a package (e.g. com.acme.model) and for each model class (e.g. Order, Client, Instrument, ...) add the required annotations (described hereafter) to the Class or field.
Multiple models
If you use multiple models, each model has to be placed in it's own package to prevent unpredictable results.
From Camel 2.16 onwards this is no longer the case, as you can safely have multiple models in the same package, as you configure bindy using class names instead of package names now.
Annotations
The annotations created allow to map different concept of your model to the POJO like :
- Type of record (csv, key value pair (e.g. FIX message), fixed length ...),
- Link (to link object in another object),
- DataField and their properties (int, type, ...),
- KeyValuePairField (for key = value format like we have in FIX financial messages),
- Section (to identify header, body and footer section),
- OneToMany
This section will describe them :
1. CsvRecord
The CsvRecord annotation is used to identified the root class of the model. It represents a record = a line of a CSV file and can be linked to several children model classes.
Annotation name | Record type | Level |
---|---|---|
CsvRecord | csv | Class |
Parameter name | type | Info |
---|---|---|
separator | string | mandatory - can be ',' or ';' or 'anything'. This value is interpreted as a regular expression. If you want to use a sign which has a special meaning in regular expressions, e.g. the '|' sign, than you have to mask it, like ' |
skipFirstLine | boolean | optional - default value = false - allow to skip the first line of the CSV file |
crlf | string | optional - possible values = WINDOWS,UNIX,MAC, or custom; default value = WINDOWS - allow to define the carriage return character to use. If you specify a value other than the three listed before, the value you enter (custom) will be used as the CRLF character(s) |
generateHeaderColumns | boolean | optional - default value = false - uses to generate the header columns of the CSV generates |
autospanLine | boolean | Camel 2.13/2.12.2: optional - default value = false - if enabled then the last column is auto spanned to end of line, for example if its a comment, etc this allows the line to contain all characters, also the delimiter char. |
isOrdered | boolean | optional - default value = false - allow to change the order of the fields when CSV is generated |
quote | String | Camel 2.8.3/2.9: option - allow to specify a quote character of the fields when CSV is generated |
|
| This annotation is associated to the root class of the model and must be declared one time. |
quoting | boolean | Camel 2.11:optional - default value = false - Indicate if the values must be quoted when marshaling when CSV is generated. |
case 1 : separator = ','
The separator used to segregate the fields in the CSV record is ',' :
10, J, Pauline, M, XD12345678, Fortis Dynamic 15/15, 2500, USD,08-01-2009
@CsvRecord( separator = "," ) public Class Order { ... }
case 2 : separator = ';'
Compare to the previous case, the separator here is ';' instead of ',' :
10; J; Pauline; M; XD12345678; Fortis Dynamic 15/15; 2500; USD; 08-01-2009
@CsvRecord( separator = ";" ) public Class Order { ... }
case 3 : separator = '|'
Compare to the previous case, the separator here is '|' instead of ';' :
10| J| Pauline| M| XD12345678| Fortis Dynamic 15/15| 2500| USD| 08-01-2009
@CsvRecord( separator = "\\|" ) public Class Order { ... }
case 4 : separator = '\",\"'
Applies for Camel 2.8.2 or older
When the field to be parsed of the CSV record contains ',' or ';' which is also used as separator, we whould find another strategy
to tell camel bindy how to handle this case. To define the field containing the data with a comma, you will use simple or double quotes
as delimiter (e.g : '10', 'Street 10, NY', 'USA' or "10", "Street 10, NY", "USA").
Remark : In this case, the first and last character of the line which are a simple or double quotes will removed by bindy
"10","J","Pauline"," M","XD12345678","Fortis Dynamic 15,15" 2500","USD","08-01-2009"
@CsvRecord( separator = "\",\"" ) public Class Order { ... }
From Camel 2.8.3/2.9 or never bindy will automatic detect if the record is enclosed with either single or double quotes and automatic remove those quotes when unmarshalling from CSV to Object. Therefore do not include the quotes in the separator, but simple do as below:
"10","J","Pauline"," M","XD12345678","Fortis Dynamic 15,15" 2500","USD","08-01-2009"
@CsvRecord( separator = "," ) public Class Order { ... }
Notice that if you want to marshal from Object to CSV and use quotes, then you need to specify which quote character to use, using the quote
attribute on the @CsvRecord as shown below:
@CsvRecord( separator = ",", quote = "\"" ) public Class Order { ... }
case 5 : separator & skipfirstline
The feature is interesting when the client wants to have in the first line of the file, the name of the data fields :
order id, client id, first name, last name, isin code, instrument name, quantity, currency, date
To inform bindy that this first line must be skipped during the parsing process, then we use the attribute :
@CsvRecord(separator = ",", skipFirstLine = true) public Class Order { ... }
case 6 : generateHeaderColumns
To add at the first line of the CSV generated, the attribute generateHeaderColumns must be set to true in the annotation like this :
@CsvRecord( generateHeaderColumns = true ) public Class Order { ... }
As a result, Bindy during the unmarshaling process will generate CSV like this :
order id, client id, first name, last name, isin code, instrument name, quantity, currency, date
10, J, Pauline, M, XD12345678, Fortis Dynamic 15/15, 2500, USD,08-01-2009
case 7 : carriage return
If the platform where camel-bindy will run is not Windows but Macintosh or Unix, than you can change the crlf property like this. Three values are available : WINDOWS, UNIX or MAC
@CsvRecord(separator = ",", crlf="MAC") public Class Order { ... }
Additionally, if for some reason you need to add a different line ending character, you can opt to specify it using the crlf parameter. In the following example, we can end the line with a comma followed by the newline character:
@CsvRecord(separator = ",", crlf=",\n") public Class Order { ... }
case 8 : isOrdered
Sometimes, the order to follow during the creation of the CSV record from the model is different from the order used during the parsing. Then, in this case, we can use the attribute isOrdered = true to indicate this in combination with attribute 'position' of the DataField annotation.
@CsvRecord(isOrdered = true) public Class Order { @DataField(pos = 1, position = 11) private int orderNr; @DataField(pos = 2, position = 10) private String clientNr; ... }
Remark : pos is used to parse the file, stream while positions is used to generate the CSV
2. Link
The link annotation will allow to link objects together.
Annotation name | Record type | Level |
---|---|---|
Link | all | Class & Property |
Parameter name | type | Info |
---|---|---|
linkType | LinkType | optional - by default the value is LinkType.oneToOne - so you are not obliged to mention it |
|
| Only one-to-one relation is allowed. |
e.g : If the model Class Client is linked to the Order class, then use annotation Link in the Order class like this :
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @Link private Client client; ...
AND for the class Client :
@Link public class Client { ... }
3. DataField
The DataField annotation defines the property of the field. Each datafield is identified by its position in the record, a type (string, int, date, ...) and optionally of a pattern
Annotation name | Record type | Level |
---|---|---|
DataField | all | Property |
Parameter name | type | Info |
---|---|---|
pos | int | mandatory - The input position of the field. digit number starting from 1 to ... - See the position parameter. |
pattern | string | optional - default value = "" - will be used to format Decimal, Date, ... |
length | int | optional - represents the length of the field for fixed length format |
precision | int | optional - represents the precision to be used when the Decimal number will be formatted/parsed |
pattern | string | optional - default value = "" - is used by the Java formatter (SimpleDateFormat by example) to format/validate data. If using pattern, then setting locale on bindy data format is recommended. Either set to a known locale such as "us" or use "default" to use platform default locale. Notice that "default" requires Camel 2.14/2.13.3/2.12.5. |
position | int | optional - must be used when the position of the field in the CSV generated (output message) must be different compare to input position (pos). See the pos parameter. |
required | boolean | optional - default value = "false" |
trim | boolean | optional - default value = "false" |
defaultValue | string | Camel 2.10: optional - default value = "" - defines the field's default value when the respective CSV field is empty/not available |
impliedDecimalSeparator | boolean | Camel 2.11: optional - default value = "false" - Indicates if there is a decimal point implied at a specified location |
lengthPos | int | Camel 2.11: optional - can be used to identify a data field in a fixed-length record that defines the fixed length for this field |
delimiter | string | Camel 2.11: optional - can be used to demarcate the end of a variable-length field within a fixed-length record |
case 1 : pos
This parameter/attribute represents the position of the field in the csv record
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @DataField(pos = 5) private String isinCode; ... }
As you can see in this example the position starts at '1' but continues at '5' in the class Order. The numbers from '2' to '4' are defined in the class Client (see here after).
public class Client { @DataField(pos = 2) private String clientNr; @DataField(pos = 3) private String firstName; @DataField(pos = 4) private String lastName; ... }
case 2 : pattern
The pattern allows to enrich or validates the format of your data
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @DataField(pos = 5) private String isinCode; @DataField(name = "Name", pos = 6) private String instrumentName; @DataField(pos = 7, precision = 2) private BigDecimal amount; @DataField(pos = 8) private String currency; @DataField(pos = 9, pattern = "dd-MM-yyyy") -- pattern used during parsing or when the date is created private Date orderDate; ... }
case 3 : precision
The precision is helpful when you want to define the decimal part of your number
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @Link private Client client; @DataField(pos = 5) private String isinCode; @DataField(name = "Name", pos = 6) private String instrumentName; @DataField(pos = 7, precision = 2) -- precision private BigDecimal amount; @DataField(pos = 8) private String currency; @DataField(pos = 9, pattern = "dd-MM-yyyy") private Date orderDate; ... }
case 4 : Position is different in output
The position attribute will inform bindy how to place the field in the CSV record generated. By default, the position used corresponds to the position defined with the attribute 'pos'. If the position is different (that means that we have an asymetric processus comparing marshaling from unmarshaling) than we can use 'position' to indicate this.
Here is an example
@CsvRecord(separator = ",") public class Order { @CsvRecord(separator = ",", isOrdered = true) public class Order { // Positions of the fields start from 1 and not from 0 @DataField(pos = 1, position = 11) private int orderNr; @DataField(pos = 2, position = 10) private String clientNr; @DataField(pos = 3, position = 9) private String firstName; @DataField(pos = 4, position = 8) private String lastName; @DataField(pos = 5, position = 7) private String instrumentCode; @DataField(pos = 6, position = 6) private String instrumentNumber; ... }
This attribute of the annotation @DataField must be used in combination with attribute isOrdered = true of the annotation @CsvRecord
case 5 : required
If a field is mandatory, simply use the attribute 'required' setted to true
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @DataField(pos = 2, required = true) private String clientNr; @DataField(pos = 3, required = true) private String firstName; @DataField(pos = 4, required = true) private String lastName; ... }
If this field is not present in the record, than an error will be raised by the parser with the following information :
Some fields are missing (optional or mandatory), line :
case 6 : trim
If a field has leading and/or trailing spaces which should be removed before they are processed, simply use the attribute 'trim' setted to true
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1, trim = true) private int orderNr; @DataField(pos = 2, trim = true) private Integer clientNr; @DataField(pos = 3, required = true) private String firstName; @DataField(pos = 4) private String lastName; ... }
case 7 : defaultValue
If a field is not defined then uses the value indicated by the defaultValue attribute
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @DataField(pos = 2) private Integer clientNr; @DataField(pos = 3, required = true) private String firstName; @DataField(pos = 4, defaultValue = "Barin") private String lastName; ... }
This attribute is only applicable to optional fields.
4. FixedLengthRecord
The FixedLengthRecord annotation is used to identified the root class of the model. It represents a record = a line of a file/message containing data fixed length formatted and can be linked to several children model classes. This format is a bit particular beause data of a field can be aligned to the right or to the left.
When the size of the data does not fill completely the length of the field, we can then add 'padd' characters.
Annotation name | Record type | Level |
---|---|---|
FixedLengthRecord | fixed | Class |
Parameter name | type | Info |
---|---|---|
crlf | string | optional - possible values = WINDOWS,UNIX,MAC, or custom; default value = WINDOWS - allow to define the carriage return character to use. If you specify a value other than the three listed before, the value you enter (custom) will be used as the CRLF character(s) |
paddingChar | char | mandatory - default value = ' ' |
length | int | mandatory = size of the fixed length record |
hasHeader | boolean | Camel 2.11 - optional - Indicates that the record(s) of this type may be preceded by a single header record at the beginning of the file / stream |
hasFooter | boolean | Camel 2.11 - optional - Indicates that the record(s) of this type may be followed by a single footer record at the end of the file / stream |
skipHeader | boolean | Camel 2.11 - optional - Configures the data format to skip marshalling / unmarshalling of the header record. Configure this parameter on the primary record (e.g., not the header or footer). |
skipFooter | boolean | Camel 2.11 - optional - Configures the data format to skip marshalling / unmarshalling of the footer record Configure this parameter on the primary record (e.g., not the header or footer).. |
isHeader | boolean | Camel 2.11 - optional - Identifies this FixedLengthRecord as a header record |
isFooter | boolean | Camel 2.11 - optional - Identifies this FixedLengthRecords as a footer record |
ignoreTrailingChars | boolean | Camel 2.11.1 - optional - Indicates that characters beyond the last mapped filed can be ignored when unmarshalling / parsing. |
|
| This annotation is associated to the root class of the model and must be declared one time. |
The hasHeader/hasFooter parameters are mutually exclusive with isHeader/isFooter. A record may not be both a header/footer and a primary fixed-length record.
case 1 : Simple fixed length record
This simple example shows how to design the model to parse/format a fixed message
10A9PaulineMISINXD12345678BUYShare2500.45USD01-08-2009
@FixedLengthRecord(length=54, paddingChar=' ') public static class Order { @DataField(pos = 1, length=2) private int orderNr; @DataField(pos = 3, length=2) private String clientNr; @DataField(pos = 5, length=7) private String firstName; @DataField(pos = 12, length=1, align="L") private String lastName; @DataField(pos = 13, length=4) private String instrumentCode; @DataField(pos = 17, length=10) private String instrumentNumber; @DataField(pos = 27, length=3) private String orderType; @DataField(pos = 30, length=5) private String instrumentType; @DataField(pos = 35, precision = 2, length=7) private BigDecimal amount; @DataField(pos = 42, length=3) private String currency; @DataField(pos = 45, length=10, pattern = "dd-MM-yyyy") private Date orderDate; ...
case 2 : Fixed length record with alignment and padding
This more elaborated example show how to define the alignment for a field and how to assign a padding character which is ' ' here''
10A9 PaulineM ISINXD12345678BUYShare2500.45USD01-08-2009
@FixedLengthRecord(length=60, paddingChar=' ') public static class Order { @DataField(pos = 1, length=2) private int orderNr; @DataField(pos = 3, length=2) private String clientNr; @DataField(pos = 5, length=9) private String firstName; @DataField(pos = 14, length=5, align="L") // align text to the LEFT zone of the block private String lastName; @DataField(pos = 19, length=4) private String instrumentCode; @DataField(pos = 23, length=10) private String instrumentNumber; @DataField(pos = 33, length=3) private String orderType; @DataField(pos = 36, length=5) private String instrumentType; @DataField(pos = 41, precision = 2, length=7) private BigDecimal amount; @DataField(pos = 48, length=3) private String currency; @DataField(pos = 51, length=10, pattern = "dd-MM-yyyy") private Date orderDate; ...
case 3 : Field padding
Sometimes, the default padding defined for record cannnot be applied to the field as we have a number format where we would like to padd with '0' instead of ' '. In this case, you can use in the model the attribute paddingField to set this value.
10A9 PaulineM ISINXD12345678BUYShare000002500.45USD01-08-2009
@FixedLengthRecord(length = 65, paddingChar = ' ') public static class Order { @DataField(pos = 1, length = 2) private int orderNr; @DataField(pos = 3, length = 2) private String clientNr; @DataField(pos = 5, length = 9) private String firstName; @DataField(pos = 14, length = 5, align = "L") private String lastName; @DataField(pos = 19, length = 4) private String instrumentCode; @DataField(pos = 23, length = 10) private String instrumentNumber; @DataField(pos = 33, length = 3) private String orderType; @DataField(pos = 36, length = 5) private String instrumentType; @DataField(pos = 41, precision = 2, length = 12, paddingChar = '0') private BigDecimal amount; @DataField(pos = 53, length = 3) private String currency; @DataField(pos = 56, length = 10, pattern = "dd-MM-yyyy") private Date orderDate; ...
case 4: Fixed length record with delimiter
Fixed-length records sometimes have delimited content within the record. The firstName and lastName fields are delimited with the '^' character in the following example:
10A9Pauline^M^ISINXD12345678BUYShare000002500.45USD01-08-2009
@FixedLengthRecord() public static class Order { @DataField(pos = 1, length = 2) private int orderNr; @DataField(pos = 2, length = 2) private String clientNr; @DataField(pos = 3, delimiter = "^") private String firstName; @DataField(pos = 4, delimiter = "^") private String lastName; @DataField(pos = 5, length = 4) private String instrumentCode; @DataField(pos = 6, length = 10) private String instrumentNumber; @DataField(pos = 7, length = 3) private String orderType; @DataField(pos = 8, length = 5) private String instrumentType; @DataField(pos = 9, precision = 2, length = 12, paddingChar = '0') private BigDecimal amount; @DataField(pos = 10, length = 3) private String currency; @DataField(pos = 11, length = 10, pattern = "dd-MM-yyyy") private Date orderDate;
As of Camel 2.11 the 'pos' value(s) in a fixed-length record may optionally be defined using ordinal, sequential values instead of precise column numbers.
case 5 : Fixed length record with record-defined field length
Occasionally a fixed-length record may contain a field that define the expected length of another field within the same record. In the following example the length of the instrumentNumber field value is defined by the value of instrumentNumberLen field in the record.
10A9Pauline^M^ISIN10XD12345678BUYShare000002500.45USD01-08-2009
@FixedLengthRecord() public static class Order { @DataField(pos = 1, length = 2) private int orderNr; @DataField(pos = 2, length = 2) private String clientNr; @DataField(pos = 3, delimiter = "^") private String firstName; @DataField(pos = 4, delimiter = "^") private String lastName; @DataField(pos = 5, length = 4) private String instrumentCode; @DataField(pos = 6, length = 2, align = "R", paddingChar = '0') private int instrumentNumberLen; @DataField(pos = 7, lengthPos=6) private String instrumentNumber; @DataField(pos = 8, length = 3) private String orderType; @DataField(pos = 9, length = 5) private String instrumentType; @DataField(pos = 10, precision = 2, length = 12, paddingChar = '0') private BigDecimal amount; @DataField(pos = 11, length = 3) private String currency; @DataField(pos = 12, length = 10, pattern = "dd-MM-yyyy") private Date orderDate;
case 6 : Fixed length record with header and footer
Bindy will discover fixed-length header and footer records that are configured as part of the model – provided that the annotated classes exist either in the same package as the primary @FixedLengthRecord class, or within one of the configured scan packages. The following text illustrates two fixed-length records that are bracketed by a header record and footer record.
101-08-2009
10A9 PaulineM ISINXD12345678BUYShare000002500.45USD01-08-2009
10A9 RichN ISINXD12345678BUYShare000002700.45USD01-08-2009
9000000002
@FixedLengthRecord(hasHeader = true, hasFooter = true) public class Order { @DataField(pos = 1, length = 2) private int orderNr; @DataField(pos = 2, length = 2) private String clientNr; @DataField(pos = 3, length = 9) private String firstName; @DataField(pos = 4, length = 5, align = "L") private String lastName; @DataField(pos = 5, length = 4) private String instrumentCode; @DataField(pos = 6, length = 10) private String instrumentNumber; @DataField(pos = 7, length = 3) private String orderType; @DataField(pos = 8, length = 5) private String instrumentType; @DataField(pos = 9, precision = 2, length = 12, paddingChar = '0') private BigDecimal amount; @DataField(pos = 10, length = 3) private String currency; @DataField(pos = 11, length = 10, pattern = "dd-MM-yyyy") private Date orderDate; ... } @FixedLengthRecord(isHeader = true) public class OrderHeader { @DataField(pos = 1, length = 1) private int recordType = 1; @DataField(pos = 2, length = 10, pattern = "dd-MM-yyyy") private Date recordDate; ... } @FixedLengthRecord(isFooter = true) public class OrderFooter { @DataField(pos = 1, length = 1) private int recordType = 9; @DataField(pos = 2, length = 9, align = "R", paddingChar = '0') private int numberOfRecordsInTheFile; ... }
case 7 : Skipping content when parsing a fixed length record. (Camel 2.11.1)
It is common to integrate with systems that provide fixed-length records containing more information than needed for the target use case. It is useful in this situation to skip the declaration and parsing of those fields that we do not need. To accomodate this, Bindy will skip forward to the next mapped field within a record if the 'pos' value of the next declared field is beyond the cursor position of the last parsed field. Using absolute 'pos' locations for the fields of interest (instead of ordinal values) causes Bindy to skip content between two fields.
Similarly, it is possible that none of the content beyond some field is of interest. In this case, you can tell Bindy to skip parsing of everything beyond the last mapped field by setting the ignoreTrailingChars property on the @FixedLengthRecord declaration.
@FixedLengthRecord(ignoreTrailingChars = true) public static class Order { @DataField(pos = 1, length = 2) private int orderNr; @DataField(pos = 3, length = 2) private String clientNr; ... any characters that appear beyond the last mapped field will be ignored }
5. Message
The Message annotation is used to identified the class of your model who will contain key value pairs fields. This kind of format is used mainly in Financial Exchange Protocol Messages (FIX). Nevertheless, this annotation can be used for any other format where data are identified by keys. The key pair values are separated each other by a separator which can be a special character like a tab delimitor (unicode representation : \u0009) or a start of heading (unicode representation : \u0001)
"FIX information"
More information about FIX can be found on this web site : http://www.fixprotocol.org/. To work with FIX messages, the model must contain a Header and Trailer classes linked to the root message class which could be a Order class. This is not mandatory but will be very helpful when you will use camel-bindy in combination with camel-fix which is a Fix gateway based on quickFix project http://www.quickfixj.org/.
Annotation name | Record type | Level |
---|---|---|
Message | key value pair | Class |
Parameter name | type | Info |
---|---|---|
pairSeparator | string | mandatory - can be '=' or ';' or 'anything' |
keyValuePairSeparair | string | mandatory - can be '\u0001', '\u0009', '#' or 'anything' |
crlf | string | optional - possible values = WINDOWS,UNIX,MAC, or custom; default value = WINDOWS - allow to define the carriage return character to use. If you specify a value other than the three listed before, the value you enter (custom) will be used as the CRLF character(s) |
type | string | optional - define the type of message (e.g. FIX, EMX, ...) |
version | string | optional - version of the message (e.g. 4.1) |
isOrdered | boolean | optional - default value = false - allow to change the order of the fields when FIX message is generated |
|
| This annotation is associated to the message class of the model and must be declared one time. |
case 1 : separator = 'u0001'
The separator used to segregate the key value pair fields in a FIX message is the ASCII '01' character or in unicode format '\u0001'. This character must be escaped a second time to avoid a java runtime error. Here is an example :
8=FIX.4.1 9=20 34=1 35=0 49=INVMGR 56=BRKR 1=BE.CHM.001 11=CHM0001-01 22=4 ...
and how to use the annotation
@Message(keyValuePairSeparator = "=", pairSeparator = "\u0001", type="FIX", version="4.1") public class Order { ... }
Look at test cases
The ASCII character like tab, ... cannot be displayed in WIKI page. So, have a look to the test case of camel-bindy to see exactly how the FIX message looks like (src\test\data\fix\fix.txt) and the Order, Trailer, Header classes (src\test\java\org\apache\camel\dataformat\bindy\model\fix\simple\Order.java)
6. KeyValuePairField
The KeyValuePairField annotation defines the property of a key value pair field. Each KeyValuePairField is identified by a tag (= key) and its value associated, a type (string, int, date, ...), optionaly a pattern and if the field is required
Annotation name | Record type | Level |
---|---|---|
KeyValuePairField | Key Value Pair - FIX | Property |
Parameter name | type | Info |
---|---|---|
tag | int | mandatory - digit number identifying the field in the message - must be unique |
pattern | string | optional - default value = "" - will be used to format Decimal, Date, ... |
precision | int | optional - digit number - represents the precision to be used when the Decimal number will be formatted/parsed |
position | int | optional - must be used when the position of the key/tag in the FIX message must be different |
required | boolean | optional - default value = "false" |
impliedDecimalSeparator | boolean | Camel 2.11: optional - default value = "false" - Indicates if there is a decimal point implied at a specified location |
case 1 : tag
This parameter represents the key of the field in the message
@Message(keyValuePairSeparator = "=", pairSeparator = "\u0001", type="FIX", version="4.1") public class Order { @Link Header header; @Link Trailer trailer; @KeyValuePairField(tag = 1) // Client reference private String Account; @KeyValuePairField(tag = 11) // Order reference private String ClOrdId; @KeyValuePairField(tag = 22) // Fund ID type (Sedol, ISIN, ...) private String IDSource; @KeyValuePairField(tag = 48) // Fund code private String SecurityId; @KeyValuePairField(tag = 54) // Movement type ( 1 = Buy, 2 = sell) private String Side; @KeyValuePairField(tag = 58) // Free text private String Text; ... }
case 2 : Different position in output
If the tags/keys that we will put in the FIX message must be sorted according to a predefine order, then use the attribute 'position' of the annotation @KeyValuePairField
@Message(keyValuePairSeparator = "=", pairSeparator = "\\u0001", type = "FIX", version = "4.1", isOrdered = true) public class Order { @Link Header header; @Link Trailer trailer; @KeyValuePairField(tag = 1, position = 1) // Client reference private String account; @KeyValuePairField(tag = 11, position = 3) // Order reference private String clOrdId; ... }
7. Section
In FIX message of fixed length records, it is common to have different sections in the representation of the information : header, body and section. The purpose of the annotation @Section is to inform bindy about which class of the model represents the header (= section 1), body (= section 2) and footer (= section 3)
Only one attribute/parameter exists for this annotation.
Annotation name | Record type | Level |
---|---|---|
Section | FIX | Class |
Parameter name | type | Info |
---|---|---|
number | int | digit number identifying the section position |
case 1 : Section
A. Definition of the header section
@Section(number = 1) public class Header { @KeyValuePairField(tag = 8, position = 1) // Message Header private String beginString; @KeyValuePairField(tag = 9, position = 2) // Checksum private int bodyLength; ... }
B. Definition of the body section
@Section(number = 2) @Message(keyValuePairSeparator = "=", pairSeparator = "\\u0001", type = "FIX", version = "4.1", isOrdered = true) public class Order { @Link Header header; @Link Trailer trailer; @KeyValuePairField(tag = 1, position = 1) // Client reference private String account; @KeyValuePairField(tag = 11, position = 3) // Order reference private String clOrdId;
C. Definition of the footer section
@Section(number = 3) public class Trailer { @KeyValuePairField(tag = 10, position = 1) // CheckSum private int checkSum; public int getCheckSum() { return checkSum; }
8. OneToMany
The purpose of the annotation @OneToMany is to allow to work with a List<?> field defined a POJO class or from a record containing repetitive groups.
Restrictions OneToMany
Be careful, the one to many of bindy does not allow to handle repetitions defined on several levels of the hierarchy
The relation OneToMany ONLY WORKS in the following cases :
- Reading a FIX message containing repetitive groups (= group of tags/keys)
- Generating a CSV with repetitive data
Annotation name | Record type | Level |
---|---|---|
OneToMany | all | property |
Parameter name | type | Info |
---|---|---|
mappedTo | string | optional - string - class name associated to the type of the List<Type of the Class> |
case 1 : Generating CSV with repetitive data
Here is the CSV output that we want :
Claus,Ibsen,Camel in Action 1,2010,35
Claus,Ibsen,Camel in Action 2,2012,35
Claus,Ibsen,Camel in Action 3,2013,35
Claus,Ibsen,Camel in Action 4,2014,35
Remark : the repetitive data concern the title of the book and its publication date while first, last name and age are common
and the classes used to modeling this. The Author class contains a List of Book.
@CsvRecord(separator=",") public class Author { @DataField(pos = 1) private String firstName; @DataField(pos = 2) private String lastName; @OneToMany private List<Book> books; @DataField(pos = 5) private String Age; ... public class Book { @DataField(pos = 3) private String title; @DataField(pos = 4) private String year;
Very simple isn't it !!!
case 2 : Reading FIX message containing group of tags/keys
Here is the message that we would like to process in our model :
"8=FIX 4.19=2034=135=049=INVMGR56=BRKR"
"1=BE.CHM.00111=CHM0001-0158=this is a camel - bindy test"
"22=448=BE000124567854=1"
"22=548=BE000987654354=2"
"22=648=BE000999999954=3"
"10=220"
tags 22, 48 and 54 are repeated
and the code
public class Order { @Link Header header; @Link Trailer trailer; @KeyValuePairField(tag = 1) // Client reference private String account; @KeyValuePairField(tag = 11) // Order reference private String clOrdId; @KeyValuePairField(tag = 58) // Free text private String text; @OneToMany(mappedTo = "org.apache.camel.dataformat.bindy.model.fix.complex.onetomany.Security") List<Security> securities; ... public class Security { @KeyValuePairField(tag = 22) // Fund ID type (Sedol, ISIN, ...) private String idSource; @KeyValuePairField(tag = 48) // Fund code private String securityCode; @KeyValuePairField(tag = 54) // Movement type ( 1 = Buy, 2 = sell) private String side;
Using the Java DSL
The next step consists in instantiating the DataFormat bindy class associated with this record type and providing Java package name(s) as parameter.
For example the following uses the class BindyCsvDataFormat
(who correspond to the class associated with the CSV record type) which is configured with "com.acme.model"
package name to initialize the model objects configured in this package.
// Camel 2.15 or older (configure by package name) DataFormat bindy = new BindyCsvDataFormat("com.acme.model"); // Camel 2.16 onwards (configure by class name) DataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class);
Setting locale
Bindy supports configuring the locale on the dataformat, such as
// Camel 2.15 or older (configure by package name) BindyCsvDataFormat bindy = new BindyCsvDataFormat("com.acme.model"); // Camel 2.16 onwards (configure by class name) BindyCsvDataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class); bindy.setLocale("us");
Or to use the platform default locale then use "default" as the locale name. Notice this requires Camel 2.14/2.13.3/2.12.5.
// Camel 2.15 or older (configure by package name) BindyCsvDataFormat bindy = new BindyCsvDataFormat("com.acme.model"); // Camel 2.16 onwards (configure by class name) BindyCsvDataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class); bindy.setLocale("default");
for older releases you can set it using Java code as shown
// Camel 2.15 or older (configure by package name) BindyCsvDataFormat bindy = new BindyCsvDataFormat("com.acme.model"); // Camel 2.16 onwards (configure by class name) BindyCsvDataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class); bindy.setLocale(Locale.getDefault().getISO3Country());
Unmarshaling
from("file://inbox") .unmarshal(bindy) .to("direct:handleOrders");
Alternatively, you can use a named reference to a data format which can then be defined in your Registry e.g. your Spring XML file:
from("file://inbox") .unmarshal("myBindyDataFormat") .to("direct:handleOrders");
The Camel route will pick-up files in the inbox directory, unmarshall CSV records into a collection of model objects and send the collection
to the route referenced by 'handleOrders'.
The collection returned is a List of Map objects. Each Map within the list contains the model objects that were marshalled out of each line of the CSV. The reason behind this is that each line can correspond to more than one object. This can be confusing when you simply expect one object to be returned per line.
Each object can be retrieve using its class name.
List<Map<String, Object>> unmarshaledModels = (List<Map<String, Object>>) exchange.getIn().getBody(); int modelCount = 0; for (Map<String, Object> model : unmarshaledModels) { for (String className : model.keySet()) { Object obj = model.get(className); LOG.info("Count : " + modelCount + ", " + obj.toString()); } modelCount++; } LOG.info("Total CSV records received by the csv bean : " + modelCount);
Assuming that you want to extract a single Order object from this map for processing in a route, you could use a combination of a Splitter and a Processor as per the following:
from("file://inbox") .unmarshal(bindy) .split(body()) .process(new Processor() { public void process(Exchange exchange) throws Exception { Message in = exchange.getIn(); Map<String, Object> modelMap = (Map<String, Object>) in.getBody(); in.setBody(modelMap.get(Order.class.getCanonicalName())); } }) .to("direct:handleSingleOrder") .end();
Take care of the fact that Bindy uses CHARSET_NAME property or the CHARSET_NAME header as define in the Exchange interface to do a characterset conversion of the inputstream received for unmarshalling. In some producers (e.g. file-endpoint) you can define a characterset. The characterset conversion can already been done by this producer. Sometimes you need to remove this property or header from the exchange before sending it to the unmarshal. If you don't remove it the conversion might be done twice which might lead to unwanted results.
from("file://inbox?charset=Cp922") .removeProperty(Exchange.CHARSET_NAME) .unmarshal("myBindyDataFormat") .to("direct:handleOrders");
Marshaling
To generate CSV records from a collection of model objects, you create the following route :
from("direct:handleOrders") .marshal(bindy) .to("file://outbox")
Using Spring XML
This is really easy to use Spring as your favorite DSL language to declare the routes to be used for camel-bindy. The following example shows two routes where the first will pick-up records from files, unmarshal the content and bind it to their model. The result is then send to a pojo (doing nothing special) and place them into a queue.
The second route will extract the pojos from the queue and marshal the content to generate a file containing the csv record. The example above is for using Camel 2.16 onwards.
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd"> <!-- Queuing engine - ActiveMq - work locally in mode virtual memory --> <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent"> <property name="brokerURL" value="vm://localhost:61616"/> </bean> <camelContext xmlns="http://camel.apache.org/schema/spring"> <dataFormats> <bindy id="bindyDataformat" type="Csv" classType="org.apache.camel.bindy.model.Order"/> </dataFormats> <route> <from uri="file://src/data/csv/?noop=true" /> <unmarshal ref="bindyDataformat" /> <to uri="bean:csv" /> <to uri="activemq:queue:in" /> </route> <route> <from uri="activemq:queue:in" /> <marshal ref="bindyDataformat" /> <to uri="file://src/data/csv/out/" /> </route> </camelContext> </beans>
Be careful
Please verify that your model classes implements serializable otherwise the queue manager will raise an error
Dependencies
To use Bindy in your camel routes you need to add the a dependency on camel-bindy which implements this data format.
If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).
<dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-bindy</artifactId> <version>x.x.x</version> </dependency>
XMLSecurity Data Format
The XMLSecurity Data Format facilitates encryption and decryption of XML payloads at the Document, Element, and Element Content levels (including simultaneous multi-node encryption/decryption using XPath). To sign messages using the XML Signature specification, please see the Camel XML Security component.
The encryption capability is based on formats supported using the Apache XML Security (Santuario) project. Symmetric encryption/decryption is currently supported using Triple-DES and AES (128, 192, and 256) encryption formats. Additional formats can be easily added later as needed. This capability allows Camel users to encrypt/decrypt payloads while being dispatched or received along a route.
Available as of Camel 2.9
The XMLSecurity Data Format supports asymmetric key encryption. In this encryption model a symmetric key is generated and used to perform XML content encryption or decryption. This "content encryption key" is then itself encrypted using an asymmetric encryption algorithm that leverages the recipient's public key as the "key encryption key". Use of an asymmetric key encryption algorithm ensures that only the holder of the recipient's private key can access the generated symmetric encryption key. Thus, only the private key holder can decode the message. The XMLSecurity Data Format handles all of the logic required to encrypt and decrypt the message content and encryption key(s) using asymmetric key encryption.
The XMLSecurity Data Format also has improved support for namespaces when processing the XPath queries that select content for encryption. A namespace definition mapping can be included as part of the data format configuration. This enables true namespace matching, even if the prefix values in the XPath query and the target xml document are not equivalent strings.
Basic Options
Option | Default | Description |
---|---|---|
|
| The XPath reference to the XML Element selected for encryption/decryption. If no tag is specified, the entire payload is encrypted/decrypted. |
|
| A boolean value to specify whether the XML Element is to be encrypted or the contents of the XML Element
|
|
| A String used as passPhrase to encrypt/decrypt content. The passPhrase has to be provided. If no passPhrase is specified, a default passPhrase is used. The passPhrase needs to be put together in conjunction with the appropriate encryption algorithm. For example using |
|
| The cipher algorithm to be used for encryption/decryption of the XML message content. The available choices are:
|
|
| A map of namespace values indexed by prefix. The index values must match the prefixes used in the |
Asymmetric Encryption Options
These options can be applied in addition to relevant the Basic options to use asymmetric key encryption.
Option | Default | Description |
---|---|---|
|
| The key alias to be used when retrieving the recipient's public or private key from a KeyStore when performing asymmetric key encryption or decryption. |
| Camel 2.12 | The cipher algorithm to be used for encryption/decryption of the asymmetric key. The available choices are:
|
|
| Configuration options for creating and loading a KeyStore instance that represents the sender's trustStore or recipient's keyStore. |
|
| Camel 2.10.2 / 2.11: The password to be used for retrieving the private key from the KeyStore. This key is used for asymmetric decryption. |
| XMLCipher.SHA1 | Camel 2.12 The digest algorithm to use with the RSA OAEP algorithm. The available choices are:
|
| EncryptionConstants.MGF1_SHA1 | Camel 2.12 The MGF Algorithm to use with the RSA OAEP algorithm. The available choices are:
|
addKeyValueForEncryptedKey | true | Camel 2.14.1 Whether to add the public key used to encrypt the session key as a KeyValue in the EncryptedKey structure or not. |
Key Cipher Algorithm
As of Camel 2.12.0, the default Key Cipher Algorithm is now XMLCipher.RSA_OAEP instead of XMLCipher.RSA_v1dot5. Usage of XMLCipher.RSA_v1dot5 is discouraged due to various attacks. Requests that use RSA v1.5 as the key cipher algorithm will be rejected unless it has been explicitly configured as the key cipher algorithm.
Marshal
In order to encrypt the payload, the marshal
processor needs to be applied on the route followed by the secureXML()
tag.
Unmarshal
In order to decrypt the payload, the unmarshal
processor needs to be applied on the route followed by the secureXML()
tag.
Examples
Given below are several examples of how marshalling could be performed at the Document, Element, and Content levels.
Full Payload encryption/decryption
from("direct:start") .marshal().secureXML() .unmarshal().secureXML() .to("direct:end");
Partial Payload Content Only encryption/decryption
String tagXPATH = "//cheesesites/italy/cheese"; boolean secureTagContent = true; ... from("direct:start") .marshal().secureXML(tagXPATH, secureTagContent) .unmarshal().secureXML(tagXPATH, secureTagContent) .to("direct:end");
Partial Multi Node Payload Content Only encryption/decryption
String tagXPATH = "//cheesesites/*/cheese"; boolean secureTagContent = true; ... from("direct:start") .marshal().secureXML(tagXPATH, secureTagContent) .unmarshal().secureXML(tagXPATH, secureTagContent) .to("direct:end");
Partial Payload Content Only encryption/decryption with choice of passPhrase(password)
String tagXPATH = "//cheesesites/italy/cheese"; boolean secureTagContent = true; ... String passPhrase = "Just another 24 Byte key"; from("direct:start") .marshal().secureXML(tagXPATH, secureTagContent, passPhrase) .unmarshal().secureXML(tagXPATH, secureTagContent, passPhrase) .to("direct:end");
Partial Payload Content Only encryption/decryption with passPhrase(password) and Algorithm
import org.apache.xml.security.encryption.XMLCipher; .... String tagXPATH = "//cheesesites/italy/cheese"; boolean secureTagContent = true; String passPhrase = "Just another 24 Byte key"; String algorithm= XMLCipher.TRIPLEDES; from("direct:start") .marshal().secureXML(tagXPATH, secureTagContent, passPhrase, algorithm) .unmarshal().secureXML(tagXPATH, secureTagContent, passPhrase, algorithm) .to("direct:end");
Partial Payload Content with Namespace support
Java DSL
final Map<String, String> namespaces = new HashMap<String, String>(); namespaces.put("cust", "http://cheese.xmlsecurity.camel.apache.org/"); final KeyStoreParameters tsParameters = new KeyStoreParameters(); tsParameters.setPassword("password"); tsParameters.setResource("sender.ts"); context.addRoutes(new RouteBuilder() { public void configure() { from("direct:start") .marshal().secureXML("//cust:cheesesites/italy", namespaces, true, "recipient", testCypherAlgorithm, XMLCipher.RSA_v1dot5, tsParameters) .to("mock:encrypted"); } }
Spring XML
A namespace prefix that is defined as part of the camelContext
definition can be re-used in context within the data format secureTag
attribute of the secureXML
element.
<camelContext id="springXmlSecurityDataFormatTestCamelContext" xmlns="http://camel.apache.org/schema/spring" xmlns:cheese="http://cheese.xmlsecurity.camel.apache.org/"> <route> <from uri="direct://start"/> <marshal> <secureXML secureTag="//cheese:cheesesites/italy" secureTagContents="true"/> </marshal> ...
Asymmetric Key Encryption
Spring XML Sender
<!-- trust store configuration --> <camel:keyStoreParameters id="trustStoreParams" resource="./sender.ts" password="password"/> <camelContext id="springXmlSecurityDataFormatTestCamelContext" xmlns="http://camel.apache.org/schema/spring" xmlns:cheese="http://cheese.xmlsecurity.camel.apache.org/"> <route> <from uri="direct://start"/> <marshal> <secureXML secureTag="//cheese:cheesesites/italy" secureTagContents="true" xmlCipherAlgorithm="http://www.w3.org/2001/04/xmlenc#aes128-cbc" keyCipherAlgorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" recipientKeyAlias="recipient" keyOrTrustStoreParametersId="trustStoreParams"/> </marshal> ...
Spring XML Recipient
<!-- key store configuration --> <camel:keyStoreParameters id="keyStoreParams" resource="./recipient.ks" password="password" /> <camelContext id="springXmlSecurityDataFormatTestCamelContext" xmlns="http://camel.apache.org/schema/spring" xmlns:cheese="http://cheese.xmlsecurity.camel.apache.org/"> <route> <from uri="direct://encrypted"/> <unmarshal> <secureXML secureTag="//cheese:cheesesites/italy" secureTagContents="true" xmlCipherAlgorithm="http://www.w3.org/2001/04/xmlenc#aes128-cbc" keyCipherAlgorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" recipientKeyAlias="recipient" keyOrTrustStoreParametersId="keyStoreParams" keyPassword="privateKeyPassword" /> </unmarshal> ...
Dependencies
This data format is provided within the camel-xmlsecurity component.
gzip
/gunzip
tool. Messages marshalled using GZip compression can be unmarshalled using GZip decompression just prior to being consumed at the endpoint. The compression capability is quite useful when you deal with large XML and Text based payloads or when you read messages previously comressed using gzip
tool.
Options
There are no options provided for this data format.
Marshal
In this example we marshal a regular text/XML payload to a compressed payload employing gzip compression format and send it an ActiveMQ queue called MY_QUEUE.
from("direct:start").marshal().gzip().to("activemq:queue:MY_QUEUE");
Unmarshal
In this example we unmarshal a gzipped payload from an ActiveMQ queue called MY_QUEUE to its original format, and forward it for processing to the UnGZippedMessageProcessor
.
from("activemq:queue:MY_QUEUE").unmarshal().gzip().process(new UnGZippedMessageProcessor());
Dependencies
This data format is provided in camel-core so no additional dependencies is needed.
Castor
Available as of Camel 2.1
Castor is a Data Format which uses the Castor XML library to unmarshal an XML payload into Java objects or to marshal Java objects into an XML payload.
As usually you can use either Java DSL or Spring XML to work with Castor Data Format.
Using the Java DSL
from("direct:order"). marshal().castor(). to("activemq:queue:order");
For example the following uses a named DataFormat of Castor which uses default Castor data binding features.
CastorDataFormat castor = new CastorDataFormat (); from("activemq:My.Queue"). unmarshal(castor). to("mqseries:Another.Queue");
If you prefer to use a named reference to a data format which can then be defined in your Registry such as via your Spring XML file. e.g.
from("activemq:My.Queue"). unmarshal("mycastorType"). to("mqseries:Another.Queue");
If you want to override default mapping schema by providing a mapping file you can set it as follows.
CastorDataFormat castor = new CastorDataFormat (); castor.setMappingFile("mapping.xml");
Also if you want to have more control on Castor Marshaller and Unmarshaller you can access them as below.
castor.getMarshaller(); castor.getUnmarshaller();
Using Spring XML
The following example shows how to use Castor to unmarshal using Spring configuring the castor data type
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring"> <route> <from uri="direct:start"/> <unmarshal> <castor validation="true" /> </unmarshal> <to uri="mock:result"/> </route> </camelContext>
This example shows how to configure the data type just once and reuse it on multiple routes. You have to set the <castor> element directly in <camelContext>.
<camelContext> <camelContext id="camel" xmlns="http://camel.apache.org/schema/spring"> <dataFormats> <castor id="myCastor"/> </dataFormats> <route> <from uri="direct:start"/> <marshal ref="myCastor"/> <to uri="direct:marshalled"/> </route> <route> <from uri="direct:marshalled"/> <unmarshal ref="myCastor"/> <to uri="mock:result"/> </route> </camelContext>
Options
Castor supports the following options
Option | Type | Default | Description |
---|---|---|---|
encoding | String | UTF-8 | Encoding to use when marshalling an Object to XML |
validation | Boolean | false | Whether validation is turned on or off. |
mappingFile | String | null | Path to a Castor mapping file to load from the classpath. |
packages | String[] | null | Add additional packages to Castor XmlContext |
classNames | String[] | null | Add additional class names to Castor XmlContext |
Dependencies
To use Castor in your camel routes you need to add the a dependency on camel-castor which implements this data format.
If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).
<dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-castor</artifactId> <version>x.x.x</version> </dependency>
Protobuf - Protocol Buffers
"Protocol Buffers - Google's data interchange format"
Available from Camel 2.2
Camel provides a Data Format to serialse between Java and the Protocol Buffer protocol. The project's site details why you may wish to choose this format over xml. Protocol Buffer is language-neutral and platform-neutral, so messages produced by your Camel routes may be consumed by other language implementations.
API Site
Protobuf Implementation
Protobuf Java Tutorial
Protobuf overview
This quick overview of how to use Protobuf. For more detail see the complete tutorial
Defining the proto format
The first step is to define the format for the body of your exchange. This is defined in a .proto file as so:
package org.apache.camel.component.protobuf; option java_package = "org.apache.camel.component.protobuf"; option java_outer_classname = "AddressBookProtos"; message Person { required string name = 1; required int32 id = 2; optional string email = 3; enum PhoneType { MOBILE = 0; HOME = 1; WORK = 2; } message PhoneNumber { required string number = 1; optional PhoneType type = 2 [default = HOME]; } repeated PhoneNumber phone = 4; } message AddressBook { repeated Person person = 1; }
Generating Java classes
The Protobuf SDK provides a compiler which will generate the Java classes for the format we defined in our .proto file. You can run the compiler for any additional supported languages you require.
protoc --java_out=. ./addressbook.proto
This will generate a single Java class named AddressBookProtos which contains inner classes for Person and AddressBook. Builders are also implemented for you. The generated classes implement com.google.protobuf.Message which is required by the serialisation mechanism. For this reason it important that only these classes are used in the body of your exchanges. Camel will throw an exception on route creation if you attempt to tell the Data Format to use a class that does not implement com.google.protobuf.Message. Use the generated builders to translate the data from any of your existing domain classes.
Java DSL
You can use create the ProtobufDataFormat instance and pass it to Camel DataFormat marshal and unmarsha API like this.
ProtobufDataFormat format = new ProtobufDataFormat(Person.getDefaultInstance()); from("direct:in").marshal(format); from("direct:back").unmarshal(format).to("mock:reverse");
Or use the DSL protobuf() passing the unmarshal default instance or default instance class name like this.
// You don't need to specify the default instance for protobuf marshaling from("direct:marshal").marshal().protobuf(); from("direct:unmarshalA").unmarshal(). protobuf("org.apache.camel.dataformat.protobuf.generated.AddressBookProtos$Person"). to ("mock:reverse"); from("direct:unmarshalB").unmarshal().protobuf(Person.getDefaultInstance()).to("mock:reverse");
Spring DSL
The following example shows how to use Castor to unmarshal using Spring configuring the protobuf data type
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring"> <route> <from uri="direct:start"/> <unmarshal> <protobuf instanceClass="org.apache.camel.dataformat.protobuf.generated.AddressBookProtos$Person" /> </unmarshal> <to uri="mock:result"/> </route> </camelContext>
Dependencies
To use Protobuf in your camel routes you need to add the a dependency on camel-protobuf which implements this data format.
If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).
<dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-protobuf</artifactId> <version>2.2.0</version> </dependency>
SOAP DataFormat
Available as of Camel 2.3
SOAP is a Data Format which uses JAXB2 and JAX-WS annotations to marshal and unmarshal SOAP payloads. It provides the basic features of Apache CXF without need for the CXF Stack.
Supported SOAP versions
SOAP 1.1 is supported by default. SOAP 1.2 is supported from Camel 2.11 onwards.
Namespace prefix mapping
ElementNameStrategy
An element name strategy is used for two purposes. The first is to find a xml element name for a given object and soap action when marshaling the object into a SOAP message. The second is to find an Exception class for a given soap fault name.
Strategy | Usage |
---|---|
QNameStrategy | Uses a fixed qName that is configured on instantiation. Exception lookup is not supported |
TypeNameStrategy | Uses the name and namespace from the @XMLType annotation of the given type. If no namespace is set then package-info is used. Exception lookup is not supported |
ServiceInterfaceStrategy | Uses information from a webservice interface to determine the type name and to find the exception class for a SOAP fault |
If you have generated the web service stub code with cxf-codegen or a similar tool then you probably will want to use the ServiceInterfaceStrategy. In the case you have no annotated service interface you should use QNameStrategy or TypeNameStrategy.
Using the Java DSL
The following example uses a named DataFormat of soap which is configured with the package com.example.customerservice to initialize the JAXBContext. The second parameter is the ElementNameStrategy. The route is able to marshal normal objects as well as exceptions. (Note the below just sends a SOAP Envelope to a queue. A web service provider would actually need to be listening to the queue for a SOAP call to actually occur, in which case it would be a one way SOAP request. If you need request reply then you should look at the next example.)
SoapJaxbDataFormat soap = new SoapJaxbDataFormat("com.example.customerservice", new ServiceInterfaceStrategy(CustomerService.class)); from("direct:start") .marshal(soap) .to("jms:myQueue");
See also
As the SOAP dataformat inherits from the JAXB dataformat most settings apply here as well
Using SOAP 1.2
Available as of Camel 2.11
SoapJaxbDataFormat soap = new SoapJaxbDataFormat("com.example.customerservice", new ServiceInterfaceStrategy(CustomerService.class)); soap.setVersion("1.2"); from("direct:start") .marshal(soap) .to("jms:myQueue");
When using XML DSL there is a version attribute you can set on the <soapjaxb> element.
<!-- Defining a ServiceInterfaceStrategy for retrieving the element name when marshalling --> <bean id="myNameStrategy" class="org.apache.camel.dataformat.soap.name.ServiceInterfaceStrategy"> <constructor-arg value="com.example.customerservice.CustomerService"/> <constructor-arg value="true"/> </bean>
And in the Camel route
<route> <from uri="direct:start"/> <marshal> <soapjaxb contentPath="com.example.customerservice" version="1.2" elementNameStrategyRef="myNameStrategy"/> </marshal> <to uri="jms:myQueue"/> </route>
Multi-part Messages
Available as of Camel 2.8.1
Multi-part SOAP messages are supported by the ServiceInterfaceStrategy. The ServiceInterfaceStrategy must be initialized with a service interface definition that is annotated in accordance with JAX-WS 2.2 and meets the requirements of the Document Bare style. The target method must meet the following criteria, as per the JAX-WS specification: 1) it must have at most one in
or in/out
non-header parameter, 2) if it has a return type other than void
it must have no in/out
or out
non-header parameters, 3) if it it has a return type of void
it must have at most one in/out
or out
non-header parameter.
The ServiceInterfaceStrategy should be initialized with a boolean parameter that indicates whether the mapping strategy applies to the request parameters or response parameters.
ServiceInterfaceStrategy strat = new ServiceInterfaceStrategy(com.example.customerservice.multipart.MultiPartCustomerService.class, true); SoapJaxbDataFormat soapDataFormat = new SoapJaxbDataFormat("com.example.customerservice.multipart", strat);
Multi-part Request
The payload parameters for a multi-part request are initiazlied using a BeanInvocation
object that reflects the signature of the target operation. The camel-soap DataFormat maps the content in the BeanInvocation
to fields in the SOAP header and body in accordance with the JAX-WS mapping when the marshal()
processor is invoked.
BeanInvocation beanInvocation = new BeanInvocation(); // Identify the target method beanInvocation.setMethod(MultiPartCustomerService.class.getMethod("getCustomersByName", GetCustomersByName.class, com.example.customerservice.multipart.Product.class)); // Populate the method arguments GetCustomersByName getCustomersByName = new GetCustomersByName(); getCustomersByName.setName("Dr. Multipart"); Product product = new Product(); product.setName("Multiuse Product"); product.setDescription("Useful for lots of things."); Object[] args = new Object[] {getCustomersByName, product}; // Add the arguments to the bean invocation beanInvocation.setArgs(args); // Set the bean invocation object as the message body exchange.getIn().setBody(beanInvocation);
Multi-part Response
A multi-part soap response may include an element in the soap body and will have one or more elements in the soap header. The camel-soap DataFormat will unmarshall the element in the soap body (if it exists) and place it onto the body of the out message in the exchange. Header elements will not be marshaled into their JAXB mapped object types. Instead, these elements are placed into the camel out message header org.apache.camel.dataformat.soap.UNMARSHALLED_HEADER_LIST
. The elements will appear either as element instance values, or as JAXBElement values, depending upon the setting for the ignoreJAXBElement
property. This property is inherited from camel-jaxb.
You can also have the camel-soap DataFormate ignore header content all-together by setting the ignoreUnmarshalledHeaders
value to true
.
Holder Object mapping
JAX-WS specifies the use of a type-parameterized javax.xml.ws.Holder
object for In/Out
and Out
parameters. A Holder
object may be used when building the BeanInvocation
, or you may use an instance of the parameterized-type directly. The camel-soap DataFormat marshals Holder values in accordance with the JAXB mapping for the class of the Holder
's value. No mapping is provided for Holder
objects in an unmarshalled response.
Examples
Webservice client
The following route supports marshalling the request and unmarshalling a response or fault.
String WS_URI = "cxf://http://myserver/customerservice?serviceClass=com.example.customerservice&dataFormat=MESSAGE"; SoapJaxbDataFormat soapDF = new SoapJaxbDataFormat("com.example.customerservice", new ServiceInterfaceStrategy(CustomerService.class)); from("direct:customerServiceClient") .onException(Exception.class) .handled(true) .unmarshal(soapDF) .end() .marshal(soapDF) .to(WS_URI) .unmarshal(soapDF);
The below snippet creates a proxy for the service interface and makes a SOAP call to the above route.
import org.apache.camel.Endpoint; import org.apache.camel.component.bean.ProxyHelper; ... Endpoint startEndpoint = context.getEndpoint("direct:customerServiceClient"); ClassLoader classLoader = Thread.currentThread().getContextClassLoader(); // CustomerService below is the service endpoint interface, *not* the javax.xml.ws.Service subclass CustomerService proxy = ProxyHelper.createProxy(startEndpoint, classLoader, CustomerService.class); GetCustomersByNameResponse response = proxy.getCustomersByName(new GetCustomersByName());
Webservice Server
Using the following route sets up a webservice server that listens on jms queue customerServiceQueue and processes requests using the class CustomerServiceImpl. The customerServiceImpl of course should implement the interface CustomerService. Instead of directly instantiating the server class it could be defined in a spring context as a regular bean.
SoapJaxbDataFormat soapDF = new SoapJaxbDataFormat("com.example.customerservice", new ServiceInterfaceStrategy(CustomerService.class)); CustomerService serverBean = new CustomerServiceImpl(); from("jms://queue:customerServiceQueue") .onException(Exception.class) .handled(true) .marshal(soapDF) .end() .unmarshal(soapDF) .bean(serverBean) .marshal(soapDF);
Dependencies
To use the SOAP dataformat in your camel routes you need to add the following dependency to your pom.
<dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-soap</artifactId> <version>2.3.0</version> </dependency>
Crypto
Available as of Camel 2.3
PGP Available as of Camel 2.9
The Crypto Data Format integrates the Java Cryptographic Extension into Camel, allowing simple and flexible encryption and decryption of messages using Camel's familiar marshall and unmarshal formatting mechanism. It assumes marshalling to mean encryption to cyphertext and unmarshalling to mean decryption back to the original plaintext. This data format implements only symmetric (shared-key) encryption and decyption.
Options
Name | Type | Default | Description |
---|---|---|---|
|
|
| The JCE algorithm name indicating the cryptographic algorithm that will be used. |
|
|
| A JCE AlgorithmParameterSpec used to initialize the Cipher. |
|
|
| the size of the buffer used in the signature process. |
|
|
| The name of the JCE Security Provider that should be used. |
|
|
| A byte array containing the Initialization Vector that will be used to initialize the Cipher. |
|
|
| Flag indicating that the configured IV should be inlined into the encrypted data stream. |
|
|
| The JCE algorithm name indicating the Message Authentication algorithm. |
|
|
| Flag indicating that a Message Authentication Code should be calculated and appended to the encrypted data. |
Basic Usage
At its most basic all that is required to encrypt/decrypt an exchange is a shared secret key. If one or more instances of the Crypto data format are configured with this key the format can be used to encrypt the payload in one route (or part of one) and decrypted in another. For example, using the Java DSL as follows:
Specifying the Encryption Algorithm
Changing the algorithm is a matter of supplying the JCE algorithm name. If you change the algorithm you will need to use a compatible key.
Specifying an Initialization Vector
Some crypto algorithms, particularly block algorithms, require configuration with an initial block of data known as an Initialization Vector. In the JCE this is passed as an AlgorithmParameterSpec when the Cipher is initialized. To use such a vector with the CryptoDataFormat you can configure it with a byte[] containing the required data e.g.
- http://en.wikipedia.org/wiki/Initialization_vector
- http://www.herongyang.com/Cryptography/
- http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation
Hashed Message Authentication Codes (HMAC)
To avoid attacks against the encrypted data while it is in transit the CryptoDataFormat can also calculate a Message Authentication Code for the encrypted exchange contents based on a configurable MAC algorithm. The calculated HMAC is appended to the stream after encryption. It is separated from the stream in the decryption phase. The MAC is recalculated and verified against the transmitted version to insure nothing was tampered with in transit.For more information on Message Authentication Codes see http://en.wikipedia.org/wiki/HMAC
Supplying Keys Dynamically
When using a Recipient list or similar EIP the recipient of an exchange can vary dynamically. Using the same key across all recipients may neither be feasible or desirable. It would be useful to be able to specify keys dynamically on a per exchange basis. The exchange could then be dynamically enriched with the key of its target recipient before being processed by the data format. To facilitate this the DataFormat allow for keys to be supplied dynamically via the message headers below
CryptoDataFormat.KEY
"CamelCryptoKey"
PGP Message
The PGP Data Formater can create and decrypt/verify PGP Messages of the following PGP packet structure (entries in brackets are optional and ellipses indicate repetition, comma represents sequential composition, and vertical bar separates alternatives):
Public Key Encrypted Session Key ..., Symmetrically Encrypted Data | Sym. Encrypted and Integrity Protected Data, (Compressed Data,) (One Pass Signature ...,) Literal Data, (Signature ...,)
Since Camel 2.16.0 the Compressed Data packet is optional, before it was mandatory.
PGPDataFormat Options
Name | Type | Default | Description |
---|---|---|---|
|
|
| The user ID of the key in the PGP keyring used during encryption. See also option |
|
|
| Since camel 2.12.2: PGP allows to encrypt the symmetric key by several asymmetric public receiver keys. You can specify here the User IDs or parts of User IDs of several public keys contained in the PGP keyring. If you just have one User ID, then you can also use the option |
|
|
| Password used when opening the private key (not used for encryption). |
|
|
| Filename of the keyring; must be accessible as a classpath resource (but you can specify a location in the file system by using the "file:" prefix). |
|
|
| Since camel 2.12.1: encryption keyring; you can not set the keyFileName and encryptionKeyRing at the same time. |
|
|
| Since Camel 2.11.0; optional User ID of the key in the PGP keyring used for signing (during encryption) or signature verification (during decryption). During the signature verification process the specified User ID restricts the public keys from the public keyring which can be used for the verification. If no User ID is specified for the signature verficiation then any public key in the public keyring can be used for the verification. Can also be only a part of a user ID. For example, if the user ID is "Test User <test@camel.com>" then you can use the part "Test User" or "<test@camel.com>" to address the User ID. |
|
|
| Since Camel 2.12.3: optional list of User IDs of the key in the PGP keyring used for signing (during encryption) or signature verification (during decryption). You can specify here the User IDs or parts of User IDs of several keys contained in the PGP keyring. If you just have one User ID, then you can also use the option |
|
|
| Since Camel 2.11.0: optional password used when opening the private key used for signing (during encryption). |
|
|
| Since Camel 2.11.0: optional filename of the keyring to use for signing (during encryption) or for signature verification (during decryption); must be accessible as a classpath resource (but you can specify a location in the file system by using the "file:" prefix). |
|
|
| Since camel 2.12.1: signature keyring; you can not set the signatureKeyFileName and signatureKeyRing at the same time. |
|
|
| Since camel 2.12.2: symmetric key encryption algorithm; possible values are defined in |
|
|
| Since camel 2.12.2: compression algorithm; possible values are defined in |
|
|
| Since camel 2.12.2: signature hash algorithm; possible values are defined in |
|
|
| This option will cause PGP to base64 encode the encrypted text, making it available for copy/paste, etc. |
|
|
| Adds an integrity check/sign into the encryption file. |
|
| Since Camel 2.12.2: provides passphrases corresponding to user Ids. If no passpharase can be found from the option | |
|
|
| Since Camel 2.13.0: controls the behavior for verifying the signature during unmarshaling. There are three values possible:
|
| String | "_CONSOLE" | Since camel 2.15.0: Sets the file name for the literal data packet. Can be overwritten by the header {@link Exchange#FILE_NAME}. " |
| boolean |
| Since Camel 2.16.0: Indicator whether the PGP Message shall be created with or without a Compressed Data packet. If the value is set to false, then no Compressed Data packet is added and the compressionAlgorithm value is ignored. Only used for marshaling. |
PGPDataFormat Message Headers
You can override the PGPDataFormat options by applying below headers into message dynamically.
Name | Type | Description |
---|---|---|
|
| Since Camel 2.11.0; filename of the keyring; will override existing setting directly on the PGPDataFormat. |
|
| Since Camel 2.12.1; the encryption keyring; will override existing setting directly on the PGPDataFormat. |
|
| Since Camel 2.11.0; the User ID of the key in the PGP keyring; will override existing setting directly on the PGPDataFormat. |
|
| Since camel 2.12.2: the User IDs of the key in the PGP keyring; will override existing setting directly on the PGPDataFormat. |
|
| Since Camel 2.11.0; password used when opening the private key; will override existing setting directly on the PGPDataFormat. |
|
| Since Camel 2.11.0; filename of the signature keyring; will override existing setting directly on the PGPDataFormat. |
|
| Since Camel 2.12.1; the signature keyring; will override existing setting directly on the PGPDataFormat. |
|
| Since Camel 2.11.0; the User ID of the signature key in the PGP keyring; will override existing setting directly on the PGPDataFormat. |
CamelPGPDataFormatSignatureKeyUserids | List<String> | Since Camel 2.12.3; the User IDs of the signature keys in the PGP keyring; will override existing setting directly on the PGPDataFormat. |
|
| Since Camel 2.11.0; password used when opening the signature private key; will override existing setting directly on the PGPDataFormat. |
|
| Since Camel 2.12.2; symmetric key encryption algorithm; will override existing setting directly on the PGPDataFormat. |
|
| Since Camel 2.12.2; signature hash algorithm; will override existing setting directly on the PGPDataFormat. |
|
| Since Camel 2.12.2; compression algorithm; will override existing setting directly on the PGPDataFormat. |
| Integer | Since Camel 2.12.3; number of public keys used for encrypting the symmectric key, set by PGPDataFormat during encryptiion process |
| Integer | Since Camel 2.12.3; number of private keys used for creating signatures, set by PGPDataFormat during signing process |
Encrypting with PGPDataFormat
The following sample uses the popular PGP format for encrypting/decrypting files using the Bouncy Castle Java libraries:
To work with the previous example you need the following
- A public keyring file which contains the public keys used to encrypt the data
- A private keyring file which contains the keys used to decrypt the data
- The keyring password
Managing your keyring
To manage the keyring, I use the command line tools, I find this to be the simplest approach in managing the keys. There are also Java libraries available from http://www.bouncycastle.org/java.html if you would prefer to do it that way.
Install the command line utilities on linux
apt-get install gnupg Create your keyring, entering a secure password
gpg --gen-key If you need to import someone elses public key so that you can encrypt a file for them.
gpg --import <filename.key The following files should now exist and can be used to run the example
ls -l ~/.gnupg/pubring.gpg ~/.gnupg/secring.gpg
PGP Decrypting/Verifying of Messages Encrypted/Signed by Different Private/Public Keys
Since Camel 2.12.2.
A PGP Data Formater can decrypt/verify messages which have been encrypted by different public keys or signed by different private keys. Just, provide the corresponding private keys in the secret keyring, the corresponding public keys in the public keyring, and the passphrases in the passphrase accessor.
- The functionality is especially useful to support the key exchange. If you want to exchange the private key for decrypting you can accept for a period of time messages which are either encrypted with the old or new corresponding public key. Or if the sender wants to exchange his signer private key, you can accept for a period of time, the old or new signer key.
- Technical background: The PGP encrypted data contains a Key ID of the public key which was used to encrypt the data. This Key ID can be used to locate the private key in the secret keyring to decrypt the data. The same mechanism is also used to locate the public key for verifying a signature. Therefore you no longer must specify User IDs for the unmarshaling.
Restricting the Signer Identities during PGP Signature Verification
Since Camel 2.12.3.
If you verify a signature you not only want to verify the correctness of the signature but you also want check that the signature comes from a certain identity or a specific set of identities. Therefore it is possible to restrict the number of public keys from the public keyring which can be used for the verification of a signature.
- If the PGP content has several signatures the verification is successful as soon as one signature can be verified.
- If you do not want to restrict the signer identities for verification then do not specify the signature key User IDs. In this case all public keys in the public keyring are taken into account.
Several Signatures in One PGP Data Format
Since Camel 2.12.3.
The PGP specification allows that one PGP data format can contain several signatures from different keys. Since Camel 2.13.3 it is possible to create such kind of PGP content via specifying signature User IDs which relate to several private keys in the secret keyring.
Support of Sub-Keys and Key Flags in PGP Data Format Marshaler
Since Camel 2.12.3.
An OpenPGP V4 key can have a primary key and sub-keys. The usage of the keys is indicated by the so called Key Flags. For example, you can have a primary key with two sub-keys; the primary key shall only be used for certifying other keys (Key Flag 0x01), the first sub-key shall only be used for signing (Key Flag 0x02), and the second sub-key shall only be used for encryption (Key Flag 0x04 or 0x08). The PGP Data Format marshaler takes into account these Key Flags of the primary key and sub-keys in order to determine the right key for signing and encryption. This is necessary because the primary key and its sub-keys have the same User IDs.
Support of Custom Key Accessors
Since Camel 2.13.0.
You can implement custom key accessors for encryption/signing. The above PGPDataFormat class selects in a certain predefined way the keys which should be used for signing/encryption or verifying/decryption. If you have special requirements how your keys should be selected you should use the PGPKeyAccessDataFormat class instead and implement the interfaces PGPPublicKeyAccessor and PGPSecretKeyAccessor as beans. There are default implementations DefaultPGPPublicKeyAccessor and DefaultPGPSecretKeyAccessor which cache the keys, so that not every time the keyring is parsed when the processor is called.
PGPKeyAccessDataFormat has the same options as PGPDataFormat except password, keyFileName, encryptionKeyRing, signaturePassword, signatureKeyFileName, and signatureKeyRing.
Dependencies
To use the Crypto dataformat in your camel routes you need to add the following dependency to your pom.
See Also
Syslog DataFormat
Available as of Camel 2.6
The syslog dataformat is used for working with RFC3164 and RFC5424 messages.
This component supports the following:
- UDP consumption of syslog messages
- Agnostic data format using either plain String objects or SyslogMessage model objects.
- Type Converter from/to SyslogMessage and String
- Integration with the camel-mina component.
- Integration with the camel-netty component.
- Camel 2.14: Encoder and decoder for the camel-netty component.
- Camel 2.14: Support for RFC5424 also.
Maven users will need to add the following dependency to their pom.xml
for this component:
<dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-syslog</artifactId> <version>x.x.x</version> <!-- use the same version as your Camel core version --> </dependency>
RFC3164 Syslog protocol
Syslog uses the user datagram protocol (UDP) 1 as its underlying transport layer mechanism.
The UDP port that has been assigned to syslog is 514.
To expose a Syslog listener service we reuse the existing camel-mina component or camel-netty where we just use the Rfc3164SyslogDataFormat
to marshal and unmarshal messages. Notice that from Camel 2.14 onwards the syslog dataformat is renamed to SyslogDataFormat
.
RFC5424 Syslog protocol
Available as of Camel 2.14
To expose a Syslog listener service we reuse the existing camel-mina component or camel-netty where we just use the SyslogDataFormat
to marshal and unmarshal messages
Exposing a Syslog listener
In our Spring XML file, we configure an endpoint to listen for udp messages on port 10514, note that in netty we disable the defaultCodec, this
will allow a fallback to a NettyTypeConverter and delivers the message as an InputStream:
<camelContext id="myCamel" xmlns="http://camel.apache.org/schema/spring"> <dataFormats> <syslog id="mySyslog"/> </dataFormats> <route> <from uri="netty:udp://localhost:10514?sync=false&allowDefaultCodec=false"/> <unmarshal ref="mySyslog"/> <to uri="mock:stop1"/> </route> </camelContext>
The same route using camel-mina
<camelContext id="myCamel" xmlns="http://camel.apache.org/schema/spring"> <dataFormats> <syslog id="mySyslog"/> </dataFormats> <route> <from uri="mina:udp://localhost:10514"/> <unmarshal ref="mySyslog"/> <to uri="mock:stop1"/> </route> </camelContext>
Sending syslog messages to a remote destination
<camelContext id="myCamel" xmlns="http://camel.apache.org/schema/spring"> <dataFormats> <syslog id="mySyslog"/> </dataFormats> <route> <from uri="direct:syslogMessages"/> <marshal ref="mySyslog"/> <to uri="mina:udp://remotehost:10514"/> </route> </camelContext>