EDI handling in DataPower XB62 - By Anders Wasen an IBM Champion
Daphne de Flavia 270003U0B0 email@example.com |
3 Comments | 5,725 Visits
I figured to share with you a little something about mastering EDI in IBM WebSphere DataPower XB62. We are not talking Yoda-style mastering here but I will take you through the setup and give you a handle on how to get it done…
First you will need to understand a thing or two about EDI, which can be somewhat of a challenge…
Once I got a call from a customer who asked for some help with a message they got from a new business partner. The guy called me up:
Client: “Hey, Anders, we have gotten some weird message”
Me: “Ok, what does it say?”
Client: “I don’t know, it must be encrypted or damaged somehow…”
Me: “OK, send it over and I’ll have a look…”
Turns out the message was EDI…
EDI comes in two main flavors, ASC X12 (a.k.a. ANSI X12) and EDIFACT, and then some offspring variants from those used by different industries, e.g. ODETTE for the car industry.
X12 is mainly used in the US (or companies communicating with the US) while EDIFACT is the European choice of EDI.
X12 and EDIFACT doesn’t really have much in common looking at the formats, they are both really messed up and hardly readable unless you like me have had the (dis?)pleasure to work with EDI for more than 15 years…
We need to understand that EDI was created way back in time, 1979 for X12 and 1987 for EDIFACT, and back then it was all about saving bytes. Storage back then was crazy expensive and data transfers even more so and on top of that very unreliable. The only way to go was to save on bytes…
That’s why a lot of smart people got together to try and figure out how to cram as much information as possible into the fewest possible number of characters.
Looking at EDI we have both addressing as well as message meta-data like version and message type included in each message, both X12 and EDIFACT, although they look very different.
In X12 it is called the ISA segment:
ISA*00* *00* *ZZ*RECEIVERID *12*SENDERID *100325*1113*U*00403*000011436*0*T*>~
While in EDIFACT it is known as the UNB segment:
UNB+UNOA:3+ SENDERID:ZZZ+ RECEIVERID:ZZZ+020102:1000+12345555+++++UNOC'
Those samples looks kind of messy don’t they?
Well, the good news is that the DataPower XB62 appliance actually understands that out-of-the-box meaning you can receive, route and handle EDI messages (both X12 and EDIFACT) without even understanding it yourself!
Now that is very useful of course but hardly what we can call “mastering” unfortunately… To get into the mastering bits and pieces we need to add another IBM product natively supported on DataPower into the pot, namely WebSphere Transformation Extender (WTX).
WTX dates back to the late -80’s actually but was then developed by a company called TSI, who became Mercator, who became Ascential who in turn was bought by IBM… WTX was built back in the days to handle EDI transformation and validation, something that it did way better than any other product on the market, and still does!
The WTX team has on top of that graced us with a “compliance” check system for DataPower. Using the compliance check we can not only validate and transform EDI but we can also auto-generate acknowledges (receipts) back to the sender including error messages and even EDI syntax errors. In X12 it is called a “Functional Acknowledgement”. In short it’s called FA, or because X12 numbers each message type they also can go under the name 997 FA.
In EDIFACT it is simply called CONTRL (and, no, that is not misspelled, message types in EDIFACT only consist of six characters).
E.g. an EDIFACT invoice is called INVOIC while X12 call it an “810” and an order is “ORDERS” or an “850”.
In order to start apprenticing for handling EDI in DataPower we do need some WTX skills to be able to transform, split and route EDI.
I will not go into the details around WTX, simply stating input- and output-formats that are going to be used.
So, in order to receive EDI files in DataPower you need to set it up to receive “Binary” files. In this post I will focus on using AS2 through the B2B Gateway service in the DataPower XB62 but you can use the same techniques in a XI52 as long as you can receive the “binary” file somehow, e.g. through FTP, HTTP POST or similar.
You would start up by setting up two partners in the B2B Partner community in the XB62, one “internal” partner (=your company) and one external. The only difference between an internal and an external partner is pretty much that an internal partner can sign and encrypt messages while an external only can verify signatures and decrypt.
The partners need to be setup using the AS2 identifier as the Business Identifier as that is the first that will be caught by the XB62. If the partners are using the same AS2 identifier as the EDI addressing you are in luck and can use only one B2B gateway. However if they differ (which they normally do) and you want to have a separate “EDI monitoring” (which I assume you want) you need to setup a second B2B gateway.
This can be done either in a separate domain (which I recommend) or in the same domain. This would be particularly useful if you receive EDI (or other messages) from a VAN provider (Value Added Network) or some partner that sends you different message types using the same AS2 identifier.
This means that you will end up with one “AS2 handler” and one “EDI handler” B2B gateway. The great thing about that (if you use different domains) is that you will also get separate B2B Transaction Viewers, one where you can monitor the AS2 stuff and another where you will see the EDI stuff, complete with partner ID’s from the ISA or UNB and the payload (=EDI message) only in the message content.
If one and the same AS2 partner sends you different messages, e.g. EDI and XML, you would also route this in the AS2 handler to end up on different domains and/or services.
So, once you got the partners setup you need to start thinking on the formats you want to use. I am going to assume you want to transform the EDI into XML. It doesn’t really matter what XML format you go for but it’s a good practice to include the same routing meta-data in the XML as you had in your ISA or UNB. One good format to go for is xCBL 4.0 in my opinion (http://www.xcbl.org/).
The incoming messages will now get caught by the AS2 handler that will un-package the AS2 envelope and then pass the payload (EDI message) to the EDI handler in turn. I always use a HTTP destination for this on a separate port, e.g. 127.0.0.1:30001. (Always keep a separate documentation for any “bounce” ports you use in DataPower to not mess things up!)
Assuming the AS2 part was dealt with successfully you will now get an EDI message into another B2B gateway. In this B2B gateway the XB62 will be able to identify the partners from the ISA or UNB automatically and you can setup a partner specific “Processing Policy” that will be executed when that particular partner is “triggered” by a message.
A Processing Policy in DataPower is a GUI where you can add “actions” to the message flow. One such action is a “binary transform” and that is what we are going to use to run a WTX map for our message.
The WTX map is built on any Windows client where you (unfortunately) need a separate license for the development tool called Design Studio. The WTX runtime however is included in the DataPower firmware and does not require any additional licensing!
Building the map in WTX is normally done in a few hours only assuming you have the EDI Industry Pack (sold separately) and are going to output some XML format, e.g. xCBL. The EDI Industry Pack also contains the Compliance Check which we are going to need.
If you want to run the compliance check, i.e. producing a FA or CONTRL you need to add that first to your Processing Policy and then add the transformation into XML at the following action.
If you are not doing compliance then you don’t need any other tools to validate the EDI message, that is automatically done for you by WTX when it reads the data.
As DataPower is a “synchronous” handler you would detect and “stop” any message not valid before the partner can close the connection. This means that the partner will automatically either get a communication error or a negative response (=error message) back.
This is one of the best perks of this solution, not having to deal with invalid EDI messages yourself. Normally a gateway or proxy just tunnels messages through into your internal systems and accepts the transmission. Not until you are going to transform or validate the message will you detect that it is invalid. This puts it on your shoulders to contact the sending partner and inform them about the invalid message and how to deal with it, i.e. resend it from the partner or alter it in your platform. This of course leads to longer processing times and adds the risk of “human error” into the picture…
With the DataPower/WTX solution you can rest assured that any message delivered into your internal systems is valid and up to your standards. WTX will make sure of it!
Talking about internal systems you might start wondering about how it got there… If you haven’t been working with B2B or external messaging you need to understand that a “gateway” resides in the DMZ (the external network zone facing the Internet). The DMZ (Demilitarized Zone) is considered an “unsafe” environment and you should only allow secure applications to reside there, such as appliances, proxies and web-servers. Any internal systems like ERP systems, HR, databases will be in the internal (Intranet) zone with a firewall in between it and the DMZ.
There are various ways of getting the messages from the DMZ to the Intranet but I always favor IBM WebSphere MQ, and especially not with the File Transfer Edition (MQ FTE) available moving message between the zones is done in a breeze… However you are free to use any of the protocols supported in DataPower that suits your needs, be it WebServices (SOAP), sFTP or even FTP… DataPower will handle the protocol bridging for you seamlessly.
This has (hopefully) given you a slight insight in how EDI can be handled in DataPower and a bit of an understanding of the DataPower appliance in general… There are more to EDI than this of course; you might need to split a message as EDI can be sent as a “group” with more than one message in the same transmission, you might want to enrich or do custom routing for certain messages or deliver custom responses back to your partner. All this can also be done using a combination of the WTX features and system variables in DataPower but that would go in the advanced section of this blog article that may or may not be written… J