SmartsIntegrator Support and Discussion Forums

Complex flat-file scenario

Share/Bookmark: Add this Thread to Digg Digg   Add this Thread to StumbleUpon Stumble Upon   Add this Thread to Del.icio.us Del.icio.us   Add this Thread to Reddit Reddit   Add this Thread to Facebook Facebook
RE: Complex flat-file scenario
by benj on Wed, Oct 29 2008 2:59 PM
Many thanks for your further feedback. I apologise for the issues in build 1.3 16 - we will investigate how this happened and ensure that it doesn’t recur.

I’ve tested case (1), and we can confirm that this is a bug. This will be addressed in build 1.3.17, due out shortly. Note that when you get the new build, you will need to set the DelimitScope for each of the prefix records to NotUniqueAcrossChildElements and this will all work correctly.

Regarding (2), this is also a bug to be fixed in 1.3.17.

I will get this build to you as soon as possible – it is in testing at the moment. I appreciate your patience.

Many thanks,

Ben.
Ben Jackson | SnartsIntegrator Support
Ben Jackson
SmartsIntegrator Technical Support
Quote This Reply
benj
benj
Joined: Thu, Feb 7 2008
Posts: 96
RE: Complex flat-file scenario
by portedquery on Wed, Oct 29 2008 2:58 PM
Ben,

I have a couple of issues to clear up before I finalize our decision to purchase the product.

1.       The parsing examples you sent me for the positional and delimited file were good. However, I found that the recommended example, example2, has issues with using the record type as the delimiter. If that type is found in any of the delimited data within the record, the parsing engine uses that to further delimit the data. You can imagine what that might look like. If there was a way to limit the parser from testing the delimiter beyond the initial delimiter within the record, I think that might correct the issue. I’m just not certain of the correct method to use.
2.       The map validation fails, and causes the application to abend whenever I attempt to test using generated source document or a file. I also attempted to rule out any potential schema issues that might have caused this condition by using the same schema for both source as well as destination. I’m not sure how to proceed on this issue. I am currently using version 16 as you recommended last week.
Quote This Reply
portedquery
Joined: Wed, Jul 2 2008
Posts: 72
RE: Complex flat-file scenario
by benj on Mon, Oct 20 2008 4:41 PM
Hi J

My sincere apologies for the delay in getting back to you, I wanted to test a few scenarios.

Essentially your approach was perfectly valid – i.e. capturing the raw data and then processing it in a map or workflow.

However, you may prefer to use some of the more advanced flat file parsing functionality.

I’ve attached an export of two schemas. The first extracts the date/time and record type into fields, and then captures an array of / separated data fields. This would be a good starting point for mapping logic, and would enable filtering on the record type field.

The second schema is more sophisticated, and allows you to specify a separate Xml structure for every record type. Note that this only supports records 01 and 26 currently and would need extending.

You'll need to download version 1.3.16 to use these schemas, which is available from http://www.smartsintegrator.com/download.aspx.

If you play with both of these using your sample file I’m sure you’ll get the idea. Let me know if I can help you with this further.

Please let me know if I can assist at all when you get onto creating any map or workflow logic, or if you have any other complicated scenarios that you require advice upon.

Thanks again, I look forward to speaking soon,

Ben.
Ben Jackson | SmartsIntegrator Support
Ben Jackson
SmartsIntegrator Technical Support
Quote This Reply
benj
benj
Joined: Thu, Feb 7 2008
Posts: 96
Complex flat-file scenario
by portedquery on Mon, Oct 20 2008 4:29 PM
Ben,

I am responsible for evaluating the "smartsintegrator" application for purchase and, after reviewing the help file as well as the application, find the lack of doc a bit daunting for me in determining whether I should recommend the product for purchase. If you have a few minutes to review the issue surrounding my, I would really appreciate your time.

I have a file to process that contains several types of records. Each record is delimited by a crlf, and a significant number of records are further field delimited by a forward slash. The record id that I need to locate is within the first x bytes, at positions 16 and 17, thus from the 2 example records, I have both a 01 and a 26 record. There are approximately 20 different record types in this file.

raw record examples:
20080919 06:30 01 D Rock Island IL
20080901 00:00 26 X /DRIVER/'Jerald Rose'/'01'/472/'3.4.4.1 29 Jun 2008'/


I created a flat file structure:







{CRLF}
Infix_ToleratePostfix
False







False
Infix
/












It produced an output file, a snippit of which follows:



20080901 00:00 26 X
DRIVER
'Jerald Rose'
'01'
472
'3.4.4.1 29 Jun 2008'




20080901 00:00 24 W
RULE


USA2005
11.0
0.0
14.0
70.0
8.0
0
0
0
1



20080906 15:15 16 X Pryor OK


20080906 16:15 01 N Muscogee OK


20080906 16:15 15 X Muscogee OK

My intent was to identify the various records, pass them to various queues, then process the individual record types utilizing a structure that would further identify the individual elements. I could then populate a structure for database insertion that would comprise elements from two or more record types. The structure I created for the entire file is obviously not adequate for easy identification of the various elements, but I had hoped I could identify the record id through the mapping process. Is this the correct strategy with your app? If so, are there any examples that I could refer to that would help me complete this evaluation? Is there a different approach you would recommend?

Your kind response is greatly appreciated.

J
Quote This Reply
portedquery
Joined: Wed, Jul 2 2008
Posts: 72
Page 1 of 1