How to Remove Duplicate Records

Hi all,
Did anyone removed duplicate records from a record list. Lets say I have 10 records. Some of them are duplicate how to remove those duplicate records. I can do this in Hash Table but it will be a time consuming one, dont work well for huge number of records immagine a file with 50,000 records.



I have the same problem.
I send you some answers.

good luck

Posted By Omar on Tuesday, June 04, 2002 - 06:52 pm


I’m working in a flow service.
I have a record list, and i want to remove some records when the value of a field it’s equal to “003”

The code is:

LOOP over /reg_SAP/BLAORD03/IDOC[0]/E1EDP01

Are there some functions where i can remove or delete the record ???

Thanks very much …

Posted By PU on Tuesday, June 04, 2002 - 07:02 pm

I think inside that loop when your condition is matched, you can use regular DROP to drop that record and it will drop it from the list.

Posted By PU on Tuesday, June 04, 2002 - 07:17 pm

I mean, DROP the contents of the record. This will leave an empty space in your record list. Then later while using the resultant array check for empty records. Otherwise you can copy the required ones into another record list. These are the two ways you can get rid of elements in a list.

Posted By Dan Green on Tuesday, June 04, 2002 - 07:21 pm

PU is right, but that will also drop the record from your pipeline editor. Be sure to do a CTRL-C on the record and CTRL-V in the Pipeline In panel for the next step of your flow. Map the pasted record to a Service In input or something else to keep it visible to you and other developers.

Posted By Omar on Tuesday, June 04, 2002 - 07:26 pm

Hi PU,

I’m sorry but i am not sure how can i do it.

I use the modifiers DROP, but this doesn’t work.

Please, can you tell me the steps ??

Thanks …

Posted By Dan Green on Tuesday, June 04, 2002 - 07:43 pm

Another thought: (because I missed the point entirely the last time)

Eventually, you will need to loop over the Record List and perform some action. Can you move the remaining steps of your Flow inside of the loop and complete the record processing only if “/reg_SAP/BLAORD03/IDOC[0]/E1EDP01/ACTION” does not evaluate to “003” ?