# Removing similar records from XML

**URL:** <https://community.cdata.com/t/removing-similar-records-from-xml/816>\
**Category:** CData Arc\
**Tags:** question, xml\
**Created:** [July 5, 2024, 4:14pm UTC](https://community.cdata.com/t/removing-similar-records-from-xml/816 "2024-07-05T16:14:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![russell-jerseypost](https://avatars.discourse-cdn.com/v4/letter/r/439d5e/32.png) [@russell-jerseypost](https://community.cdata.com/u/russell-jerseypost)\
**Post date:** [July 5, 2024, 4:14pm UTC](https://community.cdata.com/t/removing-similar-records-from-xml/816/1 "2024-07-05T16:14:58Z")

</div>

Given a file like

```xml
<?xml version="1.0" encoding="utf-8"?><statuses xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <status> <StatusID>1056</StatusID> <ReqDepot>183</ReqDepot> <ManifestDate>2024-07-05</ManifestDate> <DeliveryDate>2024-07-08</DeliveryDate> <ConNo>0000105</ConNo> <VigoID>0</VigoID> <TrackingCode>1831110000105</TrackingCode> <PalletID>1831110000105-02</PalletID> <StatusCode>DSO</StatusCode> <StatusTime>16:37:09</StatusTime> <StatusDate>2024-07-05</StatusDate> <Notes /> <PODName /> <PODTime>16:37:09</PODTime> <PODDate>2024-07-05</PODDate> </status> <status> <StatusID>1066</StatusID> <ReqDepot>183</ReqDepot> <ManifestDate>2024-07-05</ManifestDate> <DeliveryDate>2024-07-08</DeliveryDate> <ConNo>0000105</ConNo> <VigoID>0</VigoID> <TrackingCode>1831110000105</TrackingCode> <PalletID>1831110000105-01</PalletID> <StatusCode>DSO</StatusCode> <StatusTime>16:38:03</StatusTime> <StatusDate>2024-07-05</StatusDate> <Notes /> <PODName /> <PODTime>16:38:03</PODTime> <PODDate>2024-07-05</PODDate> </status> <status> <StatusID>1066</StatusID> <ReqDepot>183</ReqDepot> <ManifestDate>2024-07-05</ManifestDate> <DeliveryDate>2024-07-08</DeliveryDate> <ConNo>0000105</ConNo> <VigoID>0</VigoID> <TrackingCode>1831110000105</TrackingCode> <PalletID>1831110000105-01</PalletID> <StatusCode>DSO</StatusCode> <StatusTime>16:38:03</StatusTime> <StatusDate>2024-07-05</StatusDate> <Notes /> <PODName /> <PODTime>16:38:03</PODTime> <PODDate>2024-07-05</PODDate> </status> <status> <StatusID>1067</StatusID> <ReqDepot>183</ReqDepot> <ManifestDate>2024-07-05</ManifestDate> <DeliveryDate>2024-07-08</DeliveryDate> <ConNo>0000105</ConNo> <VigoID>0</VigoID> <TrackingCode>1831110000105</TrackingCode> <PalletID>1831110000105-03</PalletID> <StatusCode>DSO</StatusCode> <StatusTime>16:38:05</StatusTime> <StatusDate>2024-07-05</StatusDate> <Notes /> <PODName /> <PODTime>16:38:05</PODTime> <PODDate>2024-07-05</PODDate> </status></statuses>

```

I want to reduce this down to just

```xml
<?xml version="1.0" encoding="utf-8"?><statuses xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <status> <StatusID>1056</StatusID> <ReqDepot>183</ReqDepot> <ManifestDate>2024-07-05</ManifestDate> <DeliveryDate>2024-07-08</DeliveryDate> <ConNo>0000105</ConNo> <VigoID>0</VigoID> <TrackingCode>1831110000105</TrackingCode> <PalletID>1831110000105-01</PalletID> <StatusCode>DSO</StatusCode> <StatusTime>16:37:09</StatusTime> <StatusDate>2024-07-05</StatusDate> <Notes /> <PODName /> <PODTime>16:37:09</PODTime> <PODDate>2024-07-05</PODDate> </status> <status> <StatusID>1066</StatusID> <ReqDepot>183</ReqDepot> <ManifestDate>2024-07-05</ManifestDate> <DeliveryDate>2024-07-08</DeliveryDate> <ConNo>0000105</ConNo> <VigoID>0</VigoID> <TrackingCode>1831110000105</TrackingCode> <PalletID>1831110000105-01</PalletID> <StatusCode>DSO</StatusCode> <StatusTime>16:38:03</StatusTime> <StatusDate>2024-07-05</StatusDate> <Notes /> <PODName /> <PODTime>16:38:03</PODTime> <PODDate>2024-07-05</PODDate> </status></statuses>

```

The key to look out for is \<ConNo\> I need the&nbsp; **last** \<status\> node for each \<ConNo\> only.

Any thoughts on how to achieve this?

thanks in advance

---

<div class="post-metadata">

**Author:** ![James\_B](https://sea3.discourse-cdn.com/cdata/user_avatar/community.cdata.com/james_b/32/1688_2.png) [@James\_B](https://community.cdata.com/u/James_B)\
**Post date:** [July 5, 2024, 8:19pm UTC](https://community.cdata.com/t/removing-similar-records-from-xml/816/2 "2024-07-05T20:19:49Z")

</div>

Unfortunately, there really isn’t a simple way to loop over an XML record like this and filter out the duplicate records without involving some scripting of your own, because the XML Map connector does not include any elements like “Foreach Unique”.

In a separate forum entry, I have written about script where you can group elements in a format like this together in groups based on a shared key column:

[Grouping together records in a flat format (CSV) based on a shared key | Community (cdata.com)](https://community.cdata.com/cdata-arc-48/grouping-together-records-in-a-flat-format-csv-based-on-a-shared-key-109?tid=109&fid=48)

And if you do this, you could later attempt to detect the last element in this set (although you provide ConNo as your key element in the example above, and all of the elements in the source share the same value here).

There may be a better approach however - does this data start out as XML or is it in another format? Is the destination for this data a table (which you could potentially use to upsert records to the same column based on the key)?

---

<div class="post-metadata">

**Author:** ![russell-jerseypost](https://avatars.discourse-cdn.com/v4/letter/r/439d5e/32.png) [@russell-jerseypost](https://community.cdata.com/u/russell-jerseypost)\
**Post date:** [July 9, 2024, 8:08am UTC](https://community.cdata.com/t/removing-similar-records-from-xml/816/3 "2024-07-09T08:08:39Z")

</div>

Thanks. I could potentially fire this off to a custom REST API and put the logic in the API endpoint.

I am beginning to use _[n8n.io](http://n8n.io)_ as another useful tool in my EDI armoury. It’s not a replacement for cData Arc, but it can compliment it.