Solved

Source CSV file attributes as columns

  • 29 January 2024
  • 3 replies
  • 88 views

Userlevel 3
Badge +1

I have 3 different CSV files to load each week, and they always have the same filename.

I set up a “Sync All” job to read these files and load them into my destination.

However, I would like to pass the files’ creation dates or modify dates, as columns into the destination target table.  

Is this possible to do?

 

icon

Best answer by John G 2 February 2024, 18:22

View original

3 replies

Userlevel 2
Badge

Hey @DougN Nouria , there is not currently any way to do this with the CSV source. This might be tricky as we support a number of local and cloud sources to pull CSV files from. Where are you reading these files from? If it’s a local or network machine, is it Windows, Linux, or Mac?

In terms of the data, are you expecting new columns to be returned with each row having the same values (at least within a single job run)?

 

This might be something we can open as a feature request, but we’d want to understand your use case appropriately first so our developers can investigate if this scenario will be possible to support.

Userlevel 3
Badge +1

In my case, I’m using Windows CData Sync to read files off of a windows network folder.  Although I’d imagine that files in S3/AZ blob storage also have metadata with their size and upload date.

If the file job loaded in multiple source files in a single pass (whether going to a common target or to separate destinations per file format), I’d expect the rows from each distinct file to have the same entry for filename, date, etc.

I believe there is some file loading functionality which brings in the filename and the row number within the file.  So it would just be reading more metadata if possible about that source file.

Userlevel 2
Badge

I have opened this as a request with our development team. I will keep you updated when there is any progress on this ask.

Reply