Skip to main content

Synapse

New

Pipeline Metadata activity - add additional items

Vote (2) Share
Scott Powell's profile image

Scott Powell on 17 Mar 2024 02:39:58

The pipeline metadata activity currently only allows finding metadata on Lakehouse Tables and Files, and Warehouse Tables. It would be great to add the ability to retrieve names and GUIDs for all Fabric artifacts it can source.


For an example of how this could help - I have 24+ Gen2 dataflows that I use to load data in parallel. Currently I have to make a pipeline by adding 24 individual dataflow activities and selecting which dataflow to run for each activity.


If the Get Metadata activity would allow me to return dataflow IDs, I could automate this with a single get metadata activity, followed by a ForEach acvitity with embedded dataflow.


Get Metadata should be able to find:

Workspace Names and IDs

Dataflow Names and IDs

Nootbook Names and IDs

Dataset Names and IDs (for when the eventual "trigger dataset refresh" becomes available)

etc.


Basically, anything that is a valid Pipeline source or target should be discoverable through Get Metadata.


Thanks,

Scott