Scott Powell on 17 Mar 2024 02:39:58
The pipeline metadata activity currently only allows finding metadata on Lakehouse Tables and Files, and Warehouse Tables. It would be great to add the ability to retrieve names and GUIDs for all Fabric artifacts it can source.
For an example of how this could help - I have 24+ Gen2 dataflows that I use to load data in parallel. Currently I have to make a pipeline by adding 24 individual dataflow activities and selecting which dataflow to run for each activity.
If the Get Metadata activity would allow me to return dataflow IDs, I could automate this with a single get metadata activity, followed by a ForEach acvitity with embedded dataflow.
Get Metadata should be able to find:
Workspace Names and IDs
Dataflow Names and IDs
Nootbook Names and IDs
Dataset Names and IDs (for when the eventual "trigger dataset refresh" becomes available)
etc.
Basically, anything that is a valid Pipeline source or target should be discoverable through Get Metadata.
Thanks,
Scott