Timo Riikonen on 22 Jul 2024 07:31:29
Question: What does ETL always need in production?
Answer: Messaging on failures
Q: What is the best channel today?
A: Teams (and Email because you can make email rules easier)
Q: Can you do this in Fabric as of 2024/07?
A: Yes, messaging to Teams is in preview state.
What doesn't work:
- Data pipelines always use AND when combining several inputs. This means that you must use OR box to combine failures together before you report the failure. As Data Pipeline doesn't have an OR box, you must create failure reporting for every action separately.
- You can't copy-paste Teams-messaging or their content, so you have to make all of them manually one by one.
- I am not sure if can even copy paste functions, the UI is difficult.
- You can't tell what environment you are running the Data Pipeline: Dev, Test or Prod
Now this idea suggesting is on the last one:
- You can now drag Workspace ID to the Teams Message. This gets renamed as DataFactory for some reason, but don't get confused with that.
- Problem is that this ID is a cryptic key such as 8466c1bc-e477-415e-8b77-a714cd012345, so you need to add separately a decrypt table so people will know what environment was this run on.
So Data Pipeline should have "Workspace" or "Workspace name" dynamic content to choose from.
- Comments (2)
RE: Workspace name as dynamic content
I'd suggest looking into Activators to stream pipeline failures to alerts. You can get pretty granular with rules, and it supports both emails and teams messages as actions. I think these are relatively new, so might not have been available at the time this was submitted.Here's the MS link for the documentation:https://learn.microsoft.com/en-us/fabric/real-time-intelligence/data-activator/activator-introductionThe activators will give you direct access to things like the workspace name, with nothing more than a click. But, just so you are aware, it is also possible to use sempy.fabric to programmatically identify workspace names and objects. You shouldn't store a static list which would need to be maintained.https://learn.microsoft.com/en-us/python/api/semantic-link-sempy/sempy.fabric?view=semantic-link-python
RE: Workspace name as dynamic content
I don't understand why this is not implemented initially.Duplicate ideas:https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=a6df1149-1495-ee11-a81c-000d3a00bef9https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=b3ccfa82-eef7-4d52-ba47-79aed6d96466https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=1eab5cdc-776a-430f-98e7-b64300f50576https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=2b777168-fc47-ef11-b4ad-000d3a0f9a18