Timo Riikonen on 22 Jul 2024 07:31:29
Question: What does ETL always need in production?
Answer: Messaging on failures
Q: What is the best channel today?
A: Teams (and Email because you can make email rules easier)
Q: Can you do this in Fabric as of 2024/07?
A: Yes, messaging to Teams is in preview state.
What doesn't work:
- Data pipelines always use AND when combining several inputs. This means that you must use OR box to combine failures together before you report the failure. As Data Pipeline doesn't have an OR box, you must create failure reporting for every action separately.
- You can't copy-paste Teams-messaging or their content, so you have to make all of them manually one by one.
- I am not sure if can even copy paste functions, the UI is difficult.
- You can't tell what environment you are running the Data Pipeline: Dev, Test or Prod
Now this idea suggesting is on the last one:
- You can now drag Workspace ID to the Teams Message. This gets renamed as DataFactory for some reason, but don't get confused with that.
- Problem is that this ID is a cryptic key such as 8466c1bc-e477-415e-8b77-a714cd012345, so you need to add separately a decrypt table so people will know what environment was this run on.
So Data Pipeline should have "Workspace" or "Workspace name" dynamic content to choose from.
- Comments (1)
RE: Workspace name as dynamic content
I don't understand why this is not implemented initially.Duplicate ideas:https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=a6df1149-1495-ee11-a81c-000d3a00bef9https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=b3ccfa82-eef7-4d52-ba47-79aed6d96466https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=1eab5cdc-776a-430f-98e7-b64300f50576https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=2b777168-fc47-ef11-b4ad-000d3a0f9a18