Hua Yang on 07 Nov 2023 15:18:18
I tried to using spark structured streaming json files with complex schyema in Fabric. the result is wrong and it is different from the result of the batch process, which is correct. In batch , we can use InferSchema=true, but for file streaming , we need supply a schema. I Hope Fabric can have a feature similar to cloudfile in Databrick, so streaming file via cloudfile, we do not have to build a schema and manage file state.