- Is this true that Azure Stream Analytics can only accept JSON files?
- Can you possibly ingest/send pipe delimited or other file formats to Event Hub and consume them from Stream Analytics?
0
votes
What are you trying to do? You could support any format with a custom gaateway (that's how MQTT support was added). ASA would receive the parsed data so it doesn't really care about formats
– Panagiotis Kanavos
We need to ingest 1 file per 15 seconds. File format is custom it is neither JSON, nor CSV or Avro. It will require custom parsing. Because of Event Hub scalability we thought Event Hub will be useful for this task. What would you recommend?
– greypanda
Scalability means using minimal parsing. Otherwise EventHub will end up parsing instead of ingesting. One option is simply to change the format on the client side (how big are the files and how are they created?). Another option is to create a site to act as a gateway and receive the file, convert it then pass it to Event Hub. This has the advantage that you could batch multiple such files up to the 256KB limit.
– Panagiotis Kanavos
Changing the file format on the client side is not an option for us.
– greypanda
Azure Functions can be triggered by Blob events, so they can run automatically when you upload a file. You can parse the file in there and if you really need event processing, post individual events to Event Hub and ASA. (Probably need to convert all that to a blog post, sigh)
– Panagiotis Kanavos
2 Answers
1
votes
