HACKER Q&A
📣 iotdev

How to build an IoT analytics database?


Hello hackernews! My company is launching a new IoT-based product next autumn and we are looking for platform or database to store the data. Product is industrial and it will send telemetry back to cloud regular intervals through MQTT. We are planning to use some open source visualization tool for the data such as Grafana, but storing the data is still problem.

Ideally we could visualize even large intervals (5 years) of data fast but in the same time we would like to store all data sent to us for 5-10 years timespan. Sometimes it's also good to see high frequency sampled data (~10Hz) for some certain duration when ie. maintenance crew is investigating equipment failures.

Some people have suggested using products such as Elasticsearch but it seems it may get very expensive if we want to store data long times.

Prometheus was also mentioned elsewhere as an option but it seems that it doensn't support high frequency sampled telemetry very well.

I can see cost benefits for archiving the raw data into S3 or other object storage, but then query engine should support fetching from different sources. Is there some open source options?


  👤 ilkhan4 Accepted Answer ✓
I was working on this at my previous employer. We started with Elasticsearch and were planning to move to Timescale (https://www.timescale.com). Another project used Cassandra which had some issues but they were happy with overall.

You might also want to look into Thingsboard (https://thingsboard.io) which supports both Timescale/Postgres and Cassandra.