I have a computer that is recording 100 different data points into an OPC server.
I've written a simple OPC client that can read all of this data.
I have a front-end website on a different network that I would like to consume this data. I could easily set the OPC client to send the data to a SQL server and the website could read from it, but that would be a lot of writes.
If I wanted the data to be updated every 10 seconds I'd be writing to the database every 10 seconds. (I could probably just serialize the 100 points to get 1 write / 10 seconds but that would also limit my ability to search the data later). This solution wouldn't scale very well. If I had 100 of these computers the situation would quickly grow out of hand.
Obviously I am well out of my league here and I have no experience with working with a large amount of data like this.
What are my options and what should I research?