113 reputation
4
bio website
location
age
visits member for 2 years, 4 months
seen Sep 9 at 5:56

Aug
11
comment Data generation system modular design
@Matt The technology is basically open for discussion. Although, since the question was asked, we settled on Python. Currently, we intend to implement each module as a Python module which exposes a 'run' method. All modules import a shared configuration module, which is initialized by a controller script. Other than initializing the config module, the controller also runs, according to the mentioned order, all other modules. Things like managing DB connection, loggers, etc. are also done by the controller. Any feedback and further advice will be much appreciated.
Aug
7
comment Data generation system modular design
please see my second edit to the question.
Aug
7
revised Data generation system modular design
added 310 characters in body
Aug
7
comment Data generation system modular design
Yes, modules will generally consume different data (some may be run parallely on the same input data). Yes, the structure of data will change between modules, as modules in one step, consume data generated by a module in the previous step.
Aug
7
revised Data generation system modular design
deleted 2 characters in body
Aug
7
comment Data generation system modular design
Thanks for the answer. I have a few concerns. Each module needs the complete data from the previous step. So they cannot be piped. I would need to wait for a module to complete its job before running the next step. Additionally, can you please give some ideas of how should modules get the data (if not from a DB)? What are some reasonable ways to transfer data (a lot of it) between modules? I can only think of a DB for this. Also, what are some ways of building a module to receive data instead of pulling it (assuming data can't be piped)?
Aug
7
awarded  Editor
Aug
7
revised Data generation system modular design
added 286 characters in body
Aug
7
asked Data generation system modular design
Jul
30
awarded  Scholar
Jul
30
accepted How should data transfer between a client and a web API for normalized data be designed?
Jul
29
comment How should data transfer between a client and a web API for normalized data be designed?
About the number HTTP requests in options 1, let's say the API can provide datums as a batch. You give it a bunch od record IDs all at once. In this case, the number of HTTP requests can be reduced if the client asks for records as a batch in one call.
Jul
29
comment How should data transfer between a client and a web API for normalized data be designed?
Thanks for the detailed answer! You have good inputs. About coupling between API and data - can you really make a generic API? Should you? I mean, as I see it, the API should follow the structure of your data; otherwise you are unnecessarily designing two different data schemas, one for the DB and one for the API output.
Jul
29
awarded  Student
Jul
29
asked How should data transfer between a client and a web API for normalized data be designed?
Nov
16
awarded  Supporter