Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to publish a small web project that is supposed to contain some of my research results to present it to the scientific community. All my analysis I ran so far have been written in python already, thus I considered to use django web framework to provide the results. This is, because I plan to show some visualization on up-to-date data, so I can run my analysis on new data by a cron directly in django.

there are actually two questions on my mind so far, how I am going to start developing:

  1. How can I easily store numpy arrays in sqlite to keep the structure of the array and which type of django database field suits best?
  2. What is the best way to json.dump the numpy 2d-arrays to be able handle them in d3.js?

As you can see it is quite about putting numpy + django + d3.js together. My first hacking was to store the numpy results as strings, but I think this is not quite the best, since I need to parse them again after pulling them from database. Concerning the second question I already found simplejson.dumps but unfortunately it does not really work on numpy arrays and thus, sometimes I needed to hack something like this

return  HttpResponse('{"array1" : '+simplejson.dumps(list(array1))+'}')

which I actually don't really like.

So to put it together again:

What are your techniques for storing numpy structures in django database fields and how do you serialize them for http?

share|improve this question

closed as too broad by MichaelT, Snowman, durron597, GlenH7, Ixrec May 6 '15 at 19:26

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

up vote 2 down vote accepted

the best solution depends on a number of further questions, e.g.:

  1. do you need to query the DB for array-content?
    if yes, you will probably need to store the arrays as strings and query for strings (regex). if no, consider serializing with cPickle. cPickle is fast and less verbose than simple strings.
    anyway, you'll have to put your array in a text-field.

  2. do you need to read or modify the arrays in python after they are written to the DB?
    if yes, cPickle is the way to go. you read it back from the db, recreate the numpy array, read or modify and then encode the array as json.
    if no, (and your arrays are not prohibitively big) store the array directly as json and just write the db-record verbatim to the http-respose.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.