Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm in the early stages of writing an Emacs major mode for the Stack Exchange network; if you use Emacs regularly, this will benefit you in the end.

In order to minimize the number of calls made to Stack Exchange's API (capped at 10000 per IP per day) and to just be a generally responsible citizen, I want to cache the information I receive from the network and store it in memory, waiting to be accessed again. I'm really stuck as to what data structure to store this information in.

Obviously, it is going to be a list. However, as with any data structure, the choice must be determined by what data is being stored and what how it will be accessed. What, I would like to be able to store all of this information in a single symbol such as stack-api/cache. So, without further ado, stack-api/cache is a list of conses keyed by last update:

`(<csite> <csite> <csite>)

where <csite> would be

(1362501715 . <site>)

At this point, all we've done is define a simple association list. Of course, we must go deeper.

Each <site> is a list of the API parameter (unique) followed by a list questions:

`("codereview" <cquestion> <cquestion> <cquestion>)

Each <cquestion> is, you guessed it, a cons of questions with their last update time:

`(1362501715 <question>) (1362501720 . <question>)

<question> is a cons of a question structure and a list of answers (again, consed with their last update time):

`(<question-structure> <canswer> <canswer> <canswer>

and `

`(1362501715 . <answer-structure>)

This data structure is likely most accurately described as a tree, but I don't know if there's a better way to do this considering the language, Emacs Lisp (which isn't all that different from the Lisp you know and love at all). The explicit conses are likely unnecessary, but it helps my brain wrap around it better. I'm pretty sure a <csite>, for example, would just turn into

(<epoch-time> <api-param> <cquestion> <cquestion> ...)

Concerns:

  • Does storing data in a potentially huge structure like this have any performance trade-offs for the system? I would like to avoid storing extraneous data, but I've done what I could and I don't think the dataset is that large in the first place (for normal use) since it's all just human-readable text in reasonable proportion. (I'm planning on culling old data using the times at the head of the list; each inherits its last-update time from its children and so-on down the tree. To what extent this cull should take place: I'm not sure.)
  • Does storing data like this have any performance trade-offs for that which must use it? That is, will set and retrieve operations suffer from the size of the list?

Do you have any other suggestions as to what a better structure might look like?

share|improve this question

migrated from codereview.stackexchange.com Mar 5 '13 at 19:54

This question came from our site for peer programmer code reviews.

    
I'm +1ing this because I really want this mode –  jozefg Jul 29 '13 at 13:24
    
@jozefg I really want it too—this internship has sucked up most of my time, but once school starts up some more progress should be made. –  Sean Allred Jul 29 '13 at 13:25
    
I've been pretty happy just installing a browser plugin that allows me to use Emacs to fill in text box contents. Are you going to have Emacs understand Wiki markup and display the formatted text? –  kevin cline Jul 29 '13 at 14:41
    
@kevincline No, the idea is that it will just do utilitarian tasks: local question archival; advanced code editing (bopping out to the right major mode, similar to org); inserting <!-- language: blah> where necessary (depending on the mode that code editing was done in); stuff like that. See the README on GitHub for more info, and feel most welcome to suggest features. The more I know about this before hand, the better it can be designed. edit not to mention emacs keybindings ;) –  Sean Allred Jul 29 '13 at 14:47
add comment

1 Answer 1

Emacs lisp is not optimized for data processing; you might find it advantageous to use a Common Lisp for the engine and Emacs for presentation only.

Even if you decide to stick with Emacs Lisp, I suggest that you use structured data (eieio) instead of lists, and hash tables instead of alists.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.