Reputation
4,007
Top tag
Next privilege 5,000 Rep.
Approve tag wiki edits
Badges
1 18 30
Impact
~73k people reached

Dec
4
comment Storing n-gram data
In my original post, frequency was an integer (representing the count of instances) but in my latest update to the post, I changed it to FLOAT. This could be a logarithm of the integer frequency or a probability of occurrence -- whichever you want. You just need to choose one and be consistent about it.
Dec
4
comment Storing n-gram data
In terms of updating the database, obviously I haven't covered all the details and there is lots of room for improvement. For example, rather than storing nvarchars in the ngram tables, you'd probably want to tokenize into a words table (word_id INT, word NVARCHAR) and then refer to word_ids in the ngram tables. To update the tables on retraining, that's right -- you'd just update the frequency field.
Dec
4
comment Storing n-gram data
The bigramWeight and trigramWeight (etc) are how to weight the different n-grams in the overall calculation. It's a simplistic way of saying that longer n-grams have higher entropy and you may want them to "count" more than shorter n-grams.
Dec
1
comment Storing n-gram data
Good point @TomSawyer. I added sample data to the answer and gave a sample query that returns the most probable next word.
Dec
1
revised Storing n-gram data
Added sample data and a query to get most probable next word
Sep
18
awarded  Notable Question
Jul
19
awarded  Nice Question
Apr
8
awarded  Necromancer
Mar
15
awarded  Yearling
Jan
2
awarded  Great Answer
Aug
19
awarded  Good Question
Jul
2
awarded  Curious
Apr
3
awarded  Popular Question
Mar
15
awarded  Yearling
May
14
comment Duplication of view access control logic in database queries and application component
Good thinking. I like your use of decorators to intercept and check access. I agree that the query part is the most difficult. Part of the problem in the question I posted is that we want to use efficient DB level paging, which meant we didn't want to defer the access control check to the application layer.
Mar
25
awarded  Popular Question
Mar
15
awarded  Yearling
Feb
4
awarded  Guru
Jul
9
comment Automated tests for differencing algorithm
Good insight that there are two constraints here. I need complete inclusion of new content (which is a simple yes/no determination) and minimal extra noise (for which I need the threshold). Thanks.
Jul
9
comment Automated tests for differencing algorithm
If I were verifying for strict accuracy, yes. But due to the heuristic nature of the algorithm and volatility of the data, we don't want that. We want to set an acceptable threshold that will verify complete inclusion of new content and minimal extra noise.