Analyzing (un)structured data to convert it into a structured, normalized format.

learn more… | top users | synonyms (1)

5
votes
1answer
3k views

Any ASN.1 parsers written completely in managed code & safe from buffer overflows

ASN.1 is a data format used on HTTPS certificates, and various other critical pieces of infrastructure. I need to analyze some potentially hostile ASN.1 data, and want to ensure my code is safe from ...
5
votes
1answer
132 views

quantitatively comparing AST shapes

How could one compare the shape of abstract syntax trees of similar source code programs (C, C++, Go, or anything compiled with GCC...)? I guess that plagiarism detection on source code would use ...
5
votes
4answers
1k views

Programming Language Parser (in Java) - What would be a better design alternative for a special case?

Background I'm currently designing my own programming language as a research project. I have most of the grammar done and written down as context-free grammar, and it should be working as is. - Now ...
5
votes
2answers
648 views

Scripting custom drawing in Delphi application with IF/THEN/ELSE statements?

I'm building a Delphi application which displays a blueprint of a building, including doors, windows, wiring, lighting, outlets, switches, etc. I have implemented a very lightweight script of my own ...
5
votes
1answer
269 views

How should I test the HTML output my class creates?

As a learning project, I am trying to create something similar to the WebGrid that comes with ASP.NET MVC. Now this component MyGrid<T> looks like this: public class MyGrid<T> where T : ...
4
votes
4answers
1k views

What is the responsibility or benefit of a Tokenizer?

Suppose I had a grammar like: object { members } members pair pair string : value value number string string " chars " chars char char chars number digit ...
4
votes
3answers
364 views

What is the proper way to distinguish between keywords and identifiers?

I'm aware that most modern languages use reserved words to prevent things like keywords from being used as identifiers. Reserved words aside, let's assume a language that allows keywords to be used ...
4
votes
2answers
915 views

Learning YACC nowadays, does it make sense? [closed]

I have a huge project that is using YACC and I would need to fix a bug in it. I might ask someone else who wrote that to fix it but I'm interested in how compilers work. Does it make sense to learn ...
4
votes
3answers
720 views

Any good reason to open files in text mode?

(Almost-)POSIX-compliant operating systems and Windows are known to distinguish between 'binary mode' and 'text mode' file I/O. While the former mode doesn't transform any data between the actual file ...
4
votes
1answer
573 views

How to line-break an email address?

After some discussion, I have come across a rather complicated situation. Say I intend to display an email address. I have, obviously, limited space available on the screen - be that browser or ...
4
votes
1answer
472 views

Object oriented parser model review

First time I make use of Code Review. At the moment I'm creating an Android application and the company I'm developing it for, has a specific file format that can be used to give text based data ...
4
votes
5answers
927 views

What would happen if you defined your system's CSV delimiter as being a quotation mark?

Title says it all. If the CSV's system delimiter was " (as opposed to a comma or pipe or other common alternatives), how would anything deal with it? The crux of the matter is of course that by ...
4
votes
1answer
123 views

How to approach syntax errors when developing a lexical analyser?

I am writing an interpreted domain-specific language for my application. Each statement is parsed and executed as soon as lexical analyser decides that the end of statement is reached. It's handling ...
3
votes
2answers
3k views

Is there a more modern program than lex or yacc, which does not require JVM?

Is there a "modern" bison and flex equivalent which does not require the JVM or .NET or similar "heavy" runtimes? Thanks for comments, edit: By modern, I mean for example Antlr and another one in ...
3
votes
3answers
770 views

How do I translate user input into a fictitious language?

For experimental reasons, I am trying to convert user input into a fictitious language. All of the translation can be 1:1. I would prefer if I could accomplish this with PHP. Should I use gettext ...
3
votes
6answers
222 views

In which layer I should implement file parsing?

in a simple multi layer architecture, in which layer do I have to implement something like parsing file. For example: I have a file and I have to extract specific information into an object. I think ...
3
votes
1answer
269 views

Showing a grammar is ambiguous

I have the following question taken from a compilers course exam: Show that the following grammar is ambiguous. S = XcY X = a Y = b | Z Z = bW W = d | ϵ I drew the following tree: Am I correct ...
3
votes
1answer
854 views

Browser parsers vs Stand-alone parsers

Firefox/Chromium have well implemented(?) parsers. Then why are parsers like lxml and other such parsers being developed with so much effort put into it? Aren't they re-inventing the wheel? Or are ...
3
votes
1answer
213 views

Recursively parse without resorting to ugly design patterns

I'm currently building a crochet pattern parser in Java, and I've hit upon some trouble. I'll call the language used for input Crochet Pattern Code (CPC). I have a rather large writeup on the ...
3
votes
2answers
159 views

Need overview of concepts and tools to translate a DSL to regular expressions

I'm looking for a little guidance. Until this morning, this was all over my head. After spending today researching Wikipedia, StackOverflow, etc., I'd say I've got my nose above the water. I'm tasked ...
3
votes
1answer
77 views

How to parse different number types with LALR(1)

Consider a LALR(1) parser for a file format that allows integer numbers and floating point numbers. As usual, something like 42 shall be a valid integer and a valid float (with some automagic ...
3
votes
1answer
240 views

Sentence Tree v/s Words List

I was recently tasked with building a Name Entity Recognizer as part of a project. The objective was to parse a given sentence and come up with all the possible combinations of the entities. One ...
3
votes
1answer
2k views

Extracting color profile information from JPEG files

I'm trying to look up info about reading JPEG's color profile info and to my surprise there's very little open specific how-to information on that regard, but rather lots of general explanation on ...
3
votes
3answers
645 views

design for a parser to handle very large files

I have written a program which records protocol messages between an application and a hardware device which matches each application request with each hardware response. This is so that I can later ...
3
votes
2answers
497 views

Process to generate hierarchical structure based on relational data?

I have a csv of employee ids, names, and a reference column with the id of their direct manager, say something like this emp_id, emp_name, mgr_id 1,The Boss,, 2,Manager Joe,1 3,Manager Sally,1 ...
3
votes
1answer
433 views

Chosing a parser for a code beautifier

I'm in the planning stage of making a code beautifier (similar to AStyle or Uncrustify) - originally I was going to just contribute to one of those projects, but reviewing their source led me to the ...
3
votes
2answers
3k views

How to get lookahead symbol when constructing LR(1) NFA for parser?

I am reading an explanation (awesome "Parsing Techniques" by D.Grune and C.J.H.Jacobs; p.292 in the 2nd edition) about how to construct an LR(1) parser, and I am at the stage of building the initial ...
3
votes
1answer
224 views

How to support mixed grammars?

I am currently writing a grammar for a programming language (PowerBuilder) and I am also collecting documentation about parsing and compilers creation. Now I would like to add in my grammar (that ...
3
votes
1answer
162 views

Performance issues when parsing code [closed]

I was wondering about performance issues when parsing a source file that is being edited by the user (for example, when you need to give a syntax highlight). I think that the simplest approach is to ...
3
votes
1answer
208 views

What does it mean that StAX parsers aren't complete or correct?

I've been browsing the net and found theses two slides on site of XOM (XML object model): slide1 slide2 Where they refer to SAX(push) and StAX(pull) parsers. So I'm wondering, what does it mean ...
3
votes
1answer
772 views

How can I test a parser for a bespoke XML schema?

I'm parsing a bespoke XML format into an object graph using .NET 4.0. My parser is using the System.XML namespace internally, I'm then interrogating the relevant properties of XmlNodes to create my ...
2
votes
4answers
120 views

Is the var token necessary to signal variable declaration?

When creating a variable foo, Python lets you just write foo = bar. However many languages, like C# or JavaScript, require additional syntax like var foo = bar or foo := bar to signal the same thing. ...
2
votes
3answers
2k views

c++ write own xml parser vs using tinyxml

I am currently in a task to generate an XML file for an srt text file containing timestamps and corresponding text. To generate an exe file which accepts file name input and outputs the relevant XML ...
2
votes
2answers
159 views

parsing terminology: comments+whitespaces vs actual code

In languages like c/c++ spacing and comments are ignored and only actual code gets into compiler. I'm interested if there is accepted way of naming these two things? comments & spacing ...
2
votes
4answers
228 views

The right place to embed a parser : Server or the Client?

I'm writing a database management software. It has a client-server architecture where the server has the piece of code to store and retrieve. The client has to get the input from the user, connects to ...
2
votes
1answer
521 views

REST efficiency on count and sum methods

For example, if I have a Customer Transactions Table and I create a REST for it. The fields are: date, description, and total_amount. I am trying to figure out which one is much more efficient when ...
2
votes
3answers
2k views

Writing a CSS parser in C#. What do you think is the best strategy?

I'm in the middle of writing a CSS parser in C#. I'm well under way, but I also have those times where I wonder if I'm taking the best approach. The things I've considered are: Feed the CSS grammar ...
2
votes
2answers
112 views

Regular syntax — what does it mean?

I am reading "Less is exponentially more" and there is a list of advantages of Go, first of them, quote "regular syntax (don't need a symbol table to parse)". What does it mean "regular syntax"? ...
2
votes
3answers
5k views

An ideal way to decode JSON documents in C?

Assuming I have an API to consume that uses JSON as a data transmission method, what is an ideal way to decode the JSON returned by each API resource? For example, in Java I'd create a class for each ...
2
votes
1answer
1k views

How to add precedence to LALR parser like in YACC?

Please note, I am asking about writing LALR parser, not writing rules for LALR parser. What I need is... ...to mimic YACC precedence definitions. I don't know how it is implemented, and below I ...
2
votes
1answer
502 views

Idea of an algorithm to detect a website's navigation structure?

Currently I am in the process of developing an importer of any existing, arbitrary (static) HTML website into the upcoming release of our CMS. While the downloading the files is solved successfully, ...
2
votes
2answers
208 views

“Hand written” recursive descent parser with “catch all” rule

I'm trying to write a (scannerless) recursive descent parser with a "catch all" rule for the following "Mustache template" grammar (simplified here): content : (variable_tag | section_tag | ...
2
votes
1answer
212 views

content fingerprint algorithm

I have a database containing just simple URLs. It is as simple as it sounds for now and URL can link to a website or a document(i.e. anything parseable to text). Now I have a simple code which ...
2
votes
1answer
337 views

Practical reference for learning about graph reduction

Are there any practical references (with actual examples) for getting started implementing a small, lazy functional programming language with graph reduction? A reference that included the lexing and ...
2
votes
1answer
283 views

Best way to parse optional grammar rules?

I started to write a hand written parser just for fun. For simple rules it is working, but I have problem with the optional grammar rules. I marked the optional rules with a question mark. Here is a ...
2
votes
2answers
136 views

Incorporating functions into a Shunting-Yard algorithm implementation

tl;dr What would be a simple way of incorporating functions into a Shunting-Yard algorithm implementation? If only expressions like function(arg1, arg2, arg3) were allowed (where function is some ...
2
votes
2answers
125 views

Algorithm to go from infix notation to a tree

I've been trying to figure out an algorithm to go from an infix equation to a syntax tree, like so: (1+3)*4+5 + * 5 + 4 1 3 However, I don't just want it to handle operators, I ...
2
votes
1answer
137 views

Writing a parser on top of an XML-based AST: am i doing it right?

I have a sort of AST defined in XML that i'm trying to parse and evaluate. The XML tree contains the tokens and all the information i need. However, i'm finding it difficult to do it "properly". ...
2
votes
2answers
466 views

Traversing an AST using Visitors

I'm writing a compiler for a C-like language, and I'm looking for an elegant way to traverse my abstract syntax tree. I'm trying to implement the Visitor pattern, although I'm not convinced that I'm ...
2
votes
1answer
589 views

How to choose a proper parser generator for PHP

Some programmers avoid regexes in some situations (see this popular @nickf comment), perhaps using a parsing framework such as Lex/Yacc. Others prefer to stay within PHP, perhaps using regular ...