Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Are there types of killer applications, classes of algorithmic problems, etc., where it is better, in the long run, to create my own language?

PS: Just to be sure, I mean a new programming language and a compiler, not a new compiler for an existing language.

EDIT: Thank you for the answers. Can you provide some examples, where it is absolutly unnecessary to create a DSL or cases in which a DSL might be a good idea?

share|improve this question
7  
I believe that one should create a DSL for every problem. –  SK-logic Mar 28 '11 at 1:09
3  
Isn't this what LISP is great for? –  Darknight Mar 28 '11 at 9:34
1  
@Darknight, not necessarily Lisp - any language with decent metaprogramming cababilities is ok. –  SK-logic Mar 28 '11 at 10:07
1  
When you have a desire to learn about compiler internals. –  dan_waterworth Mar 28 '11 at 10:21
1  
When you think it would be fun or educational. Designing a new language that needs its own compiler never serves any useful purpose, given the amount of effort involved. (There are, of course, people who are sufficiently intelligent, educated, and experience to know when to ignore my advice.) –  David Thornley Mar 28 '11 at 14:28
show 2 more comments

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

migrated from stackoverflow.com Mar 27 '11 at 23:50

This question came from our site for professional and enthusiast programmers.

28 Answers

up vote 35 down vote accepted

It certainly is relevant for a person to write their own language for educational purposes. To learn about programming language design and about compiler design. But real-world uses are few and far between.

In writing your own language you are:

  • Adding a tremendous amount of complexity to your problem
  • Adding a significant amount of work in writing and maintaining the new language and compiler

So, if you plan to write your own language for your project then the features that it provides that other languages don't have need to offset the above costs.

Take games development for example. They often need mini-languages within their games or scripting languages. They use these languages to script out a huge amount of the in-game events that happen. However, even in this case, they almost always choose existing scripting languages and tailor them to their needs.

share|improve this answer
11  
I do have to mention that in "The Pragmatic Programmer" that writing smaller, domain-specific languages to aid in a task is incredibly helpful and encouraged. I wouldn't recommend writing a full-fledged general purpose language, but a metalanguage that generates code can be helpful at times. –  Jordan Parmer Feb 9 '09 at 18:41
4  
It is a lie. Writing a language does not add a complexity - normally it will reduce the complexity significantly. Implementing a compiler and maintaining it is a tiny piece of work anyway. –  SK-logic Mar 28 '11 at 1:07
2  
@SK-logic, "Implementing a compiler and maintaining it is a tiny piece of work anyway". Have you tried? For what processor? –  user1249 Mar 28 '11 at 2:57
1  
@Thorbjørn Ravn Andersen, I am doing it for living. Nowdays you don't have to target any given CPU directly - since there are great VMs available, like LLVM, .NET, even JVM. And if you're not going to do too much of the expensive optimisations, even targeting a "real" CPU is not a big deal - see the OCaml compiler for an example of this primitivistic approach. –  SK-logic Mar 28 '11 at 9:49
6  
@Thorbjørn Ravn Andersen, by definition compiler is translating from one language to another. The level of that target language does not matter anything. And no one sane will implement a full optimising compiler back-end for a DSL - it is better to reuse the existing one. Actually, most modern DSLs are compiled into C. As for assembler and linker - they've been always considered separate from compilation, since the very early days of the system programming. –  SK-logic Mar 28 '11 at 10:11
show 2 more comments

Are there types of killer applications, classes of algorithmic problems, etc., where it is better, in the long run, to create my own language?

It depends.

Let's take our brain. It seems to be such a complex mess that we encounter borders with ANY programming language (at least now). So maybe in order to actually virtualize our brain we need other approaches and therefor other semantics and syntaxes.

Generally speaking, there are yet such complex topics which might lead to other strategies which would also include a 'better' language for a given scenario.

share|improve this answer
add comment

Your 'edit' seems to be a substantially different question ("when should I build a DSL?" rather than the original question which people understood as "when should I build a new, general purpose programming language"). It seems people have thoroughly answered the 'original' question but there are few answers giving specific criteria for when to use a DSL. So I propose a checklist :

  1. Your userbase is larger than a few people, generally non-technical and / or with restricted system access (hence it's unreasonable to expect them to learn / use an existing general purpose language). If it is within your dev team or software organisation you could instead say "just write a script".
  2. Your users need to use it sufficiently often, with sufficiently varied and changing behaviours needed (i.e. you can't just provide a fixed library of functions maintained by you)
  3. The behaviour that users can specify is too complicated to specify as data (e.g. you can't achieve it using a database table, or a user-input matrix, or a list of tasks, or a key-value collection...think carefully because you can achieve a lot of complexity with these). If you can achieve the behaviour using data input or configuration instead of DSL then you probably should because it will be much less work. Some kind of conditionality, or composability / chaining together, or modelling a few different abstractions may be signs that the behaviour you need is too complex for plain data / configuration
  4. But the behaviour is still restricted enough that you can specify it in a concise DSL. A big danger is 'platform bloat' e.g. if users start to request "can you just add...?". If they need to connect to the internet, or read and write from the file system, or open and close processes - then this is no longer a DSL. (I have seen this happen for real...users allowed to embed small python calls, gradually growing to python scripts, and eventually destroying any limits / modularity / performance)

If all of these are true then a DSL may be appropriate.

share|improve this answer
add comment

Let me just quote Paul Vick, ex chief developer of the VB compiler and now working on Project Oslo and the M language:

It’s mind-bendingly, stupendously difficult to build a new language, even one that’s largely based on an existing one. Yet many programmers think, “hey, I use languages, how hard can this be?” and go at it. … probably over 98% of them fail to ever gain any traction at all, but god bless the optimists because without them we’d never get the 2% of languages that succeed. I’m personally willing to sacrifice the millions of dollars and hours wasted on languages that never make it just so that we can get languages like C# and Java and Ruby and Python and so on.

So the fact that coming up with a new language is a bad idea shouldn’t dissuade people from developing new DSLs, it should just give them pause and hopefully a little humility. The key, I think, is to start small and to stay small.

DSLs: Definitely a bad idea!

share|improve this answer
7  
VB != VBA. By the way, is it even legal to criticize VBA on this site? After all, Joel helped develop it, right? –  Konrad Rudolph Feb 9 '09 at 14:17
1  
although pragmatic programmer was such a good book, recommendation of DSLs in that book was plain stupid. In the same way that they recommended learning a new language every year, IMHO that's pretty stupid as well. –  dr. evil Apr 21 '09 at 8:56
2  
I just edited your answer to point to Paul Vick's article again rather than the Google cache. In 2011 he "reset his blog" and deleted all the VB content, but in 2012 he put it back although with different URLs. Sounds like he was personally having a hard time when he deleted that stuff. –  MarkJ Oct 4 '13 at 11:00
2  
@MarkJ Thanks a lot. And, wow, that article does not make for pleasant reading. Hope he’s doing better now. –  Konrad Rudolph Oct 4 '13 at 11:04
2  
Thanks for the kind comments, I'm actually now working on JavaScript and, yes, things are quite a bit better. :-) Not sure why the original link didn't work, I tried to get all the old link styles working, I'll take a look at it. –  panopticoncentral Feb 20 at 17:29
show 9 more comments

Generally speaking the answer would be a big NO. Of the hundreds of languages out there there is usually one that will suit your problem.

However there are circumstance where it is a rational option to develop a new language:-

  • When one of your competitors now owns one of your main development platforms. I am thinking of Googles current reliance on Java and their development of "go", (it helps if you have an author of the most successful language ever on the payroll!).
  • When you have to write a ton of code for a new platform and the existing languages are verbose and error prone -- e.g. php for web development.
  • When you come across problems of scale and parallelism which were never encountered before because nobody ever had this much hardware to process this much data before -- e.g. Scala and (to a certain extent GO).
share|improve this answer
add comment

You might want to read parts of Martin Fowler's upcoming DSL book, if you're thinking of writing you own language.

I can't really think of a business case to create a language from scratch other than it being a tremendous learning experience.

Edit: for DSL's there are plenty of business cases, but the key here is not to get carried away and Keep It Simple.

share|improve this answer
add comment

Tom Van Cutsem recently wrote an essay answer to this very question:

http://soft.vub.ac.be/~tvcutsem/whypls.html

Bullet summary (from that page):

  • Language as syntactic abstraction mechanism: to reduce repetitive "boilerplate" code that cannot be abstracted from using another language's built-in abstraction mechanisms.
  • Language as thought shaper: to induce a paradigm shift in how one should structure software (changing the "path of least resistance").
  • Language as a simplifier: to boil down an existing paradigm to just its essential parts, often to increase understanding and insight.
  • Language as law enforcer: to enforce important properties or invariants, possibly to make it easier to infer more useful properties from programs.
share|improve this answer
add comment

There is nothing wrong with writing your own programming language if you can leverage existing tools. In todays world this would mean that you either define it in a syntax usable to an existing language (like Java or C#) or write a small transformation system (macro expander) that generate code in an existing language.

Going all the way to machine code is reinventing a LOT of wheels...

A very good reason for a DSL is to represent domain data in a succinct way. This allows domain experts to work with the data directly instead of having to go through others. The trick is then to have the resulting programs in an easy to process form.

share|improve this answer
add comment

One reason is for educational purposes, as already stated. But there are more. For example, there are many research languages like Sing# on the Singularity operating system and BitC on Coyotos that have been designed because the existing languages don't offer the required features (for instance verification on a language level).

share|improve this answer
add comment

The last time I set out to do that on a hobby project I started specifying what I wanted the syntax to look like and realized partway through I was reinventing prolog. Other languages that might be a good fit when you think you need to invent a language are lisp, lua, or something like Haskell. Basically, all those languages you ignored in college because you thought they would never be useful.

share|improve this answer
show 1 more comment

Probably never.

Lua is the best choice you can get if you want to embed the language in basically any other language.

Small Domain Specifc Languages are currently in, and it make sense in some applications.

Other then that, reasons are mainly academical.

Creating a language when its not needed, is really a bad thing to do due to the complexity involved in developing and maintaing it. I have seen many projects that introduce some kind of scripting language specific only to that program, and it was the thing that was slowing down development of the base thing by huge amount. Good examples are for instance automation languages like Phantom, AutoHotKey, AutoIt. Those tools would be IMO much better if they used some well known emeding langauge like Lua.

share|improve this answer
show 1 more comment

When is it reasonable?

When you feel like it!

Don't listen to these people who have snarky comments that basically say:

"Don't do it because its too hard and X language is better than any language you can come up with".

The thing is, creating a DSL happens all the time. A framework is a DSL. A macro is a DSL. Every time you write a function for your program, that is part of a DSL. Sure, its within the bounds of the grammar, but vocabulary is part of a language. This is why industries often create their own vernacular: its more efficient!

If "don't do it" was the right answer, we would all be writing COBOL and Fortran.

share|improve this answer
2  
Really? I would consider frameworks, macros and functions all to be things that help a language maintain domain independence. –  CurtainDog Apr 21 '09 at 9:26
2  
@CurtainDog, it only becomes part of the language if its part of the standard library. Otherwise it is a "dialect" of the language. –  Unknown Apr 21 '09 at 22:30
add comment

What language is good at is compositionality or putting the same components together in different ways.

If your domain problem just needs you to set a bunch of orthogonal switches, a language probably doesn't add much over forms, a graphic UI or a straight text-config. file. (I'm assuming here that a file full of key, value pairs isn't what you mean by a "language".)

OTOH, if your configuration is like real language eg. verbs and nouns can be put together in many different (and novel) combinations to any degree of complexity, then a language is going to become almost inevitable, because the combinatorial explosion of trying to specify what you want by any other method overwhelms.

share|improve this answer
add comment

I don't think you can program without creating a new language, so it's good to realize that's what you're doing and understand the issues.

  • What is a language?
    Vocabulary, syntax, and semantics.

An off-the-shelf language like VB, Java, C#, etc. is just a base language. As soon as you add classes, methods, etc. to it, you have added vocabulary and semantics. There are many ways to implement languages - parsing & translating, parsing & interpreting, macros on top of an existing language, adding classes&methods to an existing language.

  • What do you want a language to do?
    Be good for expressing problems concisely.

How do you know if you've done this? The measure I use is edit count. If one-sentence requirement A comes along, I proceed to implement the requirement in code. When I'm done & got all the bugs out, I check in the code, and the code repository gives me a list of the changes I made, B. The smaller B is, the better the language is. Averaged over the space of real & possible requirements, that measure tells me how "domain specific" the language is.

  • Why is conciseness good?
    Because it minimizes bugs.

If it takes N code changes to implement 1 requirement, and you sometimes make mistakes, then the number of bugs you introduce is roughly proportional to N. In the limit where N = 1, it is almost impossible to introduce a bug without trying to.

Note that this is a direct challenge to the "code bloat" we see nowadays.

ADDED: In response to your request for an example, see differential execution. I won't say it can be understood quickly, but it does significantly reduce UI code.

share|improve this answer
show 2 more comments

My perspective is that DSLs are generally a "weak idea", and it's more productive in the long run to use a standard language and build your domain-specific needs as a library of the "non-DSL".

However, it may turn out to be that your needs are custom enough that it's preferable to have a DSL(not just a slightly modified gcc or lisp implementation) for your company. Many companies use drop-ins of current languages that target what they are doing, without writing/maintaining their own language. E.g., I've heard that PHP has a nice drop-in; Lua is designed around being a drop-in, ModelView uses Python and AutoCAD has AutoLISP as its scripter.

share|improve this answer
add comment

When to create your own language?

When you want to, as a large hobby project.

For a domain-specific language. These can be quite elaborate; look at what goes on in the Interactive Fiction (or text adventure) community by checking out the archive.

When your aims are very ambitious, and you think you can make a real advance, like Paul Graham's Arc project.

Also, in any sufficiently adaptable language (perhaps C++, definitely Common Lisp) in the process of developing low-level constructs.

When to avoid it like you would, I hope, avoid a cliche like avoiding it like the plague?

When it's to be the basis of continuing development for real projects. It will always wind up lagging seriously behind what's commercially available for cheap, and will cripple further development. I worked for a company with its own version of COBOL, and never want to work at another company that maintains its own language. We watched other versions of COBOL get better capabilities, and better tools, while we were stuck with the same problems. (I don't want to work with COBOL ever again, but that's another story.)

The situations in which you might create your own language don't fall into this. Hobby projects are not used for real development. Something like Arc will succeed (and get multiple implementations and further evolution and development) or fail (and nobody else will use it). A small domain-specific language is only part of a project, and since it's small it can be improved over time. A text adventure language is used to write individual games, and those games, in addition to being hobby projects, are almost never used for continuing development.

share|improve this answer
add comment

Except for self-educational purposes, I would like to claim that there is today no need whatsoever to create your own language. In any circumstance. Ever. Regardless of what you want to do, there are boatloads of existing languages you can take/adapt to your needs.

share|improve this answer
show 3 more comments

Learning exercises aside, it is reasonable to create your own programming language only when you understand other languages, your specific problem domain, and the way that existing languages address that problem domain and this understanding is thorough enough that you know a new language is a reasonable solution without needing to ask the question.

share|improve this answer
add comment

Wouter was known for creating a new language for any new idea. You can draw inspiration from his work: Wouter's programming language page.

share|improve this answer
show 1 more comment

You can, just don't catch yourself in the anti-pattern "Recreating the square wheel".

Meaning you are recreating whats already been done, only poorer than the original(s).

share|improve this answer
add comment

It's completely reasonable if done to extend your skills and to learn.

Other than that, if you have to ask the question, then it's not. If you're trying to figure out whether you can deal with a certain class of algorithm or a certain problem domain better than existing languages, first off you need to be an expert in the area you are addressing. You will know it is appropriate when your skills and experience tell you so.

And you could be wrong about that, too, but you'd need another expert to convince you of that (or to show you that you aren't the expert you think you are). A lively discussion that would be, not a simple Q-and-A as you'll find here.

share|improve this answer
add comment

Designing languages can be fun. But you don't have to ristrict yourself to programming languages.

If I build a moderately complex application, I like to add a kind of macro/scripting language to make it easier to execute complex repetitive tasks. Most users won't use this functionality, but the few that do use it are very grateful. Besides, i make sure it is valuable for the support people to help them fixing customer problems.

share|improve this answer
add comment

Seems like the main reason you'd want a new language is that you start discovering patterns in your code that existing languages don't handle well. But there are a bunch of problems with creating your own language. You'll be missing out on all the libraries and frameworks that are built up for existing languages. You'll spend a lot of time designing and implementing the new language, which is all time you don't have to spend on the real programming task. You'll spend a lot of effort convincing other developers that they should use your language. And, you'll have a hard time recruiting and training new developers.

Why not write in a language like Lisp that lets you extend the language as you discover new patterns? Then, you get all the power of a new language with all the benefits of an established language.

share|improve this answer
add comment

Only if your team's core business is programming languages.

I've worked on a programming language that was created in a financial company.

Clearly, for the architect himself this was a great challenge and improved his own skills.

Inevitably, the language couldn't grow or improve at anywhere near the rate that something like C# or Java could - they have teams dedicated to doing that.

The language soon stagnated as no one new wanted to take on the task of improving someone else's pet project.

The original architect left. The language withered and died after 10 years.

Those 10 years were hell for anyone who had the misfortune to work on a dead-end language.

So go ahead, create your own language but please don't ask anyone else to actually use it. Please don't expect anyone else to thank you for it.

share|improve this answer
1  
Interesting case study... could such stagnation be averted by targetting a language for say the Java or .NET platforms. That way the language can 'grow' as more is added to the base libraries. –  CurtainDog Apr 21 '09 at 9:35
2  
I'm not sure why you'd create a language that targeted another one like Java. Why not just use Java or C# to start with? –  Joe R Apr 21 '09 at 10:24
add comment

I suggest that the key questions are, "What problem am I trying to solve?" and "Who gets the ROI?"

If you're trying to build your own skills and experience, then full speed ahead, but not in a production system that's supposed to solve someone else's problem.

share|improve this answer
add comment

It is always "feasible", to use the word in your (original) question, but it is not very often useful and very rarely optimal given the abundance of well-supported and mature languages and frameworks that exist.

It is an interesting intellectual challenge, however.

share|improve this answer
show 2 more comments

It definitively depends on the situation. As nosklo said - If you have a good idea, a brand new concept or something like that I would highly recommend you to do that.

In general I would suggest to rely upon established technology.

But if you're interested in creating your own "language" you should check out: YACC & Lex

share|improve this answer
add comment

One reason could be to create it as an experiment to learn about language design and compiler building.

Another reason could be to build a scripting language into an application when you don't have the option to add a third-party API.

share|improve this answer
add comment

protected by MichaelT Feb 20 at 16:30

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.