Help me deal with higher level languages

Question

I'm trying to learn Scala (I have previously glanced at Erlang, Haskell, Ruby and had similar issues). I do enjoy it, but I feel bad about some things it does and I wonder if that's just due to all the time I've spent with C++.

1. Keeping it clean

I'll start with a problem of my own. With anonymous functions and loops hidden in foreach I feel encouraged to put everything in one line. I quickly end up with something hideous. It's even worse in Ruby where variables don't need to be declared. It's a mess. I wonder if there are any guidelines that can be followed to improve readability or if this is something one just has to get use to when programming in a concise language?

2. Basic control structures

While foreach, map, etc. methods work great in simple cases they fail me in more complex ones. Containers preserve order of objects, and in a way, these methods hide that order form me. Let's say, I want to build an array B from the elements of array A with prime indices. I could do

var i = 0
val B = A.filter( e => { i+=1; is_prime( i ) } )

but it doesn't feel right. There is a duplication of counters - one inside filter, and one of my own. I see no reason why there wouldn't be a function which knows the position of the element it is processing. Am I missing something or is there some logic behind it? Now I just feel encouraged to use the usual C style loops.

3. Scala's hidden inconsistency

Scala seems to boast having so much of it declared in it's libraries. With all the rules Scala needs to make those libraries feel comfortable, the language becomes as big as any other, which seems like missing the point, but that isn't what bothers me. What I dislike is Tuple. With it's special syntax, it acts like a language construct rather than an ordinary class. It feels like Scala is denying that many things are built into it. I'd appreciate it if someone could explain to me why such synthesis of library and language is so often implemented (not only in Scala).

4. Efficiency

Firstly, I understand the benefit of having immutable objects in parallel algorithms, but how often do you actually need the concurrency? There are issues with it other than shared mutable objects. My point is that concurrent design has it's limitations and using it everywhere, like functional languages seem to do, must be terribly inefficient.
Secondly, even if we do consider immutable state a necessity, some algorithms I've seen are ludicrous. I'm thinking about the ones that work with lists as head::tail. I find it hard to believe that an algorithm, described in that way can be efficient. Again, I feel encouraged to write more imperative code. Am I underestimating optimizations or should sensitive algorithms really avoid some concepts?

Answer 1

This is mostly from a Ruby perspective.

1: Have you heard this joke?

Patient: Doctor - it hurts when I do this!

Doctor: Well, don't do that then.

If you're doing something and it leads to unreadable code, stop doing it. Just because you can put something on one line, doesn't mean you have to. It's possible to write very readable code full of anonymous functions, but you do have to think about it.

As for declaring variables, I'd say that that's just accidental complexity that you're shielded from in Ruby, and not having to declare everything makes the code cleaner. But that's a matter of opinion.

2: When you're iterating through a collection, you generally don't need to know the index of the item - all that matters is that you're doing something to each element. The index is an implementation detail you shouldn't concern yourself with. In the case where you do need an index, in Ruby you can do:

arr = ['cat', 'dog', 'monkey']
arr.each_with_index { |animal, index| puts "#{animal} is at index #{index}" }

3: Blurring the distinction between language construct and arbitrary code is one of the things that makes higher-level languages so expressive. In Java, depending on the types of the arguments you would either do a.equals(b) or a == b. In a higher-level language you don't have to worry about that distinction. The client code just has to know it's comparing two objects, and the library code can worry about how that works.

4: If all you care about is efficiency, code in C++ or assembler. Higher level languages aren't built primarily for efficiency (though many can be very efficient if used sensibly). They're designed to be easier to read and write, and to convey your ideas in a way that other programmers will understand.

Answer 2

It's hard to answer the first question because you don't show what you mean by it. Maybe it is something you should get used to, maybe you are going overboard with it. Breaking long computations into subparts is encouraged, even if the language don't actually impose it.

When it comes to indices, you should avoid using them. An index refers to the absolute position of an element, which is hardly ever needed in algorithms. More often, what is important is the relative order. For example, if you need to compare an element to its adjacency, you can use sliding. If you truly need absolute indices, use zipWithIndex.

Tuple is, indeed, an exception. However, aside a few concessions to Java interoperability, it is the only one that is more or less gratuitous. Others, such as function, are required to achieve goals of the language.

As for efficiency, you are obviously ignorant of persistent data structures. For instance, head::tail is constant time, both when composing and when decomposing. For instance, look at this implementation of split (the first one), which beats Java's precisely because it uses List and :: instead of array. And how many times have you cloned some data structure before passing it or when receiving it? How many times have you worked around a function being destructive? You never have to do that with immutable data.

Answer 3

Flexible control structures

As others have said, Scala actually has a really nice high level syntax. For example, generalizing your code a little,

for ((e, i) <- elements.zipWithIndex;
     if isPrime(i);
     ...) yield e

The nice thing about this is that it's all user extensible. Scala for comprehensions translate into regular calls to methods map, flatMap, filter, and foreach, which you can define yourself on your own classes.

This high-level style does incur a performance penalty. For maximum performance, you might want to go with while loops, and you'll get almost exactly the same byte-code as the equivalent Java.

Immutability is optional

Regarding your point about efficiency, Scala tries to make a functional (immutable) style natural and easy, but it certainly doesn't enforce it or even twist your arm. The best evidence of this is to look at the implementation of Scala's collection libraries, where mutable variables and while loops are commonly used for brevity and performance (rather than "purely functional" tail-recursion, say), even when constructing immutable collections. The external interface presented to the user is more important than the internal implementation details. Overall, Scala is very pragmatic in this aspect.

Benefits of a functional style

As you mentioned, one benefit of immutable data structures is that they make concurrency easier. But there are additional strong benefits to using a functional style,

• It is easier to reason about the correctness of code with immutable objects. For example, if I call a big, complicated function that someone else wrote, I can be sure the function can't change any of the immutable objects I pass as parameters.

• Persistence of immutable objects makes it easier to implement certain code. For example, modifications to a mutable Array basically "lose" the original (unmodified) array. However, an immutable Vector in Scala will be preserved no matter what operations you later perform (if you want to replace an element, say, you can simultaneously have a new Vector and the old unmodified one). Scala lets you have your cake and eat it too, because operations to create new modified Vector instances are efficient (order log(N)). As Daniel Sobral says, immutable data structures completely obviate the need to clone (deep copy) anything.

Answer 4

don't worry - all these are common issues.

1. this comes with practice. also, your choices of scala and ruby are perhaps not helping.

2. i can't help you with scala syntax, but you would probably enumerate the data structure (create a stream with (index, value) tuples) and then work on that.

3. that's scala, unfortunately

4. cons lists are actually pretty cool. this sounds like your still stuck in a "random access" "array like" mindset (also apparent in 2 above?). again, much of this can be fixed with practice, but some is a real issue with functional programming. at the end of the day, you may want to return to an impure language - that's completely ok - but i would suggest sticking with functional ideas a little longer first, until they become clearer.

overall, i would say that if you're noticing these problems then you're close to overcoming them. and i would suggest that going back to haskell might help. it's more painful in some ways, but will help with (1) and (3) and so help you focus (you're grouping together a lot of ideas in "high level languages", and swapping between a lot of languages; each is different).

alternatively (and this worked for me, i think), give up on the whole idea for a while - say, 6 months on another project, in a language you are familiar with - and then come back to it. sometimes you just need time for the new ideas to sink in.

either way, what i'm trying to say is: you're not wrong in what you're seeing, but you're also not good enough yet to separate the real problems from a lack of fluency; it's normal to find this confusing; you're making progress; you just need to find your own way to move forwards.

Answer 5

• Keeping it clean. The usual general rule is that reading good code (in the language of choice) is helpful. More specifically, functional code is more compact - but that also means that you should have even smaller functions with meaningful names, to improve again readability. For instance, consider this Position class, taken straight from actual code implementing an hexagonal grid for a board game:

  object Pos {
      val directions: List[Pos] = List(Pos(-1, 1), Pos(0, 1), Pos(1, 0))
      val increments = directions ++ (directions map (- _))
  }
  case class Pos(x: Int, y: Int) {
      def +(p: Pos) = Pos(x + p.x, y + p.y)
      def *(i: Int) = Pos(x * i, y * i)
      def unary_- = Pos(-x, -y)
      def adjacents = Pos.increments.map(this + _)
      def adjacentTo(p: Pos) = this.adjacents.contains(p)
  }
  

  All those methods are one-liners, and many of them could have been defined inline in client code. Yet, giving them names will make the client code more readable - see adjacentTo which is a client of adjacents and computes whether two cells are next to each other. I don't claim this code is perfect, but that often the size of a function in a functional program is smaller than in Java.

  • Efficiency: In addition to the others, I'd like to point out that you are worrying about efficiency of details; while writing your program you should concentrate on having high-level efficient algorithms for the places where it really matters. "Premature optimization is the root of all evil", as D. Knuth said: often to optimize code you must reduce its clarity or maintainability. You should do so only for hotspots identified through profiling, since they will be very often unexpected. And also for them, before addressing the details you should concentrate on the big picture.

Answer 6

With respect to ...

Keeping it clean and** Basic control structures**, you are right to wonder if there are any guidelines that can be followed to improve readability, but that doesn't mean that you won't have to get used to it : )

For example, to build primes in a more "readable" way ... it is recommended to use the userscore character:

def main(args : Array[String]) {

    var a = List(0, 1, 2, 3, 4, 5)

    var b = a.filter(isPrime(_))

}

def isPrime(n: Int) = (2 until n) forall (n % _ != 0) 

The underscore is used in several different circumstances ... mostly when you want a variable name but don't care what the name of the variable is, as in the case above.

To make the above code even more concise:

    var b = a.filter(n => {(2 until n) forall (n % _ != 0)})

but personally ... I like the initial one much better. If I encountered the expression just above, my initial reaction would be to "extract method" to make the code clearer.

With respect Scala's hidden inconsistency and what you call the synthesis of library and language ... I see it less as the synthesis between library and language but the ability to blur the distinction what the language designers provide and what others can provide in that language (whether it be in the library or in my own code.) Ultimately providing a language that is more able to change with my own needs.

Martin Odersky explains in Programming Scala:

Eric Raymond introduced the cathedral and bazaar as two metaphors of software development. The cathedral is a near-perfect building that takes a long time to build. Once built, it stays unchanged for a long time. The bazaar, by contrast, is adapted and extended each day by the people working in it. In Raymond’s work the bazaar is a metaphor for open-source software development. Guy Steele noted in a talk on “growing a language” that the same distinction can be applied to language design. Scala is much more like a bazaar than a cathedral, in the sense that it is designed to be extended and adapted by the people programming in it. Instead of providing all constructs you might ever need in one “perfectly complete” language, Scala puts the tools for building such constructs into your hands.

and then goes on to give an example in Scala and Java using factorials:

Many applications need a type of integer that can become arbitrarily large without overflow or “wrap-around” of arithmetic operations. Scala defines such a type in a library class scala.BigInt. Here is the definition of a method using that type, which calculates the factorial of a passed integer value:

 def factorial(x: BigInt): BigInt =
      if (x == 0) 1 else x * factorial(x 1)

Now, if you call factorial(30) you would get:

 265252859812191058636308480000000

BigInt looks like a built-in type, because you can use integer literals and operators such as * and with values of that type. Yet it is just a class that happens to be defined in Scala’s standard library. If the class were missing, it would be straightforward for any Scala programmer to write an implementation, for instance, by wrapping Java’s class java.math.BigInteger (in fact that’s how Scala’s BigInt class is implemented).

Contrast that to using BigInteger in Java ... and you start to see why the synthesis of language and library is desirable:

import java.math.BigInteger

def factorial(x: BigInteger): BigInteger =
    if (x == BigInteger.ZERO)
        BigInteger.ONE
    else
        x.multiply(factorial(x.subtract(BigInteger.ONE)))

The previous example demonstrates that Scala lets you add new types that can be used as conveniently as built-in types.

And if that wasn't enough ... from Odersky:

The same extension principle also applies to control structures. This kind of extensibility is illustrated by Scala’s API for “actor-based” concurrent programming.

And finally ... on Efficency, I agree. Using functional languages everywhere does have its limitations and in some cases is either inefficient or complicated. That is one of the reasons I especially like Scala. It doesn't force me to use functions over objects where objects and mutable state is the better choice.

Answer 7

To summarize and, hopefully, simplify what others have said here, you are going through a barrier on a road from one theory of programming to another very different conceptual framework. Your journey is actually even more difficult, because your path leads you even farther than that into a brave new programming world!

Firstly, you are grappling with significant differences between the object-oriented vs. functional programming approaches. That test of courage is a very old initiation rite that has blocked many before you. It is similar to the terrible time most of us had adapting from C to C++, with what then seemed C++'s infernally inefficient way of doing things. I suggest that you read, or at least browse, the time-honored old standard, SICP, The Structure and Interpretation of Computer Programs, for which free versions are still available (SICP in PDF, and SICP in HTML). Understanding the fundamentals of functional programming will smooth your ride considerably.

Secondly, however, your transitional barriers are yet higher. Some of us have noticed that we like certain features of object-oriented programming and certain features of functional programming. We want to have our cake and eat it two [sic (not "sick")]! This has led to a new wave of languages that some have called "object-functional." In these newer languages—among which you will find Scala, Fantom, and F#—you will find features from both programming paradigms. Perhaps this will remind you of the history of your own C++ programming language, which permitted blended procedural and object-oriented programming in what initially seemed to many of us a toxic brew! Many bottles of aspirin, acetaminophen, and ibuprofen were emptied before we collectively developed best practices that made C++ the workable and powerful tool it is now. When dealing with an object-functional language, such as Scala, you will want to carefully consider how to select the best tools from Scala's kit to solve the particular problems on which you are working. Apply the right tool in the correct context and you will find that Scala repays you handsomely for the time you invest in learning its proper use.

So I advise you to continue along the path you started because I can sense that you saw some vision of something worthwhile down that long and winding road! I assure you that many who have gone before you would not gladly turn back, perhaps as you would probably not care to revert to assembler language, except under extreme duress! You are in pursuit of a worthy goal and I wish you well!