Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We have a requirement in our project to generate a big CSV file every 2 hours using a Java program.

This file will have around 60,000 lines (around 120 characters per line). I am not sure about the size yet.

I would like to know if I would run into any memory issues because I will open the file using FileWriter and then keep writing and then finally close the file.

Should I worry about the size of the file? If yes, are there any other good techniques to write into a big file in Java other than using FileWriter?

We are using Java 5.

share|improve this question
    
What version of Java are you using? –  Martijn Verburg May 8 '12 at 13:39
    
Java 5, updated the question. thanks. –  java_mouse May 8 '12 at 13:40
    
Yo... I mean Nes. Damnit, lemme get my magic 8 ball. –  Will May 8 '12 at 18:10
    
If speed matters, try a really big (multi-megabyte) buffer. It sped my file writing by a factor of 10. Of course, your results may vary.... –  RalphChapin May 10 '12 at 16:56
add comment

4 Answers

up vote 13 down vote accepted

No, you shouldn't. The point of a file is to store things outside random access memory; the FileWriter's size is constant, and likely to be pretty small, all things considered, even if it's a buffered FileWriter. The constant rewriting might cause I/O load or CPU spikes, but almost certainly not memory shortage.

share|improve this answer
    
I thought the file data will be kept in memory until I close the filewriter? looks like my assumption is wrong. –  java_mouse May 8 '12 at 13:43
8  
Nope, a BufferedFileWriter will keep a certain amount of data in memory, but that amount is a fixed quantity, depending on the environment it's called in. It doesn't grow unbounded with the number of bytes you push through it - that would be a recipe for disaster! –  Kilian Foth May 8 '12 at 14:05
add comment

As Killian Foth wrote it, you should have no problem at all, 60000 lines is not at all that big. I just wanted to suggest you to use any of the free CSV parsers provided here under the "Commons CSV" initiative at http://commons.apache.org/csv/ instead of writing your own implementation.

I've used Super CSV for a few projects and I certainly did not have any problem with it.

share|improve this answer
1  
I used openCSV. I like it. 60K lines is nothing. My old laptop processes that in a second. –  ahoffer May 8 '12 at 16:29
    
Great to hear Jalayn! We've just released a new version of Super CSV with heaps of bug fixes, new features and a brand new website. Oh, and it's now in Maven central :) –  Hound Dog Nov 7 '12 at 21:24
add comment

Do not use FileWriter. Not because of performance concerns (Java's IO classes do not keep everything in memory, ut 60k lines is nothing even if they did), but because it does not allow you to choose the character encoding. It will implicitly use the platform default encoding, which means text outside ASCII can get corrupted.

Instead, use an OutputStreamWriter wrapping a FileOutputStream. Or, even better, a CSV library, which should handle all these issues.

share|improve this answer
    
Or instead of OutputStreamWriter, use NIO classes (FileChannel with ByteBuffers)? Do all the CSV libraries handle encoding? I took a quick look at SuperCSV, and didn't see anything about handling encoding. –  Sam Goldberg May 8 '12 at 16:04
1  
@Sam Goldberg: you're right, it seems to operate on Reader/Writer and leave that concern to the caller. –  Michael Borgwardt May 8 '12 at 17:39
    
@MichaelBorgwardt You're right - Super CSV was written using IoC, so it's up to you to supply a reader/writer - that way you can write to a file, zip file, HTTP response, etc. We've just released a new version - please check it out :) Oh, and as for character encoding, I've always found Joel Spolsky's article on Unicode to be excellent. –  Hound Dog Nov 7 '12 at 21:18
add comment

You might consider using BufferedWriter, though that probably won't help significantly with performance, it's a best practice in any case, since I imagine the number of lines won't always be 60,000.

Have you considered zipping the file afterwards? If you intend on having a lot of these files laying around, might be in your best interest to zip it up after it's been written to, especially if you're going to be creating these files once every couple hours.

For what concerns memory, you probably have nothing to worry about unless you're working on a system with very little memory, in which case you should use BufferedWriter and explicitly set the buffer size.

share|improve this answer
1  
What's a BufferedFileWriter? –  Michael Borgwardt May 8 '12 at 14:34
    
Oops. I meant BufferedWriter. Fixed. –  Neil May 8 '12 at 15:10
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.