Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Do you know how Data Scientists use Python and if it requires data, where I can get sample data to practice with? I'm currently learning the language and plan on applying for Data Scientists jobs, so it would be helpful if I knew if there were specific types of tasks where I'd use python. If it's relevant, I use C# regularly.


migration rejected from Jul 25 '13 at 6:17

This question came from our site for professional and enthusiast programmers. Votes, comments, and answers are locked due to the question being closed here, but it may be eligible for editing and reopening on the site where it originated.

closed as primarily opinion-based by gnat, MichaelT, GlenH7, Dan Pichelman, BЈовић Jul 25 '13 at 6:17

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

Data is most valuable when it can be shared and the results of analysis can be shared. I have seen a number of Python projects make great use of Django. I would recommend that you invest some time getting to grips with the basics of Django. – PenFold Feb 11 '12 at 15:20
I do not agree with the point stated above, analysing data and presenting your results are two different things. – AsTeR Feb 11 '12 at 15:53
I'd assert that "data scientist" is a pretty vague term, straddling about a dozen fields and a whole lot of definitions. Any chance of some more specifics on at least the industry you're looking at? – Fomite Feb 11 '12 at 20:38
Good point. The software industry. – user1106278 Feb 11 '12 at 21:13
up vote 4 down vote accepted

Any type of data parsing is going to be of two types. Ascii or binary.

I would practice and learn how to 'get' data from multiply sources. This could be from files, from network connections, from stdout of subprocesses, serial port... Etc...

for ascii data, you would must probably be using strings methods for splitting and storing the data.

Binary data is where you'd really want to practice. You have to worry about the endianess of the data, worry about the specific bit lengths of different datatypes on different platforms, following specifications for the format of the data, actually parsing the raw data (get familiar with 'struct' and packing and unpacking data).

as for examples, search for tutorials on each piece I've mentioned. Most items you can create a temp generator of data to create the other side of a process (such as a tcp server, a subprocess spitting out stdout, something to generate files).

Good luck

Thanks, g19fanatic. That's exactly the type of detailed answer that I wanted. I remember pulling out large chunks of hair dealing with endianess. Thankfully, it'll be less of a problem this time around because I have less hair. So Data Scientists use Python primarily for data parsing? – user1106278 Feb 11 '12 at 19:09
It is apart of it. It all starts with parsing without parsing, you dont have anything to analyze – g19fanatic Feb 12 '12 at 18:23

I'm not the most experienced in that field but I use Python for data mining and experimental data analysis and modification.

Why Python for that ?

  • Many library designed around science and data analysis (yes SciPy, Numpy but also some more specialised like Pandas and NetworkX and some good visualisations utility like matplotlib)
  • Python transforms my ideas into code in very few lines of code
  • Python learning curves let you easily dive in the code (use some nice interpreters like: Reinteract or bpython ). e.g.: file i/o and text processing is trivial so I frequently use it just to write small format converter.

I wrote an article in french on "Why Python ?" if you can read it.

If you want some data you can found some on sites like UCI Machine Learning Repository :

  • some are ready for processing in data mining
  • some others require some preprocessing like this one.

** EDIT **

A good video about Pandas at PyCon.

Thanks, AsTeR. I forgot about UCI's database. – user1106278 Feb 11 '12 at 19:04
Remembering old times at school uh ? ;) – AsTeR Feb 11 '12 at 19:50
Shudder... The only memories I have of school are memories I'd rather not have. :) – user1106278 Feb 11 '12 at 20:38
+1 for introducing me to NetworkX. Looks very promising from their website. – Buttons840 Feb 16 '12 at 17:10

I'd recommend this book.

Machine Learning: An Algorithmic Perspective

It uses data sets from the UCI Machine Learning Repository. Even if you don't precisely want to do machine learning, but are more interested in statistics there are a bunch of data sets here that might interest you.

The main library for doing science with Python is Numpy. Enthought has a great Python distribution which includes Numpy, matplotlib, and many other libraries to get you started.

Also check out: Numpy 1.5 Beginners Guide. It's a great introduction and will get you up and running doing simple things quickly. The author's blog also has some good introductory material.

Hope that helps. An aside (noting you're a c# programmer as well). I still haven't been able to get Numpy to work successfully in IronPython. I have to use normal CPython. If you happen to get that figured out please let me know! Having Numpy within .NET would be really nice.

Thanks, Svaha. I'll be sure to let you know. – user1106278 Feb 11 '12 at 21:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.