The tag has no usage guidance.

learn more… | top users | synonyms

0
votes
0answers
39 views

Download latest Excel files from a website regularly

First, I'll try and describe my problem, then I'll explain how I am trying to get around it. Problem: I regularly download excel files from the internet, this occurs every three months. There's a ...
2
votes
0answers
33 views

How much processing to do in the crawler? - good crawling practices

I am currently working on a pet project in Python with scrapy that scrapes several ebay-like sites for real-estate offers in my area. The thing is that some of the sites do seem to provide more ...
-1
votes
1answer
130 views

What should I consider when scraping content from 6k+ Websites?

I am working on a project that will scrape data from roughly about 6,000 websites. I am considering using a server/client model in which the server is responsible for scraping the data and importing ...
0
votes
2answers
222 views

Is there a limit to how often I should access a website programically to avoid DDOS? [closed]

I have my own website that displays some data I get from webscraping. I run the webscraping script on my website host server. I just realized that maybe I'm accessing other websites too quickly ...
3
votes
1answer
349 views

Android App with Ruby Backend Server

I'm working on a personal project to help me branch out and learn some new/different technologies. I'm a .NET programmer but I want to learn Ruby and how to develop Android apps. I have developed ...
3
votes
1answer
161 views

Scraping data from website and passing into Office - a lot of restrictions [closed]

Recently, I was asked to help with some side optimization project at our company, I've made some good research. I'm still not 100% sure if this is most efficient way to do this. Problem: Scraping ...
-2
votes
1answer
518 views

Crawler - programming language choice [closed]

I'm currently developing a web crawler. The first version was developed in Node.js and runs pretty well. The issues that I encountered with Node.js are in no particular order: slow URL and query-...
0
votes
1answer
339 views

Can I scrape a website for font stylings?

I am trying to scrape websites for valuable text for example the title of an article, the author's name, and other distinguished text. I cannot always guarantee that this sort of text will have ...
0
votes
1answer
629 views

Create a loop that visits every link on a site [closed]

I want to create an HTML scraper in Ruby on Rails and I want to implement a progress bar and an ability for it to pick up where it left off if it's interrupted while scraping. I think the best way to ...
0
votes
1answer
478 views

Scrape Intranet site without a web server

I'm trying to rebuild a simple c# time tool that displays certain statistics about time worked for the web, however, I don't have access to a server so I can't use server code like PHP which I would ...
0
votes
3answers
637 views

When there is no API

When it is necessary to integrate with a web application, and an API is unavailable, is it a viable solution to simulate a web browser interacting with the web application as a real user would ...
-1
votes
1answer
155 views

Getting Started with Data Collection and Analysis [closed]

I have a very very limited understanding of SQL and PHP. I have a few projects I want to try and work on, which will require doing some web scraping for data collection and then storing, sorting ...
4
votes
2answers
10k views

What is the way to go to extract data from websites? [closed]

I've been thinking about a side project that envolves web data scraping. Ok, I read the Getting data from a webpage in a stable and efficient way question and the discussion gave me some insights. ...
2
votes
1answer
601 views

Preventing crawler from interfering with user tracking

I'm scraping text from various webshops (no images/videos or other data). I'm no expert on user tracking, so I'd like to know if there's a way for me to write my crawler so it won't interfere with the ...
-1
votes
5answers
1k views

Blocking IP address for web-scraping service

Background Consider the following scenario: Link. User provides a link to some poorly formatted website (e.g., creative commons content). Scrape. Server downloads the content (web scrape), always ...
21
votes
3answers
5k views

What will happen if I don't follow robots.txt while crawling? [duplicate]

I am new to web crawling and I am testing my crawlers. I have been doings tests on various sites for testing. I forgot about robots.txt file during my tests. I just want to know what will happen if ...
0
votes
3answers
325 views

From where do financial firms obtain the stock data to analyze [closed]

I've been thinking about the subject and I've always wondered, from where do they obtain the data that is analized in applications, I've looked at Nasdaq's website and they do not seem to have any ...
2
votes
1answer
3k views

Improving performance for web scraping code

I have a website in which the code scrapes other websites for getting the accurate data. While the code works good but there a decent lag in performance because the code firsts downloads the html ...
-1
votes
2answers
587 views

How do I capture information from a website that doesn't provide an API? [closed]

Do you know any good tutorials, frameworks, anything that can help me to write code that captures information from a website that don't have a public API, or hasn't been written in a RESTful way? I'...
0
votes
2answers
7k views

A good tool for browser automation/client-side Web scripting [closed]

I'm interested in adopting a tool/scripting language to automate some daily tasks connected with fighting forum spammers. A brief overview of these tasks: analyze new registrations and posts on a ...
2
votes
5answers
1k views

Data Scraping - One application or multiple?

I have 30+ sources of data I scrape daily in various formats (xml, html, csv). Over the last three years Ive built 20 or so c# console applications that go out, download the data and re-format it into ...
8
votes
4answers
11k views

Patterns and practices for Web Scraping in .Net (C#) [closed]

I will be putting together an application to automate an external web site/application. In some instances I will need to navigate the site as a user would (some links I need to follow cannot be ...
68
votes
7answers
11k views

How to be a good citizen when crawling web sites?

I'm going to be developing some functionality that will crawl various public web sites and process/aggregate the data on them. Nothing sinister like looking for e-mail addresses - in fact it's ...
3
votes
4answers
9k views

Which language is the most flexible for scraping websites? [closed]

I'm new to programming. I know a little python and a little objective c, and I've been going through tutorials for each. Then it occurred to me, I need to know which language is more flexible (python, ...