1

Whate are alternatives to using cookies/client-side storage for a PHP/MySQL based site on Apache.

Scenario/Requirements:

  • I want to try using some anti-bot code to prevent specific scrapers etc. from accessing the site.
    • I would like to run this code before launching the rest of the site (before DB access etc.).
    • I don't want to constantly run the same code on every page-load after a visitor has passed the initial check.
  • I'd like to avoid the use of Cookies/Client side storage if at all possible.

The only solution I can currently think of is to write files to the server based on the visitors IP/UA, or to write a list of them to a single file.

Yet this has the limitation of multiple users through a proxy/same connection, etc ...

So, any ideas/suggestions?

Or am I simply over working the issue?

Improve this question

3 Answers 3

Reset to default
3

In general case, in order to keep track of users and for example to know which one is logged on, sessions are used. But sessions mean session IDs stored in cookies¹; it's just that if you need to remember N entries for a user, there is only one cookie involved, not N.

In your particular case, on the other hand, you care about protecting your website against illegitimate use (and avoid repetitive checks). It means that sessions are not appropriate: why a scrapper would ever send session cookie to you?² Instead:

  1. Detect the IP address of the request before starting processing it,

  2. If you have no information about the IP address:

    • Detect if it is a scrapper (like you currently do),

    • Store the result in cache (or in database, or both),

  3. Load the result from cache.

  4. Respond according to the result. If it's a scrapper, send a "please stop scrapping" page; if it is a legitimate user, process the page and send the ordinary response.

¹ PHP is also able to pass session ID in URIs. It's ugly and must be used only when you can't do anything else.

² Remember, legitimate users may have their cookies disabled for some reason. The fact that a client doesn't send the cookies to you doesn't automatically mean it's an automated process.

Improve this answer
4
  • The problem I'm having is how to store a value somewhere that means I can call it and check it before running the anti-code checks again. I don't want to have to run that every single page-load if they already passed once. You mentioned Cache ... in what way do you mean cache? Keep al og file and see if they are in that file ... if so permit access without further checking? (Thank you for the answers so far)
    – theclueless1
    Apr 3, 2012 at 13:59
  • My concern with this method is that the "Please stop scrapping" page would need to be designed to allow for legitimate users to say "Hey, I'm not a scrapper. I can verify I am human"
    – John Ruf
    Apr 3, 2012 at 16:19
  • @NeoModulus: how scrappers detection is done is outside of the scope of the question. And yes, if the OP does need it, he can create a method which will let the legitimate users to remove their IP address from blacklist (for example by filling a CAPTCHA).
    – Arseni Mourzenko
    Apr 3, 2012 at 16:29
  • I'm not overly concerned about blocking legit users. I seriously doubt any real person will be attempting to hit specific URLs directly (why would a normal person be trying to access my site with "?r=http://...." etc.? // Yes, I may well provide a simple method to unblock ... yet I will jsut as happily block them again (and for longer). I see no point in allowing them to retry an attack, and if they can pattern-out the auth method, then they can automate that too. // So do we think logging to a txt file should be sufficient/viable?
    – theclueless1
    Apr 4, 2012 at 15:04
1

It sounds like you want to be using Sessions. Try this tutorial for more information if you are not already familiar with sessions: http://www.tizag.com/phpT/phpsessions.php

Improve this answer
1
  • Thank you for the response. I'm looking at sessions - but they still require a Cookie (bots don't often use them) or URL appendments (ugly and alterable/avoidable).
    – theclueless1
    Apr 3, 2012 at 14:03
0

In a stateless environment you will have to persist the value somewhere; database, cookies, even a text file on disk. Some ways of persisting are better than others.

It sounds like you are more concerned about the performance hit of retrieving the persisted value. I would pick a solution and log how long that section of code takes per page load to test if the solution is acceptable. You can further refine your tests to see how much network/disk IO is used to retrieve the persisted value.

You may consider approaching the problem differently as well. You could add a user authentication element with captcha to reduce automated site access if the sites requirements allow for it.

Have you considered how to address if elements of your site such as images stored on disk are accessed directly and bypass the code check?

Improve this answer
1
  • In reverse... Yes - I've figured a way to rpevent any direct access to files. I can force everything through a script in one form or another. // I'm not worried so much as to access/retrivial times so much as constantly querying unneccesarily (Why should I check against a users IP on the 40th page hit, when they have passed the previou 39?). // The reason for the desire to not hit the DB is that it my sites already use DBs - and some attacks may be aimed at the DB. At least a flatfile approach may be a tad more secure?
    – theclueless1
    Apr 4, 2012 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.