Can anyone help me think of a way to dynamically generate the patterns section of a tool I am creating? I'm not sure how to store and generate these "patterns" dynamically.
What the program does is take a big list of links (100,000), puts them in a database, groups them by domain and then curls a page from each domain looking for backlinks.
Here is the database scheme, you can see that the Domains table is were most of the information is stored because we group the URLs together by the domain: http://sqlfiddle.com/#!2/9a4d7
So now we know that some domains are live (and have a backlink) and some are dead (no backlink). This is relatively easy but now for the fun part.
I need to derive "patterns" from the live links. For example find a list of all live link domains that have more than 25 links from that domain. So if joesblog.blogspot has 33 pages that link to my domain that matches this pattern. Here is my list of patterns:
- Domains that include a homepage link
- Domains grouped by top level domain (.com, .org etc)
- Domains that returned a 405 header response
- URLs with matching directory structures
- Domains that contain the word _ _ _.
- URLs that contain the word _ _ _ in their path.
- Common anchor text.
- Common title tags.
- Common backlink targets (what page of your site does the link point to).
The problem is that the patterns are changing CONSTANTLY. There being moved around, added, edited, removed and anything else you can think of. I really need a to build a content management system of sorts to handle these patterns. But how would I store something this intricate in a database?
Has anyone ever delt with a similar problem and how did you solve it?
If I could just store whole functions and MySQL statements in the database that would be great (but horrible wrong).
DISCLAIMER: This is an internal tool. Please don't ask me why I'm building this or claim that the requirements are wrong. This was designed by my manager and its my task to make it work because I am a developer at the company that needs this tool. Thank you!