I'm working on a "user submittable" PHP tutorial website, and I was wondering what URLs I should allow for users to submit.

For instance:
www.something.com - should I allow this or not? because it's a website rather than a specific page of a tutorial

www.something.com/this.html/htm
www.something.com/this.php/asp/cfm
www.something.com/this.php?id5 /asp/cfm
As well as folders..

Any other file types i should allow? I want to be quite strict as I don't want users linking to other things..

PS: I put this in general programming, as I'm not asking for actual coding help- and this section needs filling up .

__________________
MitchStanley.co.uk - Coming Soon

You simply can't guess the contents of a page by it's URL(s). The only way to be strict is to actually look at each URL that gets submitted.

You might also consider it necessary to check periodically for changes to those websites. Some one could post a URL that is a tutorial one day and something completely different the next. Perhaps for this you could keep a cache of the page's source and check for changes greater than 10% using whatever metric so you can flag it as "possibly changed and something you should check out". You could automate this if you're allowed to run cron jobs on your server.

I imagine there are still ways around this though e.g. A page of 'junk' with a fast-acting redirect -- you'd only have to change the URL which could be less than 10% (for example) if the page is padded out with fluff.

Any system is going to require manual checking to begin with and regularly there-after if you want to be strict.

Edd

__________________
Visit me at: mr-edd.co.uk
Languages: Python | Lua
Compilers: MinGW | MSVC++9
Libraries: Boost | gtkmm
Reference: Dinkumware | a.c.l.l.c-c++ FAQ

I plan on adding a "Report Bad link" option for users to users which will help bring attention to changed websites, and a rating system for un-useful/incorrect tutorials which will help prevent some things. I suppose that's all I shall do for now..

Except prevent links to zips/rars, executables, media files..

One thing worth noting is that if you use the fopen command to open an HTTP stream, and the URL you're trying to access is invalid or generates an error response, (i.e. 404 Not found), the fopen call will return false. Which could come in quite handy to double check links aren't broken.

I don't really agree with a URL checker because people can use folder names or a rewrite engine (i.e. mod_rewrite and URL shorteners) to explain the URL. Also, there are many other languages that things could be written in for it to work. What you should do is disable certain extensions that could possibly have malicious stuff or unsavvy or canny content. If you want something interesting, you could have a user-based acceptance system where tutorials would go in a certain section until checked as legitimate by a couple users or something to that extent. Just my two-cents worth, because there are too many extensions to count that are legitimate on the web, and too many other ways to bypass extensions.

__________________

I would personally just check to headers that are returned from the page--emulate a browser, but only send the headers to get just the headers back (I can look into that if you want me to.)

I think that the headers option is a good one. I think fopen() would not encounter problems with URL rewriters or supplied folders, because the request is sent to the server the page holds, and will be managed by it so that the correct page is returned.

I think a URL pattern that should be disabled is links to people's local hard drives :P

The "report broken link" option could be extended to other things too, since there are more than just links to be wary of on the Internet these days. For instance, an excess in images/ads that causes page loads to be really slow, offensive content surrounding the tutorial, an excess in popups or malicious or dangerously broken scripts, etc. It is possible that the person who submits the link does not think those things are much of a problem, or it is possible that those things change over time. There could be other reasons.

... Maybe to check if a link has changed you could, instead of caching source code, cache a few paragraphs of the tutorial, so that you can tell if the tutorial is still there or not. I.e. you could copy in three random paragraphs, and then look for them in the source code of the page, and if none of them are there, the link should be checked.

_jameshales

__________________
Death to the non-believers!