Summary:
1. Start simple with technology already there, facing re-scalability issues later OR
2. Write a (to your mind) scalable system that will cost hours of work, eventually performing computationally worse in the very large scale..


Hey guys,

I just started a new project, and this time I am not sure where it will take me from a computational point of view. And I am breaking my nuts and bolts thinking about the issue of scalability.

I know there is a lot of literature out there on the issue. But it didn't really help me on coming to grips when regarding scalability issues.
What I came across, however, are descriptions of how other companies facing the same problems have had. In that sense most started with no abstraction of their software model from the tools they used. Starting with MySQL and Apache was always a good start, it seemed, and later parts of the software have been rewritten.

I mean there are basically the following choices:
1. start with a simple achitecture and code as usual, using mySQL and regular apache, working with PHP, JS, and python.

2. Write own database concept and own webserver that would do what I want them to do. (bogus..)

3. Use what is already there (as in 1) but virtualize the actual architecture from its environment (Webserver = Apache + other stuff), Database (virtual data-space mapped into SQL/XML etc DB spaces), etc.

Forgetting 2 there are two options left. Start simple and see where it goes, eventually having to rewrite your entire code and break your head about how to re-scale the system once people are using your software (yes, it's going to be some sort of social network.) ; write a perfectly consistent and scalable system from scratch and make it work on paper, and spend millions of hours about how to make it work (writing you virtual machines).

Or maybe a trade-off between option 1 and two, using common virtualization methods, which would make the system more scalable, but which could turn out to not give me the consistency during expansion later.

I mean writing software concepts when you know what your program shall do is fun, and I am sure it makes fun, too, to write the VMs. But on the other side, there is no way to tell that there is at all a way to virtualize the system, and even more importantly, the entire system might perform terribly bad after all from a computational point of view. (Desiging the consistent architecture to be comptuationally efficient is one thing, including the VMs and the underlying architectures seems impossible to me given by background).

So what I ask you is what your gut-feeling, what you experience tells you what I should do.

Thanks in advance for any help on the issue. Some ressources would help or maybe there is like this big "tag" on the subject that I didn't yet come across and that I just have to type into google to find all the answers. I didn't manage to do so, yet. So sorry if my question is either trivial or has already been answered.

Sincerely,

Jerrey

I would put the infrastructure in place to be able to scale later with minimal downtime.
Depending on what data you are storing I would suggest looking in Cassandra: The Apache Cassandra Project as this seems to be a lot more efficient that using SQL for storing certain information.

Also you may want to ditch Apache and look into nginx: nginx

Cache as you much as you can before PHP reads it, use a faster more efficient language like C or Java to do the heavy lifting and leave Ruby/Python or PHP to be the front-end languages which basically just render cached and preprocessed data.

__________________
I'm off to sigcont.com

Callum Jones

I have to agree. Here, it's important not to reinvent the wheel, but you also have to make sure the wheel does everything you will need it to, or can be easily extensible. This solution fits that.

Scalability hasn't been an issue for me with web applications - most major applications that go through all this trouble end up writing their own server software in C, like was suggested.

And for kind-of avoiding scaling issues, avoid Ruby. I haven't used it, but I hear nasty things about people having to rewrite in other languages when their apps grow too popular.

__________________
"My code does not contain any bugs; it randomly generates new features."
WikiHowNot

I wouldn't say avoid Ruby. RoR, PHP, Python/django are great for front end languages, any heavy lifting should be best left to Scala/Java, C, C# etc

There isn't really a need to rewrite web application servers if you choose robust ones like nginx or lighttpd.

Sorry for the late response. I skimmed through it earlier, though.

And I was kind of dumb. I don't know why I didn't think about splitting it into two areas, like a front-/ and backend.
Of course it is failry easy to write my main architectural principles on the server-side, and using the standart tools (servers, etc) to let a half-baked front-end version of the system communicate efficiently with the back-end system. I don't even need to virtualize or anything and can think about scalability later, looking at the different modules. So, yes, thanks for bringing that up ^^.

And to the other suggestions. nginx looks preferable and I actually use it now. Kassandra seems to go into the right direction but I haven't get to actually work with it yet.

Concerning front-end / back-end computation and data processing, I mostly rely on C actually for the back-end and serve from cache as stated.

So thank you.