What is obviously missing in this research?
There are four things that immediately come to my mind when asked, "What didn't you do that most reasonable engineers would care about, before making this kind of decision?" We have discussed our desire to continue this research in the Future section of the site.
This is a valuable, but by no mean all encompassing list.
- Issue 1: Security
- Issue 2: Performance
- Issue 3: Refactoring
- Issue 4: Enterprise Concerns / Integration
When we set out to attempt this research we immediately surrendered issues 1 and 2 to the arena of "Would be really neat, but not going to have time. Maybe in future research." The upside is that if you have the exact same hardware for all 8 Platforms then you could have an honest bake-off for performance and security. Performance of course itself being a very complex topic for example Load -vs- stress testing.
The security conversation is also a large and complex subject area. Would you have a team of hackers attempt to hack each box within some set of parameters? Would you use statistics about the number of sites hacked in a year, and what technology they are running? PHP is the clear loser if we do this, but should a technology be punished for the fact that it is widely used by what Microsoft refers to as "Mort's"? Bad programming in any language is bad programming, is a technology bad if it is accessible to the ignorant and widely used? This degrades into opinion very quickly.... so how would we objectively measure security of a technology platform?
The last two Issues we actually thought we would attempt to solve through the Profile 3 version of the application. In many respects we believed and still believe that Profile 3 is the most important profile to actually develop metrics around as it most closely reflects what we actually do for a living. Unfortunately, this also makes it the most time consuming. We were thrilled to get the profile's one and two across the finish line for seven of the platforms. Please consider we all had full time jobs and were taking a full load at W.P. Carey's School of Business.
We thought we would address refactoring by having all Profile 3 screencasts start from the final code base of Profile 2. Then force a change in database structure to challenge the flexibility of the different ORM frameworks. Additionally, we moved the database design away from the simple one object one table structure. This may also play heavily in the impact on performance, unless the practitioner of the particular framework anticipated this, but how well does their platform support them making this change?
The addressing of Enterprise Concerns or Integration is also a broad topic and is highly subjective given your industry and technology infrastructure, but the obvious things we could identify were a need to support LDAP/SSO, Web Services (SOAP,REST), some Messaging JMS or MQ, and I18N/Unicode/UTF-8 support. Many of these issues would actually be show stoppers for some of the platforms, forcing them to use their ability to wrap native code or limiting the options to accomplish the integration.
We are sure there are other issues, but are there any major ones that we have missed?