CTO Roundtable: Big Data
Interesting discussion led by Michael Brown, CTO at comScore at our CTO Roundtable breakfast meeting.

Michael and his team manage and analyze one of the largest datasets processed by a local company. It is fascinating how thinking has to evolve when the volumes reaches billions of rows of data per day. Some of the key takeaways for me were:

• Leverage sort before processing
• Shard the dataset to create smaller more manageable files
• Parallel processing improves turnaround and allows for scalability
• Open source tools have reached a level of maturity to be relevant solutions to consider
• Most of the processing is now accomplished on smaller commodity machines
• Machine memory is more of a limiting resource than hard disk
• Solid-state disks can be selectively used to improve performance for IO intensive processes
• Security is best managed by selectively separating and focusing on securing sensitive data
• SQL is still king as it is very easily grasped by non-technical folks. Therefore, it is easier to find and develop talent with SQL skills.

Micheal has clearly lived through the evolution of technology for managing Big Data over the past 10 years. I enjoyed the event a lot.

Exploring the power of 4G
I moderated a panel at this event organized by the Northern Virginia Tech Council (NVTC). The panel was nicely balanced between the Telcos and users. After an initial discussion about what is 4G, the discussion focused on what applications does the audience see enabled by 4G. Most of the discussion was focused on new applications for mobile, not essentially requiring 4G.

Here is a short list of some of the ideas that came up:

• Mobile Payments
• Viewing mobile invoices on handsets
• Real-time translation of conversation from one language to another
• Better usage statistics captured by the network
• Faster delivery of analytics and visualization of data for better decision making
  
• etc.

Video Conferencing Demo: The session ended by Dr Andy Barbash, Head of Neurology at Holy Cross Hospital setting up a live video conference with 5 of his colleagues from across the country. The quality was choppy at times but one can see the mobile bandwidth is rising to a level where mobile video-conferencing will soon be a reality.

We also discussed the need for standards for application and data portability on smart phones.

Overall, it was a very interesting discussion.