How Some Things are Done in Facebook: Operations at Facebook

via slashdot: an amazing talk on several aspects of operations at fb. include infrastructure, configuration management, code deployment, monitoring etc.

A few things I took from the talk:

• over 300 TB of data live in RAM served out of memcached
• 
  HipHop for code transformation from PHP to C++
  CFengine for configuration management
  
• 
  BitTorrent for code deployment {several 100s megabytes of code deployed to 10s of 1000s of machines to facebook.com }
  
• 
  Ganglia for monitoring
  
• 
  On-Demand tools: used for things like on-demand fix a mistake that escaped engineers in testing/deployments stages. Facebook used to use DSH but they outgrew it so FB built their own tool. Don't know if they made open source yet, the lecturer said he could not talk too much about it. ????? it allows making changes very quickly.
  

Facebook pushes code at least once a day and weekly pushes out new features not necessarily turning them on.

Normal software development structure Software engineers -> QAs -> operations. Facebook gets rid of the middle layer and delegate its tasks to first layer.

Facebook is loose on change management but very heavy on logging

very interesting talk. link here

Also notice youtube new feature. still in beta but amazing { transcribe audio and translate sub-titles.}



very cool.