One of the hazards of my career as a systems researcher is that I sit through a lot of talks. Many of them are great—some are not great, but fewer than you might think—and taking notes during the talk helps me remember what I’ve heard. Actually putting those notes into some sane state helps even more, so I’ve resolved to try to post a summary and/or notes from at least some of the talks I see.
These notes are long overdue from a talk James Hamilton gave about cloud computing at Amazon. You can find the original talk (with links to video) on the UW CSE Colloquia archives.
Unfortunately the talk was so back-and-forth between topics and packed pretty densely with information that I really didn’t know how to summarize it other than a giant nest of bulleted lists. (It’s also kind of appropriate since it’s often how he summarized talks on his blog as well.)
- Massive Innovation in Storage Today (akin to 20-25 years ago)
- more relational db than ever before
- more nosql db than ever before
- distributed storage big
- all happening in the cloud
- DB/Storage world
- one-size does not fit all
- 30-year-old assumption are no longer valid
- computing becoming absurdly cheap
- as the cost goes down, the number of things it makes sense to do with computers goes up
- cloud computing is different (and real and going to change things)
- Scale!
- AWS is now bringing in the same amount of compute that ran Amazon.com in 2000, every day!
- Scale is great! means that small gains still matter when multiplied by big numbers
- Where does money go?
- Chart updated every few years because
- (1) you want to work on the right problems and
- (2) most data out there is garbage
- People aren’t here
- You *must* automate, so it’s not relevant
- People get things wrong too
- smart people doing boring things => mistakes
- smart people doing cool, hard things => good stuff
- servers are 57%, power distribution cooling 18%, space is 4%,
- look up more details
- ratio which is networking gear is going up (he’s spending half his time on networking)
- Chart updated every few years because
- Limits to Computation
- what limits apps on infinite cores?
- parallelism (we’ll ignore this)
- getting data to/from cores (which really becomes power)
- power: cost rising and will dominate
- storage & memory b/w is lagging cpu
- memory and disk walls
- memory wall
- short term: more lanes to memory (uses too much power in long-term)
- long term: chip stacking with through silicon vias (much less power, but more heat)
- used in cell phones today, looking good in lab at high performance
- will likely make the memory wall less scary than you might think
- disk wall
- density will keep scaling and grow for 10-15 years
- rotational speed looking less good => bandwidth and latency won’t improve
- >15k rpm is not economically viable, too much power at higher rpm
- both increases power to rotate, and number of chips to pull data out
- predicts death of 15k rather than rise of 20k
- amazon does not buy 15k rpm disks, where you need speed buy nand flash
- disk becomes tape
- sequential I/O is a problem, but not nearly as a bad as random I/O perf
- 3TB disk takes 8 hours to read sequentially, takes a month to read randomly
- “you cannot afford to randomly seek ever” is almost true
- where this matters, it’s all memory and nand flash
- what limits apps on infinite cores?
- Sea change in networking
- networking gear doesn’t seem to be obeying moore’s law of cost
- this is broken! why? monolithic, non-commodity => expensive, not innovative
- changing, you can now buy raw ASICs
- marvel, intel, broadcom, etc.
- can now make open source, open platform
- software defined networking
- centralized control plane, commodity software
- centralized is better if everyone is in the same administrative domain
- openflow is uninteresting except that it has real force behind it and it seems to be unifying
- client side
- virtualized NICs in h/w SR-IOV
- infiniband world to get from apps straight to h/w
- MapReduce
- reaction to RDBMs don’t scale, this does
- one of two things that normal people can write and parallelizes (SQL & MapReduce)
- NoSQL
- another reaction to RDBMs don’t scale
- key-value at scale is something we’re willing to give up some things
- aside: CAP-theorem should be taken literally, but not used as an excuse to not do things
- he argues that eventual consistency is hard to reason about and unnecessary
- we can build consistency that’s as fast
- he argues sliding-knob for consistency.
- really!? does it have more than 2 settings? –Colin (see below)
- Client Storage Migrates to Cloud
- local silicon cache with disk in the cloud
- Open Source & Cloud Influence
- dirt cheap, commoditized computing when looked at overall
- makes a lot things possible that weren’t possible before
- Summary
- cloud + scale => increasing rate of innovation
- costs down => compute up
- networking upset
- difficult problems are all related to persistent state
Q&A:
- Does the consistency know need more than 2 settings?
- There are some more than 2 settings for consistency (i.e., how to behave in partitions), but 2 covers a lot
- Is nand flash in front of disk as a cache a good idea?
- yes, but dedup and compression make more stuff fit in caches
- still need to put *ice* cold data on disk. most data is write only, perfect use for disks!
- Reusing heat for useful things
- challenge, it’s low-grade heat by heating standards
- a few useful things
- if data center is downtown, it totally makes sense to heat locally, transport is inefficient
- if out in the country in a cold area, then you can use heat to grow high-value crops for longer
- not clear
- very hard to extract power from this heat
