This talk will examine how your programming language’s ability to provide easy concurrency can have an impact on your system architecture and AWS bill, with a focus on Python and Go! It will take an example application — an event-processing service requiring storage for ~700/sec PUTs of data, each on average ~70KB — and walk through a few approaches to meeting this demand cost-effectively using S3, SQS, Kinesis, DynamoDB, and Hadoop. It will also provide an introduction to Go as the language used by this service, and show how its concurrency model makes it easier to achieve a lower-cost implementation than would be possible with Python, Ruby, or Node.js.
Cloud platforms like AWS offer a diverse range of services that provide almost limitless access to storage, computation, databases, queues, and logs. Services like AWS's S3, SQS, Kinesis, and DynamoDB provide these vastly-scalable systems behind the relatively high latency of an HTTP-based endpoint — meaning that high throughput can be achieved, but only by using high amounts of concurrent API requests.
Scripting languages like Python, Ruby offer a fun hacking experience often without the extra semicolons and boilerplate required by "less fun" languages. However programs written in Python, Ruby, and Node.js all suffer from the same basic limitation — they run in a single-threaded interpreter that cannot perform concurrent computation. To make use of multiple cores, a programmer typically must run many processes in separate interpreters, and use IPC to communicate between them; other options involve making using event-based environments for using asynchronous I/O like Python's Tornado, Gevent, and Twisted, Ruby's EventMachine and Celluloid, or Node.js.
However Go, a relatively new systems programming language by Ken Thompson, Rob Pike, and Robert Griesemer, seems to offer the best of both worlds: fun hacking without the semicolons, but also easy access to concurrency and a compiler offering type-checking and runtime performance closer to that of C++ and Java.
Inspired by discussions at the end of the last Python meetup, Chris Erway, chief architect at AppNeta, learned Go over the past couple of months and implemented several experimental services using the language that consume Kinesis and SQS messages and send output to S3, DynamoDB, AMQP, and Kinesis. He will present a brief introduction to Go’s language and features, with an eye towards similarities and differences with Python. While Go is not the best option for every application, he will highlight an example of how easy access to concurrency (using the “go” keyword) can lift many design constraints preventing programs written in scripting languages from being as fast and inexpensive as possible.
Free pizza and beer at 6:30, talk starts at 7pm. We will be at the Founders League / Betaspring space, on the third floor.
Hello all, there haven't been regular Python meetups since the "Royal Python Society" back in 2012, but Providence's Pythonistas are working to bring them back in 2015!
Our first Python meetup of 2015 will feature a talk by David Stuebe, titled:
Generators: x = yield 42
And other applications
David will talk about using generators in Python, and where using generators can be faster and more flexible for data processing.
Doors at 6pm, talk begins at 7pm. Pizza and beer will be catered courtesy of AppNeta. Hope to see you all there!