tag:blogger.com,1999:blog-11788780.post2537509446612004035..comments2023-12-29T13:22:33.104-08:00Comments on JJinuxLand: Async: To Be or Not To Bejjinuxhttp://www.blogger.com/profile/03270879497119114175noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-11788780.post-70674554639221571552013-01-19T13:04:35.199-08:002013-01-19T13:04:35.199-08:00Thanks, Gus. Erlang wasn't as popular back the...Thanks, Gus. Erlang wasn't as popular back then as it is now. My guess is that none of the early IronPort people even knew about it. In contrast, Sam Rushing already knew how to solve the async problem in Python.<br /><br />Python will never be as good as Erlang at what Erlang does. Hence, for certain network servers, it makes a lot of sense to use Erlang. However, Python has so many other advantages that it probably makes sense to use Python (and gevent) as the "main" language for a company.jjinuxhttps://www.blogger.com/profile/03270879497119114175noreply@blogger.comtag:blogger.com,1999:blog-11788780.post-13036314139686912222013-01-14T15:11:48.093-08:002013-01-14T15:11:48.093-08:00Since you mentioned it, I'm curious why didn&#...Since you mentioned it, I'm curious why didn't Ironport use Erlang instead of developing a new concurrency framework for a slow interpreted language like Python?<br /><br />If you were starting today (2013), do you think Erlang is the right tool for network appliances like Iron port's?<br /><br />BTW, great blog!<br /><br />Cheers,<br /><br />Gusgusnoreply@blogger.comtag:blogger.com,1999:blog-11788780.post-20479217332674689372012-05-28T11:36:50.485-07:002012-05-28T11:36:50.485-07:00verte, thank you for the excellent comment! In ge...verte, thank you for the excellent comment! In general, I agree with you.<br /><br />gevent is non-deterministic, but it's not as bad as threads which can context switch at any time. Since it can only context switch when doing IO, the problem isn't nearly as heinous. Sure, that's not perfect, but it's a lot easier for me to wrap my brain around than threads.<br /><br />As for multi-threading in Swing, I wrote some quick tricks here (http://jjinux.blogspot.com/2007/12/python-some-concurrency-tricks.html). Basically, I avoid mutable, shared state like the plague.jjinuxhttps://www.blogger.com/profile/03270879497119114175noreply@blogger.comtag:blogger.com,1999:blog-11788780.post-47278916589135756952012-05-28T00:21:57.763-07:002012-05-28T00:21:57.763-07:00The distributed computing example is a summary, so...The distributed computing example is a summary, so I won't try to summarise it here. The introduction is about a page.<br /><br />"The problem with threads" deals with the explosive nondeterminism resulting from shared-state concurrency. To be clear, I would also consider gevent to be shared state concurrency*, in that you can't look at a function and be able to tell from its body if it contains a context switch or not - that could be hidden away inside some function that we call.<br /><br />This is a significant burden on the programmer. As someone who maintains a moderately-sized swing application rife with concurrency bugs, I think I can say, until programmers are forced to think about concurrency from the outset, maintenance is a battle between introducing new code and trying to figure out the way it interacts with existing locks and tasks.<br /><br />But don't take my word for it: the article mentions a concrete example of an application written by concurrency experts that mysteriously deadlocked once they bought a machine with more cores.<br /><br />In general, I'm all for runtimes and compilers that figure out details so you don't have to. I like dynamic types, I like garbage collection. But concurrency is a more complicated subject, I think, and it deserves very explicit language from the programmer. (Concurrent Haskell is an interesting example - the language is functional, the concurrency features serve only to give greater control over communication to the programmer).<br /><br />* important side note: finalisers actually introduce shared state concurrency in many languages, python included. See eg. <a href="http://wingolog.org/archives/2012/02/16/unexpected-concurrency" rel="nofollow">unexpected concurrency</a>wlesliehttps://www.blogger.com/profile/12093160811053013157noreply@blogger.comtag:blogger.com,1999:blog-11788780.post-8493822376592222372012-05-24T14:03:31.092-07:002012-05-24T14:03:31.092-07:00Peter, C#'s await keyword reminds me of EventM...Peter, C#'s await keyword reminds me of EventMachine in Ruby. It looks like it's a way to use blocks as callbacks. Is that right?<br /><br />I do think having blocks makes callback-oriented programming easier to read, but it's still not as easy to read as, say, gevent's approach.jjinuxhttps://www.blogger.com/profile/03270879497119114175noreply@blogger.comtag:blogger.com,1999:blog-11788780.post-36917250007486854962012-05-24T14:00:38.043-07:002012-05-24T14:00:38.043-07:00verte, sorry I haven't read anything more than...verte, sorry I haven't read anything more than the abstract for "the problem with threads", and I haven't read "the distributed computing example in E." It's going to take me a while to get to those. In the meantime, do you care to summarize?<br /><br />I actually wasn't arguing for threads. I don't actually like threads. Erlang has something it calls processes, which I think is ideal. gevent has greenlets which are actually a lot more deterministic than threads.jjinuxhttps://www.blogger.com/profile/03270879497119114175noreply@blogger.comtag:blogger.com,1999:blog-11788780.post-78750864162553721402012-05-24T00:23:09.832-07:002012-05-24T00:23:09.832-07:00I don't have enough experience with C# 5's...I don't have enough experience with <a href="http://msdn.microsoft.com/en-us/library/hh191443%28v=vs.110%29.aspx" rel="nofollow">C# 5's await</a> keyword to judege it properly, but it sure makes for an easier read. I wonder if something like that would be possible with some helper function in python... E.g.: with await(asyncMethodInvoication): process result<br /><br />And the point of async not always being the right solution is certainly a valid one!Peter Zsoldoshttp://zsoldosp.blogspot.comnoreply@blogger.comtag:blogger.com,1999:blog-11788780.post-16509707027711976342012-05-24T00:21:14.179-07:002012-05-24T00:21:14.179-07:00Event-driven stuff doesn't scale. 8^)
You mig...Event-driven stuff doesn't scale. 8^)<br /><br />You might be able to write an event-driven HTTP server. But once you try to combine it with an event-driven DNS resolver and an event-driven database client, the state space has exploded.<br /><br />You might then write a set of tools - help from the compiler or runtime - to essentially reinvent a cooperative threading package. And that works great. But at some point the difference between your threading package and the next one comes down to semantics.<br /><br />The true Holy Grail of [server] scalability would be to route around the barrier presented by the operating system [which is itself a coroutine system] and have an OS with no kernel/user wall and without artificial limits on scalability. Depending on your politics, you could call it an "in-kernel server" or a "user-space tcp stack".Sam Rushinghttps://www.blogger.com/profile/13115847299260965994noreply@blogger.comtag:blogger.com,1999:blog-11788780.post-71109249482680800082012-05-23T20:19:59.548-07:002012-05-23T20:19:59.548-07:00I know that a lot of arguments about async these d...I know that a lot of arguments about async these days are around performance, but the primary motivation for it actually is conceptual. I'm sure you've read <a href="http://ptolemy.eecs.berkeley.edu/publications/papers/06/problemwithThreads/" rel="nofollow">the problem with threads</a>, since it gets thrown around on #python quite a bit, but a better one-page explanation of the conceptual simplicity of async is the <a href="http://wiki.erights.org/wiki/Walnut/Complete#Distributed_Computing" rel="nofollow">distributed computing example</a> in E.<br /><br />I imagine the sync/async divide is similar to the functional/stateful divide, where there are implementation details that drive performance issues, but the more interesting aspect is a matter of how we think about problems, and what problems become significantly simpler to understand when posed asynchronously. Something you may have considered is what it would take to design a sensible memory model for python or javascript in a threaded vs an async world (if you imagine they are mutually exclusive).wlesliehttps://www.blogger.com/profile/12093160811053013157noreply@blogger.com