April 2019 Monthly Update
Clojurists Together News
Hello CT Community -
Here is the final update from our Q1 projects. We appreciate the work of Neanderthal and Aleph.
Our Project Leader, Daniel Compton, is speaking about Clojurists Together soon! Catch him at one of the following local events:
NYC Meetup: Funding OSS- Clojurists Together
May 15th in New York City, NY
6:30 - 8:30pm
Heart of Clojure
August 2nd in Leuven, Belgium
Thanks for your continued support of Clojurists Together!
- Simple Neural Network Training APIApril 3, 2019The stage has been set for wrapping up the simplest version of a complete neural network API, and its key part that offers the entry for the /learning/ functionality - the training API.
- Initializing WeightsApril 10, 2019As the iterative learning algorithm has to start somewhere, we have to decide how to initialize weights. Here we try a few techniques and weight their strengths and weaknesses until we find one that is good enough.
- Learning a RegressionApril 15, 2019A great moment has arrived. We are going to apply our neural networks implementation to a regression problem. The network is going to learn a known function, which enables us to see how well it learns, and why it doesn't do a great job. We are also going to get some hints for improvements. But, hey, it works!
- Weight DecayApril 23, 2019In this article we explore a simple but useful technique for keeping weights from growing too big. Weight Decay is useful as a regularization technique that improves generalization, and can help with improving even the basic learning on the technical level.
- MomentumApril 29, 2019Today we are goind to implement momentum, a ubiquitous learning optimization technique. What's more, we'll do it without any performance penalty. Find out how many lines of Clojure code it will take.
- GPU Accelerated Neural Networks in 6 Lines of ClojureWaiting to be published next weekIs it possible to program a fast implementation of neural networks in 6 lines of any language? What about neural networksthat run on both CPU and GPU?
Promoting Neanderthal in the non-Clojure community
During the Clojurists Together funding period, I have published 17 long articles. 6 articles got some nice publicity by reaching the frontpage of Hacker News.
From February 1st to April 29th, the articles had 57,294 views by 28,570 unique users. This data is from Google Analysics after the bots and spam were heavily filtered out, so the reported traffic seems genuine. Cloudflare shows much higher numbers, as expected.
I plan to continue writing on this topic and release a series of beautifully typeset books that use Clojure to explain the attractive and popular topics of Deep Learning, Data Analysis, GPU programming and HPC to wider programming community. This series roughly cover a third of the book "Deep Learning for Programmers" that I will open for early access in May.
General bug fixes and improvements to the Neanderthal suite of libraries
During these 3 months, I released versions 0.22.0, 0.22.1, and 0.23.0, which contain new functions useful in deep learning and general high performance programming and upgraded the underlying OpenCL engine.
- MQTT support
The biggest and the most time-consuming chunk of the effort. Right now the implementation covers MQTT3.1 client publish flows (including fire-and-forget and at-most-once use cases). Subscriptions are still in progress. Flow control is implemented on top of Manifold's stream put/take operations. It makes code easier to read/maintain, but probably not the most performant approach. Potentially we can get better performance implementing flows as Netty handlers. To make the final decision I still need to implement both and run performance tests to compare both approaches.
- Helper to extract SSL session
This information is critical to have when working with mutual TLS. TCP connection exposes SSL session through the field in
def-map-type. Exposing new field with Java-object value for HTTP might introduce problems with backward compatibility because Java objects usually play badly with print/read/eval cycles. I think helper covers the use case tho’.
- Activity logger for HTTP & TCP servers
API looks the same way it looks for clients. :log-activity param includes LoggingHandler as a top-level handler. There’s still an open question on how to log activity for each child handler (read “for each open connection”). Not to confuse users, I would probably merge both loggers to be instantiated using the same configuration key. So you don’t need to know what “handler” and “childHandler” are.
- Accurate handling of proxy exceptions
The previous implementation throws ProxyConnectionTimeoutException in case of any proxy connection error. Which is obviously misleading. The fix provided is capable of dealing with 3 different types of exceptions: connection timeout, HTTP proxy exception with input headers attached (introduced https://github.com/netty/netty/pull/8824), generic connection exception (e.g. "disconnect", "no response" etc). Note, that HTTP proxy response headers usually are the only available source of debugging information, so it’s critical to keep them available for end users.
- FileNotFoundException is finally fixed
The branch is merged with other changes from master, I think file regions support was the biggest blocker there. The response format is updated in order to hide exception details. Now server return by default generic error. Mostly due to safety reasons.
- Properly initialize per-message deflate handshaker
The problem was originally reported and discussed in https://github.com/ztellman/aleph/issues/494. In short words, WebSocket server compression didn’t work because the server was not able to communicate available extensions during the handshake. To be able to test the result of the handshake, the result was exposed through the stream description (https://github.com/ztellman/aleph/pull/498) and after some communication with Netty’s community, the initial problem was solved in a somewhat “hacky” way by firing HTTP request through the pipeline after WebSocket deflator was added. It would be great if we could bury this state propagation deeper not exposing the event to the rest of the pipeline e.g. using EmbeddedChannel, but that’s almost impossible to do because the handler itself is not marked to be Sharable, so it fails to be added to more than one pipeline. The solution works even tho’ it’s not beautiful enough.
- Update Netty to 4.1.36
“WebSocket client handshaker to support "force close" after timeout” https://github.com/netty/netty/pull/8896 was merged into Netty 4.1.35.Final. The idea behind that PR was to introduce force timeout for WebSocket client instead of waiting for the server to close the connection forever. Initially I was going to implement this functionality for Aleph but finally ended with Netty’s PR. It took quite some time to polish and merge, but as soon as Netty version is bumped in Aleph, I’m going to port force close settings here. BTW, it will work even without exposed settings as the timeout defaults to 10 seconds in Netty.