March 2018 Monthly Update
By Daniel Compton
Welcome to the second monthly update for Clojurists Together.
Clojurists Together news
This month we had around 45 members sign up (we still need to process any signups at the end of this month), and one company member join. We’ve had a really strong response from individual contributors, but not so much from companies. If you work for a company that uses Clojure, please consider talking to your manager about supporting Clojurists Together. We recently conducted a survey of our members and will be publishing the results of the survey, and a request for proposals, for our second round of funding. More on that soon.
We continue to work on automating the sign-ups on the website, to make a faster and smoother flow for people who’d like to join.
From Lee Hinman:
- March 1 - March 15
The last couple of weeks have seen the following changes in clj-http:
- I released clj-http 3.8.0. This includes many fixes and new features that were added in the first part of the Clojurists Together work, check out the changelog for the full list of changes that went into 3.8.0, one of the bigger releases!
- A user requested a way to get the full HTTP request/response string for requests as sent on the socket level, I implemented the sending side of this by adding an option to override the socket factory, returning a socket
OutputStreamthat captures the raw data so it can be inspected later. This is behind the new
:capture-socketoption. In future work I plan to add a way to capture data read in at the socket level (think curl’s
- It’s not required for clj-http to build a new
HttpClientobject for every request, and up until a recent change there was no way to tell clj-http that you’d like to reuse a particular client. With a commit this week you can now retrieve the HttpClient used and send your own with the
:http-clientparameter. This allows users to do more in-depth customization of the client without relying on the option being added in clj-http first. Part of this also includes polishing the client creation functions (for synchronous and asynchronous clients) that clj-http uses so they can be used by anyone using the library.
- When performing an asynchronous http request, a cancellable BasicFuture is returned. However, this future did not cancel the HttpRequest object, it now aborts the HttpRequest and I’ve added documentation around how to perform request cancellation (if for instance a certain timeout has been reached).
- Aleph uses some of clj-http middleware and ran into a problem this week with transit+json responses that were coming back empty. I spent a while triaging this and determining where aleph and clj-http diverge, discovering that this is not an issue with clj-http, so that’s nice to hear.
- Finally, I’m working on setting up an environment to test NTLM credentials. A user reported that the NTLM currently implemented in clj-http wasn’t working, so I need to have a way to reproduce this issue.
That’s it for this block!
- March 15 - March 30
Here’s what I’ve been up to for the last couple of weeks:
- While Clojure has its own logging framework, the Apache HTTP client prefers to use log4j for logging. Since this can be an invaluable debugging tool I wrote documentation and example code for how to configure clj-http (or rather, a project using clj-http), see this commit for the work.
- Speaking of logging, I am hoping it can help track down an issue I spent time trying to reproduce where the synchronous connection pool is blocking forever for a user. Additionally, one I enabled the logging I notice that Apache is logging an “Illegal state ACTIVE” error when the integration tests are run. I spent quite a while tracking this one down and it ended with rewriting the way that asynchronous connection pools are handled, no more error now!
- Sometimes HTTP servers want to read a request’s headers, then do something without dealing with the body of the request, for example, reading a header with some authorization token before consuming a large binary upload body. A user was running into an issue where instead of returning a 413 like the server told it to, it throws an exception about the socket having its pipe broken due to the connection being prematurely closed. Turns out there’s an RFC about this, but it’s unevenly implemented by browsers. To make things even more complicated, apache’s http client incorrectly (throws the exception) handles it when used synchronously, but correctly returns the 413 when using asynchronously. I believe this is due to a difference in Apache’s HTTP implementation. I’m planning on revisiting this with the 5.0 client to see if the bug persists.
- Speaking of the 5.0 client, I have started a branch called apache5-upgrade for moving clj-http to be compatible with it. With compilation as the first target I have been working to get everything switched over to the new classes. Not everything is 1-to-1 and there are many breaking changes, but I have moved headers.clj, cookies.clj, multipart.clj, and a stripped-down version of conn_mgr.clj over to the new classes. Next up is core.clj and client.clj, the two biggest ones.
Here’s to more improvements in the future!
From Bruce Hauman:
-March 1 - March 15
Today I published a
figwheel-core project to the master branch of
Figwheel-Core provides all the main code for determining what to
reload and how to reload/notify the client. It is completely
independent of the REPL and server implementations.
It only has one real dependency and that is ClojureScript itself. It uses the established
communication with the client. This means it doesn’t matter what your client is.
figwheel-core really feels like a Figwheel thesis project for me, a
Rich Hickeyan distilling of Figwheel down to its essentials. It has
been a very rewarding experience.
figwheel-core work has been my focus for the first half of
March. After this I’m going to tackle a
figwheel-repl which is
planned to be completely free of reloading code and will focus
on a strong stable connection and sane multiplexing.
Much thanks to Clojurists Together and its fantastic supporters!
March 15 - March 30
Much like 2 weeks ago, I have published a
Figwheel-repl project to
the master branch of the
Figwheel-REPL is only a ClojureScript
repl-env and doesn’t do anything specific to help with automatic file reloading. As such, it is more similar to Weasel in function than to Figwheel.
It is intended to be a single
repl-env that will work on as many
platforms as possible: including Browser, Node, Worker, ReactNative,
etc. It is also intended to handle multiple clients, think browser tabs, much more gracefully than the current Figwheel REPL.
It is also different in that it only evaluates code on a single client by default. You will still be able to choose to broadcast an eval operation to all connected clients if you prefer. You can also provide a filter function when you create the Figwheel repl-env, to filter the connections to the set of connected clients you want an eval operation to be sent to.
Multiple REPL behavior
figwheel.repl namespace currently offers some ClojureScript
functions to help you list and choose which connected client to focus on.
figwheel.repl/conns macro allows you to list the connected clients:
cljs.user> (figwheel.repl/conns) Will Eval On: Darin Session Name Age URL Darin 25m /figwheel-connect Judson 152m /figwheel-connect nil
figwheel.repl/conns call lists the clients available for the REPL to target.
All connections are given easy to remember session names. The intention is that this will help you easily identify which browser tab your, through the REPL client feedback in the browsers dev-tool console.
Will Eval On: Darin indicates that the
Darin client is where
the next eval operation will be sent to because this is currently the
youngest connected client.
This youngest client heuristic for choosing which client to evaluate on, allows for a simple understanding of which REPL is the current target of eval operations. Open a new browser tab, or start an new node instance and that becomes the new eval target.
If you want to focus on a specific client:
cljs.user> (figwheel.repl/focus Judson) Focused On: Judson
From now on all evals will go to
Judson unless the connection to
Judson is lost in which case the behavior will revert to selecting
the youngest connection.
You can confirm that the repl is currently focused with:
cljs.user> (figwheel.repl/conns) Focused On: Judson Session Name Age URL Darin 28m /figwheel-connect Judson 155m /figwheel-connect nil
I think this goes a long way toward solving a problem that has existed since the very beginning of Figwheel.
Attention toward embedding the figwheel-repl endpoint
The other problem that I’m currently trying to work out is how to best support embedding the Figwheel REPL endpoint in your server.
For larger projects it simplest to use figwheel connection as a side-channel, a separate REPL connection, that is distinct from your projects HTTP server. Figwheel’s use of Web-sockets and CORS make this side connection a simple matter. But inevitably there are situations where you want to embed the Figwheel endpoint in your server. So I’m giving this some serious attention.
In addition to the Web-socket connection, I have implemented a simple HTTP polling connection which should allow anyone to embed figwheel-repl ring middleware into their stack. (Side note: I’m also looking at long polling).
It is too bad that as a community we haven’t landed on an agreed upon Ring web-socket interface, as this makes it much harder to allow simple embedding of a web-socket endpoint into the server of your choice. But I’m going to do my best to facilitate this by making it easier to create a web-socket endpoint from the provided api.
On a side note: I’m also considering making the default server a the
ring.jetty.adapter as it is such a common dependency.
Work for the last month of Clojurists Together
My work over the next month is to complete both
I’m hoping that I can also finish a
figwheel-main (a proxy for
cljs.main) to tie it all together. My intention for
that it will the most important functionality of figwheel-sidecar but
will follow the
cljs.main arg conventions. It will provide solid
file watching capabilities along with a solid figwheel-repl connection
and make it trivial to turn on auto file reloading.
I’m excited by what amounts to a completely new Figwheel, which will be a simpler Figwheel minus a tremendous amount of complexity.
In our recent survey and in previous member communications, a few people have mentioned that they’d like to have more visibility into how the projects are chosen. We had chosen not to show the voting results for the projects that didn’t get in, as we didn’t want unsuccessful applicants to feel bad if the results were released showing that their project wasn’t selected. We have a few changes planned to increase transparency without discouraging applicants:
- Show the voting results and voting rounds for all applications, but redact the names of all projects but the top 5
- List all projects that applied.
These two measures should help backers understand how we voted. If you have any thoughts on this, either as a backer or an open source contributor, please get in touch.