September 2022 Monthly Update

By Alyssa Parado

Project: Maria.cloud, Matt Huebert

We’re excited to announce that I (Matt) have received a three-month grant from ClojuristsTogether to work on Maria.cloud. This is the first of three monthly updates.

What is Maria?

Maria combines carefully written curriculum with a clean, no-install editor. It is designed to introduce complete beginners to fundamental ideas of programming, with an editing environment that encourages REPL-driven development and structural editing from day one. It has been widely appreciated as a good tool for introducing Clojure to beginners and used at several workshops like ClojureBridge.

What did we propose?

We built Maria 5 years ago and would like to bring it back into active development, starting with a refactor/simplification of the core of the editor so that we have a good base to build on top of.

Top priorities would be (1) replace the selfhost compiler with sci for a more lightweight runtime, (2) replace the code editor with CodeMirror 6 using the cm6 clojure mode I wrote last year for Nextjournal, (3) upgrade to latest version of ProseMirror, (4) add a “publish” feature so that people can share what they make with the world. This funding may not cover all of the work listed above, so we’ll remain open to receiving funds from elsewhere. This would be the first funding we have ever taken for this project.

What are some benefits of these changes?

What has been achieved so far?

Progress is automatically deployed to https://2.maria.cloud on every push to main. So far I’ve implemented:

The editor itself is a nuanced, fidgety thing which requires a lot of careful attention to get right.

Next steps

Ancillary tools (aka the scenic route)

In the course of this work I’ve also spent time on a couple of support tools.

In js-interop I’m working on a j/js macro, which is like a “deep” version of j/lit, meaning that literals become JavaScript data structures ({} => object, [] => array, keywords => strings), and destructuring forms in let/fn/defn are treated as js by default. Literals identified as belonging to Clojure proper or tagged ^:clj are not touched. This was inspired by @borkdude’s experiments in new cljs compilers. j/js can make interop-heavy code easier to read and write but is not without tradeoffs; one needs to be extra-aware of whether one is looking at code in a “js” or “clj” context. It was particularly helpful in writing code related to ProseMirror/CodeMirror. I’m quite sure I want something like this to exist but the API/behaviour remains in flux. See the PR.

I’ve resumed work on yawn, a hiccup compiler/interpreter that targets React. I was planning to stick with Reagent but it lacks a protocol that would enable custom rendering of arbitrary types, which we need for our viewers (eg. to render shapes properly). Yawn is designed with performance in mind and processes hiccup forms at compile-time where possible. It is REPL-friendly via support for react-refresh, so re-evaluating a view will immediately update on-screen while preserving state & without re-rendering from root.

The end

Thanks again to Clojurists Together & all its supporters for making this work possible!

Project: Tablecloth, Ethan Miller

This is the update for my “Column for Tablecloth” project, which has been generously funded by Clojurists Together. The goal of this project – explained in more detail in my first update – is to give the column a presence within the data processing library Tablecloth. In other words, by the end of the project, users of Tablecloth should be able to interact exclusively with a column using functions that generally take a column and return a column: (column argx argy) => column.

Over the past three months, I developed some conversation about the project within the online Clojure and Scicloj communities and on a piece of unexpected work: the datatype API for Tablecloth columns. These two efforts haven’t necessarily overlapped, but they’ve certainly been mutually reinforcing. I’ll start by discussing the work I’ve done on datatypes.

Checking Datatypes on the API

The work on datatypes came out of my first PR for this project, which established the new tablecloth.column.api namespace and added a few key functions to get things rolling. One of the functions I added was typeof – inspired by R’s function of the same name – that returns the data type of the elements in the column. E.g.:

(typeof (column [1 2 3]))
;; => :int64

It turned out that the story about types in Tablecloth, unbeknownst to me, was a bit more complicated. Unlike tech.ml.dataset, Tablecloth has a type hierarchy (defined in tablecloth.api.utils). So how should typeof work with this hierarchy? Should typeof return the child or “concrete” types? (e.g. :int64) or the “general” type (e.g. :integer)?

After some discussion and experimentation we settled on the following:

So:

(def mycol (column [1 2 3]))

(typeof mycol)
;; => :int64

(typeof? mycol :int64)
;; => true

(typeof? mycol :boolean)
;; => false

(typeof? mycol :integer)
;; => true

(typeof? mycol :logical)
;; => false

That the focus is on the concrete type with an ability to check the general type follows the tendency visible in both Python’s Numpy and in R. In both, checking the datatype will yield the concrete type. Both also have type hierarchies. In Numpy, there’s a somewhat clunky function issubdtype that can be used to check the parent-child relationship; in R, you can ask: is.numerical or is.logical, etc.

We accomplish something similar to R with just the typeof and typeof? because we can ask about the concrete type with (typeof? col :int64) and the general type with (typeof? col :numerical). So for now this is the type syntax we settled on.

The most interesting design decision was whether or not to return the concrete type or the general type when querying the type of the column. It is true that this is the behavior of at least two of the main data processing libraries. So we are in good company. In conversation, Jon Antony (author of the visualization library Hanami, among other things) gave this further reason for choosing the concrete type:

FWIW, count me in for returning the concrete type as well. At root this stuff is (or should be) about performance and when I’m checking types in this context I always want the concrete type. Some extra ‘higher level’ stuff might be nice, but likely won’t be used much.^1^{#fnref1 .footnote-ref}

I like this reasoning because it is based in a sense of what is useful in practice. Another reason that occurred to me is that, if we ask for the type and get back the general type, we are actually throwing away information. The column elements' types have a concrete type that provides more information about what they are and how they are stored in memory. Why throw that away by default?

If the user wants to know the general type of the elements columns, we will let them ask about that like so:

(->general-types (typeof (column [1 2 3])))
;; => #{:integer :numerical}

That is where we’ve landed so far on the datatype API. Please if you have any comments or questions, reach out and let me know what you think.

Project with Bozhidar Batsov

Here are the highlights from my Clojure work for the past couple of months:

As usual there are also many things that are in the works and are yet to be released.

Project with Michiel Borkent

ClavaScript

This is a new project: a CLJS syntax to JS compiler for the niche use case where you want to write JS, but do it using CLJS syntax and tooling instead. ClavaScript comes with a standard library that resembles CLJS but is built on bare JS ingredients. As such, Clava comes with the usual JS caveats, but we can still have our parens and enjoy a slim bundle size!

Cherry

Cherry is similar to ClavaScript, but it does emit CLJS-compatible code (with the persistent data structures, etc). The compiler code is almost identical to Clava's, but with a few tweaks here and there. E.g. {:a 1} in Clava means: a JS object with a "a" key and 1 value, but in cherry, {:a 1} means the same as in CLJS. The goal of both Clava and Cherry are to reduce friction between CLJS and JS tooling. Both projects should be considered experimental for now. Challenges in both Clava and Cherry is the REPL, since both projects compile to ES6 modules and ES6 module imports are immutable.

On ClojureDays 2022 I will give a talk titled "ClojureScript reimagined" which will shed more light on both projects.

Scittle

Execute Clojure(Script) directly from browser script tags via SCI. See it in action.

Scittle received two new plugins: one for promesa.core and one for cljs.pprint. Also error messages were improved.

Babashka toolbox

Babashka toolbox is a port of Clojure toolbox and gives an overview of bb-compatible libraries and projects.

Babashka CLI

Turn Clojure functions into CLIs!

Several new options have been added: :validate, :require, :restrict. Also error handling was made more flexible.

Babashka CLI proper is now part of babashka. Also see my blog posts about it:

Babashka

Native, fast starting Clojure interpreter for scripting.

Nbb

Scripting in Clojure on Node.js using SCI

Clj-kondo

Static analyzer and linter for Clojure code that sparks joy

Bebo

Run Clojure scripts on Deno via SCI. I'm not exactly sure how useful this is to the wider Clojure community, but I got curious about deno and decided to give this a go.

Quickblog

Light-weight static blog engine for Clojure and babashka

A lot has been happening in this project, with the help of Josh Glover. Check out the changelog. The blog you're currently reading is made with quickblog.

SCI

Configurable Clojure interpreter suitable for scripting and Clojure DSLs.

This is the workhorse that powers babashka, nbb, bebo, and many other projects.

Several bugfixes and enhancements were made in the last two months.

Neil

A CLI to add common aliases and features to deps.edn-based projects.

Neil now has a new subcommand which defers to deps-new. Also neil test was added to run tests using the Cognitect-labs test runner. Much thanks to rads who contributed a lot.

Process

Clojure library for shelling out / spawning subprocesses

Minor updates and fixes.

Fs

File system utility library for Clojure.

Minor updates and fixes.

Pod-babashka-buddy

A babashka pod around buddy core (Cryptographic Api for Clojure).

The latest release adds wrappers for buddy.sign.jwt and provides an aarch64 binary.

See changelogs.

Dynaload

The dynaload logic from clojure.spec.alpha as a library

This library was made compatible with nbb.

Deps.clj

Upgrades and minor fixes.

Sci.configs

A collection of ready to be used SCI configs

Moved cljs.pprint config from nbb into this project.

Project with Dragan Djuric

For this period, I have a pretty good stuff to show: I’ve produced a fully working RNN implementation on CPU and GPU! It was somewhat harder than I had hoped, but I’ve put some extra work to it and I can finally say that Deep Diamond supports recurrent neural networks. And even better, for the user, everything is automatic: the user only needs to put something like (rnn [128]) at the desired place in the network description! I’ve also written a tutorial on RNN that showcases a Hello World example from start to finish.

I won’t dwell too much on the details other than notice that this big milestone for Deep Diamond coincides perfectly with the end of this CT funding round.

Besides that, I continued my long-term efforts to learn about music and sound and develop Clojure Sound. In this period, I mainly used already released Clojure Sound to make a Clojure program for ear training. I’ve started a series of blog posts that explain every detail of its development, and showcases Clojure Sound’s features for dealing with MIDI hardware.

Software produced in this period:

Blog posts published (and several other sound related in the pipeline):

Project with Thomas Heller

Time was mostly spent on doing maintenance work and some bugfixes. As well as helping people out via the typical channels (eg. Clojurians Slack).

Current shadow-cljs version: 2.20.1 Changelog

Notable Updates

Project with Nikita Prokopov

Final two months of Clojurists Together long-term funding. What a year!

We started in September last year intending to build Clojure-native desktop UI framework, but nothing but ideas at our hands. After several Zoom interviews, discussions, blogging, and A LOT of coding, one year after, we have a working prototype.

It’s far from complete, unfortunately (which is expected, given how enormous the task is), but it’s real, you can touch it, you can play with it, you can build simple stuff in it, and it already feels like magic.

Early adopters report that overall developer experience is way superior to anything web or other frameworks can offer. And it’s in Clojure! With REPL!

So, what exactly happened in the last two months, and where are we now?

HumbleUI:

JWM:

Skija:

DataScript:

I also created a new Sublime plugin, which may or may not be useful for Clojure development (I use it every day now and enjoy it a lot): https://github.com/tonsky/Sublime-Executor

Overall, working with Clojurists Together has been a fantastic experience and I’m really excited about what we were able to do together.

Work on Humble UI is not stopping, though. It is now full steam ahead and aiming at its first debut at Dutch Clojure Days this Autumn. Wish me luck!