I've also been doing the fastai course, where you learn Gradio and pytorch.
Python has such a messy library story. I'm not a python developer, and coming into this ecosystem and trying to make things work with pip, conda, docker, etc. It's a mess.
I like Gradio, and built a few small apps, but it is still messy compared to Bumblebee.
Livebook + Bumblebee is magical. I'm productive in an instant, and the opportunity to build with Elixir and Phoenix makes this so exciting.
I agree on python libraries. There are probably contextual/philosophical reasons why the dependency management works the way it does, but I also don't use it a ton so its not obvious to me. RubyGems with Bundler and Node with NPM/Yarn have pretty much just worked for me. Pip and pipenv just feel clunkier somehow and there seems to be more tweaking to get things working.
I don't love the node/npm library ecosystem entirely, but it seems like at least the tools are consistent: I can generally run npm or yarn or pnpm and it will work with the same package.json file. There are often weird issues with ES6 or common JS, but generally the libraries have a fix when that is an issue.
With python, when I look at a new repository, I'm always trying to figure out, ok, does it have a requirements.txt (which means pip, right?). Or, do I need to use conda and activate it? Does anyone still use virtualenv (and why not)? And, often the requirements listed in the repository seem to only work in certain contexts. I've tried normalizing it all using a docker wrapper, but, especially for ML work, I'm then fighting with the CUDA libraries, and running against the correct GPU host libraries, etc. It feels so complicated and requires you to have been in the ecosystem for years to have the intuition on where to go next.
And, it probably makes things even more complicated that my host system is nixos: I don't think python ever considered immutable filesystems in the worldview.
I was hoping that the fastai course would smooth out some of this for me, but when I started trying to run on my own hardware, it quickly fell apart. This has always been the case for me with ML: trying to just get something running so I can experiment is such a battle that I give up.
Elixir just seems like it has a more sane approach for libraries and generally seems to work the right way. That's a good foundation for progress for me.
Hi everyone, glad to be back here with the official announcement!
It is late here but feel free to drop questions and I will answer them when I am up. Meanwhile I hope you will enjoy the content we put out: the announcements, example apps, and sample notebooks!
Congrats on the launch, this is fantastic. I love seeing these releases. They bring me so much joy & excitement.
Can you provide some thoughts on the benefits of doing ML on Elixir vs. Python? Is the benefit the language semantics? Is it much easier to get distributed work in Elixir ML vs Python ML? Are the tools better/smoother? Are there ML ops improvements? Perhaps there’s a blog post I missed :)
I have written a bit about potential benefits in the initial announcement of Nx [0] and then on its first release [1].
In a nutshell, there has been trends in Python (such as JAX and Thinc.ai) that argue functional programming can provide better abstractions and more composable building blocks for deep learning libraries. And I believe Elixir, as a functional language with Lisp-style macros, is in an excellent position to exploit that - as seen in "numerical definitions" which compile a subset of Elixir to the CPU/GPU.
I also think the Erlang VM, with its distribution and network capabilities, can provide exciting developments in the realm of federated and distributed learning. We aren't exploring those aspects yet but we are getting closer to having the foundation to do so.
Regarding ML ops, I believe one main advantage is explained in this video announcement. When deploying a ML model with Nx, you can embed the model within your applications: you don't need a 3rd-party service because we batch and route requests from multiple cores and multiple nodes withing Erlang/Elixir. This can be specially beneficial for projects like Nerves [5] and we will see how it evolves in the long term (as we _just_ announced it).
Finally, one of the benefits on starting from scratch after Python has paved the way is that we can learn from its ecosystem and provide a unified experience. You can think of Nx as Numpy+JAX+TFServing all in one place and we hope that doing so streamlines the developer experience. This also means libraries like Scholar [2] (which aims to serve a similar role as SciPy) and Meow [3] (for Genetic Algorithms) get to use the same abstractions and compile to the CPU/GPU. The latter can show an order of magnitude improvement over other currently used frameworks [4].
What will be interesting (and exciting) is seeing how GPU nodes are incorporated into platforms like fly.io, to really make this “elixir for ML/AI” thing seamless.
I'm used to having a Python middleware in my Go web application just for ML, The idea of achieving the same with just one monolith application using Elixir is very tempting.
Has anyone switched to Elixir from Go for writing web apps? What has been your experience like?
i switched from JS to Elixir. there are many benefits you can google yourself, but one thing i've realized for myself is that in regards to webapps LiveView is a real innovation. now PHP and others are copying what Elixir is doing, which is great, but it seems like Jose and others in the space are keeping Elixir on the frontline of innovation for a long time still. time and time again (like with the ML stuff) it amazes me how much Elixir is actually inventing new stuff and thinking outside of the "just a language" box. i almost quit programming honestly, because i just couldn't take the needless complexity of everyday life and Elixir saved my career. now i get shit done.
What's the motivation for the name? I get "numbat" to "numerical", but not immediately seeing any connection for bumblebee.
Maybe a more serious question: is anyone using Elixir ML in production? I'm absolutely gobsmacked at the quantity and quality of development effort that's gone into it (and use Livebook daily, though not for ML stuff). It's clearly a major focus for the team. I'm wondering if it's ready for production adoption, and if so, if anyone has used it "in anger" yet.
We use it in production at Amplified! It’s been a joy. We train our own large language models and have fine tuned and deployed using Elixir. I talked about it at ElixirConf this year (https://m.youtube.com/watch?v=Y2Nr4dNu6hI).
We were able to completely eliminate a few python services and consolidate to all Elixir for ETL and ML.
It should not be too hard! From what I saw Whisper is similar to Bart, and we have Bart. The missing piece is a library for audio processing to tensors.
If I have a Huggingface model that I've finetuned, can I load it using Bumblebee? I fine-tuned ConvNext and changed it into a multi-label classifier and saved it as a PyTorch model. It works great but being able to use it in LiveBook instead of Jupyter Notebook would be fantastic.
I think I'd have to convert the format, but what then?
No need to convert the format! Bumblebee supports loading parameters directly from PyTorch. You can just specify `{:local, path/to/checkpoint}` in `Bumblebee.load_model`
Alternatively to EXLA/Torchx, any thoughts on supporting an ML compiler frontend like Google IREE/MLIR by generating StableHLO or LinAlg? This could pave the way towards supporting multiple hardware targets (Vulkan-based, RVV-based, SME-based etc.) with minimal effort from the framework.
Yes, we have been eyeing IREE but no work has started yet. However, our compiler architecture was designed exactly to make this possible, so it will be definitely welcome if someone decides to trail ahead on this road!
I’ve been following live-view and elixir (and LiveBook as a result) and the people working on it are incredible! Inspired me to start learning elixir on the side a few weeks ago after admiring from afar.
Elixir is quirky and the functional aspects especially take getting used to (eg the learning guides say you should rarely write for loops), but the runtime and some of these amazing tools really show the potential. I don’t know if I’ll ever use it professionally (I hope so, honestly) but I think I’ll learn something that’ll make me better regardless.
LiveView has such a unique and refreshing take on web-dev (I say having never written any significant amount of HTML), that seems to make it behave much more like a local application. And LiveBook builds on that to be a next-gen alternative to Jupiter notebooks. I can totally imagine live book expanding into a more general purpose interactive playground type tool.. good for internal tools and dashboard and development experiments… almost a WebUI alternative to a REPL. I feel like all that’s missing is being able to inject/attach to an existing VM.
That's basically how my team have been increasingly using it. Simply connect Livebook to a locally running Phoenix project and you have a Livebook REPL into your server. When you're dealing with complex data, pulling from different sources and have to build up a bunch of context before you iterate on a function it's super useful to be able to break up that code into chunks, take form inputs[0] along the way and document any quirks. We keep a bunch of livebooks committed in the repo to help debug and iterate on the more complex parts of our codebase.
A couple weeks ago a client asked for some stats that came out of the blue (the data was collected, but they weren't exposed at the dashboard), his app was at fly.io, I've set up a wireguard tunnel to connect a livebook to the app and we went in a couple hours exploration of all the data that wasn't exposed at the dashboard, I took a ton of notes and the client spent a lot of time gushing about how he never had that much access to the underlying data in projects before without having to wait at least a couple weeks, to be able to work with live data almost instantaneously is a plus that I don't know if I'm able to give up at this point.
It was a new client and a short project but since then they closed a couple years contract to a new project, Livebook is indeed magic.
This is pretty amazing, are the models more intelligible in Elixir than they are in python? I was under the impression that there lots of separate pieces that are all configurable on the python side but that organisation seemed quite messy (when I looked at the Stable Diffusion code anyway). Will have a look at the elixir shortly. Running these models is cool, is there the ability to train new models too?
You can train models with Axon (the neural network library) but it is not yet fully integrated into Bumblebee. We will start exploring those topics next and hope to provide an equally seamless experience.
Jose, this is all very exciting and cool congrats to the team. I have been trying to wrap my head around this (ML is _not_ one of the areas I have a lot of background in) to see if I could figure out how to pull in something like Whisper (speech to text/translation/transcription model) and if I am getting it it looks like its just missing a little bit of hookups to the audio parsing -> tensor. if you are looking for votes to extend the core models implemented that would be my vote -- or fodder to blog about how to pull in a model that does not currently exist it looks like that one would have a lot of learnings and show the process well. Anyways this all is so cool and is amazing -- I did not realize those core parts were so built out (axon etc).
I am guessing no, otherwise that would have been showcased. I think most of the python implementations are actually C/C++ implementations, which would be hard to beat, or would be easily patched to be faster.
I linked to a podcast (but here it is again) where Sean discussed the implications of using XLA and how that reduces the translation back and forth between python and the numerical libraries underneath. It sounds like the Axon team made some choices that are paying off in the in that respect and that other teams are migrating towards.
Yes, we have an open issue to leverage that in a future release. The issue is that MPS misses many operations, so we will see how challenging it will be.
Awesome release, I might want to try this in my Phoenix project. Are there any major disadvantages to doing this vs bringing Python ecosystem into the mix, aside from obviously, missing out on tooling?
I think perhaps the bigger issue is that you will be trailing ahead a new path which means fewer resources if you get stuck. BUT you can join the #machine-learning channel of the Erlang Ecosystem Foundation or ask around in the Elixir Forum!
I have given a presentation on the fundamentals of Livebook at the LIVE Workshop @ SPLASH 2022 just this week. The video is not officially out yet but you can check the pre-recorded version [0] (Google Drive link)
As a Jupyter notebook user who has not tried Livebook (yet) i can summarize after watching:
- in Jupyter notebook state is hard to reproduce because execution can go back and forth between cells and that is not tracked. in Livebook execution is always top to bottom (still interactive and you can have branches)
- built-in package management
- multiplayer (the same notebook can be accessed and manipulated by multiple users)
Great summary! I would only add the "Smart cells" bit, which can help you automate and/or learn a variety of tasks through a high-level UI while still being able to drop back to code!
I've trained models using Jupyter and Livebook (though only smaller toy models [1]) so I can deposit my 2 cents here. Small disclaimer that I started with Jupyter, so in some sense my mental model was biased towards Jupyter.
I think the biggest difference that'll trip you up coming from Jupyter is that Livebook enforces linear execution. You can't arbitrarily run cells in any order like you can in Jupyter - if you change an earlier cell all the subsequent cells have to be run in order. The only deviation from this is branches which allow you to capture the state at a certain point and create a new flow from there on. There's a section in [1] that explains how branching works and how you can use it when training models.
The other difference is that if you do something that crashes in a cell, you'll lose the state of the entire branch and have to rerun from the beginning of the branch. Iirc if you stop a long running cell, that forces a rerun as well. That can also be painful when running training loops that run for a while, but there are some pretty neat workarounds you can do using Kino. Using those workarounds does break the reproducibility guarantees though.
Personally while building NN models I find that I prefer the Jupyter execution model because for NNs, rerunning cells can be really time-consuming. Being able to quickly change some variables and run a cell out of order helps while I'm exploring/experimenting.
Two things I love about Livebook though are 1) the file format makes version control super easy and 2) Kino allows for real interactivity in the notebook in a way that's much harder to do in Jupyter. So in Livebook you can easily create live updating charts, images etc that show training progress or have other kinds of interactivity.
If you're interested to see what my model training workflow looks like with Livebook (and I have no idea if it's the best workflow!), check out the examples below [1][2]. Overall I'd say it definitely works well, you just have to shift your mental model a bit if you're coming from Jupyter. If I were doing something where rerunning cells wasn't expensive I would probably prefer the Livebook model.
Not to distract from the awesomeness of this too much, but: what would it take to do the same type of thing but in Ruby? Being able to easily build rails apps around ML stuff would be absolutely awesome for us boring rails guys vs building python APIs for rails apps to talk to…
The amount of work varies because there are many different design decisions you can make.
We have been working on this (Nx + Axon + Livebook + Bumblebee) for almost 2 years. One person full-time (part-time for the first year) and 3 part-time. But we have taken a foundational approach: instead of providing bindings to high-level libraries, we choose to build the whole foundation in Elixir and only leverage 3rd-party libraries at the compiler level. It looks like this:
Everything is done in Elixir, except for the (pluggable) Compilers layer, which can be written in anything. The ones we use by default are from Google/Facebook and are written in C++.
This design gives us agency and resiliency: we have control over how to evolve the whole stack and we can swap between different compilers depending how state of the art evolves. We are more expressive too, as we are not limited by design decisions done in libraries written in a separate language.
But you can skip ahead provide direct bindings to libraries such as torch + torch.nn. This will cut corners but also will officially tie you to a Python library.
However it is important to notice that embedding a neural network model inside your Phoenix app, as done in the video, is only really practical in a language that can fully leverage concurrency and effectively run both CPU/IO workflows in the same OS process. I am not following the progress of concurrent Ruby but my rough understanding is that it is not there. So in practice you would need to deploy your machine learning models to a separate service anyway.
It builds on top of Nx, Numerical Elixir, which allows for models to be compiled for cpu or gpu. Done this way, the model does not have to run by executing BEAM bytecode. Additionally, Liveview and Livebook enables the equivalent of Jupyter notebooks. All of this has been happening for the past couple years.
The advantage here is data being fed into the model can use frameworks like Broadway (streaming ingestion) or Membrane (realtime media streaming), or Nerves (BEAM at edge / IoT), or as a controller for unreliable agents.
Digging into the source of its dependencies a bit, it looks like it uses NIFs (inlined functions) to bind into your choice of a couple of different potential C backends, the big ones being Torch and Google's XLA compiler.
So performance shouldn't be an issue at all, when compared to similar "high-level language calling low-level language for heavy tasks" situations, like Tensorflow or PyTorch. It does, however, come at the cost of sacrificing the extreme stability BEAM applications typically enjoy: misbehavior by the backend can cause hangs or even crashes of the BEAM node they're running in.
Yup. And it can be mitigated somewhat by isolating workloads that might crash BEAM, while using dist erl so nodes can still communicate with each other.
Rust is an exciting option for NIF, but those backends are not written in it.
I’m not expert here but I think most of that risk is mitigated these days since BEAM introduced Dirty Schedulers/Dirty NIFs for any NIFs that take over 1ms to execute… I think this isolates (or can isolate) their issues to only effecting other work on the same Dirty scheduler, aka other Dirty NIFs…
Dirty schedulers isolate the potential scheduling issues, but there's no memory isolation between NIFs and the rest of the BEAM state, so there's danger there.
Best practice for doing this with the BEAM (the Erlang VM) is to just run two VMs, one for your app code and a “dirty” one that is allowed to crash and be restarted without bringing down your app VM.
If you're considering running a second (or multiple) BEAM for your nifs, you should also consider running your native code as a c-node or a port program.
A compelling use case is where your website and numerical engine are running on different machines. In that circumstance, your main application is already isolated from issues on the numerical engine.
This is really amazing, and will make wiring up some of these models together much easier I think. Elixir is making great strides in being a fantastic choice for ML
The ergonomics of Bumblebee are so perfect.
I've also been doing the fastai course, where you learn Gradio and pytorch.
Python has such a messy library story. I'm not a python developer, and coming into this ecosystem and trying to make things work with pip, conda, docker, etc. It's a mess.
I like Gradio, and built a few small apps, but it is still messy compared to Bumblebee.
Livebook + Bumblebee is magical. I'm productive in an instant, and the opportunity to build with Elixir and Phoenix makes this so exciting.
I'm blown away.