I am the original founder of the HoloViz.org project, which provides open-source Python libraries for scientific, engineering, and analytical data exploration and visualization, PyViz.org, which catalogs all the open visualization tools available for Python, and Pandata.pydata.org, which provides a curated set of highly scalable data processing tools.
I have been a vocal proponent of open source since grad school in the 1990s, where I embraced the new (and still quite rough!) Linux operating system and GCC compiler. My goal was to ensure that my research software would be reproducible by anyone, without any restrictions. Approximately fifteen people ever tried to use any of that esoteric brain simulation code, but I kept focusing on open science when those around me were locked into proprietary tools that I felt were not worth that loss of freedom.
Once I became a computer-science professor in 2004, I embraced Python and taught courses arguing that open source and flexible “scripting languages” like Python were the way we could finally achieve the still-elusive goal of software reuse (instead of starting every new system from scratch!). However, I felt like a curmudgeon shaking his fist at the sky, with those fanciful ideals seemingly impossibly far away from reality.
Two decades later, nearly every cloud computer runs Linux, and Python is now the most popular computing language in the world. Instead of starting from scratch, it is now the norm to stitch a large collection of libraries into a complete Python or Javascript system, requiring very little new code. Open source won, and software reuse is real! I feel privileged to have seen such a profound shift in how software is written and used. Today’s scientists and engineers no longer have to choose between a proprietary platform with hard-coded functionality, or else writing entire systems from scratch in a language like C (like I had to do as a young researcher).
My own group’s open-source tools started out modest, with Param.HoloViz.org starting in 2003 to help make our brain simulator have modular, well-defined software interfaces, and HoloViews.org joining it in 2010 to make it easier to use Matplotlib for complex multi-dimensional data. Our tools were easy to distribute since they were pure Python, but they needed to be combined with C or Fortran libraries like NumPy to do any real work, which limited their use to old coders like myself who could compile those languages. Anaconda.com started to change that in 2012, providing pre-compiled libraries that allowed numerical computing in Python to really take off.
By 2015 I had left academia along with three of my former Ph.D. students and we all joined Anaconda’s consulting group, which let us focus on our open-source tools nearly full time, instead of working on them when they were meant to be writing papers and grant proposals. For each customer, we now walk in with deep expertise on Python’s open-source tools, offering to quickly put together solutions reusing existing libraries to save them time and money. We then pitch the customer on new open-source capabilities that we propose to add to the open tooling so that they can use it without needing to maintain the tools, and so that they can later take advantage of advances funded by other customers. In client project after client project, we have steadily expanded our tools to include Panel.HoloViz.org for building web apps in Python, Datashader.org for rendering enormous datasets, Colorcet.HoloViz.org for perceptually accurate colormaps, hvPlot.HoloViz.org for quickly exploring data, GeoViews.org for working with Earth-centered data, and now Lumen.HoloViz.org for exploring data using natural human language thanks to new Generative Artificial Intelligence (GenAI) large language models (LLMs).
Somewhere in there we introduced the name HoloViz to group these libraries together and show that they are all maintained as a group, and created HoloViz.org and Examples.HoloViz.org to show how to use them all together. In the meantime, the Python software ecosystem had become so diverse and complex that we needed to create PyViz.org as an unopinionated overview of what became hundreds of visualization libraries. The dashboarding and web applications section of PyViz.org went from four tools at the beginning of PyViz to now dozens of competing libraries. In my career, we have gone from having no software available to reuse, to having software but having to struggle to build and connect it together, to today’s problems of having to choose between so many confusing options.
Along the way, our tools ended up having a real impact, far beyond our original academic community. Param was used only in our brain simulation framework for its first ten years of life, but it is now downloaded from PyPI and anaconda.org over a million times each month. All told, the HoloViz tools are downloaded 10 million times a month, compared to maybe a single thousand people who have ever read my academic books or papers. Open source has had a much more profound impact than just about anything else I have done in my professional career. What a journey it has been!
Thanks to Philipp Rudiger and Jean-Luc Stevens (the original authors of Panel and HoloViews), Jefferson Provost and Chris Ball (the original authors of Param), and all the many other members of the HoloViz team past and present.
Read stories shared by other maintainers.
This story was published under CC BY-SA by the author.