How can we build more intuitive, responsive and playful interfaces for performing realtime graphics?

I have been thinking about how there are no IDEs for creative coding that doesn't assume the user is a beginner programmer, or doesn't want to code. Think of Touch Designer, a node based programming environment for creative production, it is a great way to organize algorithms in a visual way, but it doesn't allow the full range of motion coding allows. Processing and P5.js are famous for getting creative people into coding, through its accessible interface and reframing "compiling" into "playing", but what happens after compiling no longer is intimidating?

The creative coder moves to IDEs that are made for software engineering, for functionality, and for enterprise. This is the vast majority of IDEs.. the VAST majority.

 I am not expecting to create the next big creative coding interface, but to prod in this area by creating a series of light hearted sketches called "Fantasy IDEs" imagining the IDE as a playful interface for creative coders. 

My investigation stems from my experience as a live coding performer, specifically in writing GLSL code in front of audiences. This background provides a unique perspective on the intersection of programming as both a technical and performative practice, informing my approach to reimagining the IDE as a creative medium.

I created an IDE that has more narrative - a beginning, a middle and an end, with unfolding features, as the performance would go on. I created a windowing system that would spawn small windows, each containing a different shader. This was so that I could start with a blank screen, and slowly one by one populate the screen with windows. I would periodically make the small windows full-screen to write and modify shader code live. I also made the windows bounce around and bump into each other, for visual eye candy, and to create movement. Some of the windows included webcam feed as well.

As the performance progresses, the audience is invited to engage with a chatbot, called Charbot, introduced via a QR code they can scan on their phones. Charbot serves as both a playful critique to the current trend of embedding AI-based agents within software development tools. Charbot can answer questions about the live code in plain English. This is done through sending the current GLSL code on the screen to OpenAI API querying it to explain the code and send it back to the audience in the group chat. This dynamic, collective interaction is designed to deepen the engagement and understanding between the audience, the performer, and the technology that underpins it.

Another IDE I had developed was an ”IDE that always compiles”, a cheeky name for a dysfunctional IDE. While writing the code in this IDE, if the code has an error uncorrected for more than one second it gets sent to OpenAI using API asking to correct it, and the corrected code replaces the code I was writing. This turned into an adversarial experience of trying to code as fast as I could while it would take my ideas in a different but bland and more average direction. Every coding session would end in an ML laced hell feedback loop of completely made up incorrect code. 

Here are some sketches I made using this system:

Finally I was thinking about how source control is very different for creative programmers versus software engineers. For the vast majority of coding, the code progressively becomes more optimal, better for enterprise. The creative process on the other hand is more branching. Its softer. I took the idea from the Khronos Projector by Alvaro Cassinelli (you can see that on the left of this image) where you can scrub a video through time and applied it to source control.

So if you take the soft pressure from a mouse:

 You can push the code back in time to different versions in a soft way

I hope to create more Fantasy IDEs to express how a coding environment to be more expansive and dreamy over being productive or efficient!