Creative Coding: Weather, Vision, and Sketching

Here is code, info, etc for "Creative Coding: Weather, Vision, and Sketching" module of Web Design at Gallaudet University, as taught by Nina Lutz and Prof Max Kazemzadeh, Spring 2021. With assistance from Skylar Kolisko.

Info

I've been doing creative and/or visual coding + math stuffs since 2015 when I started working with Ira Winder in City Science at the MIT Media Lab. I have primarily done Processing and p5.js.

I post (almost) daily sketches made wtih p5.js on my Twitter.

This is some documentation for 2 sessions at Gallaudet that I am teaching as part of Prof Max Kazemzadeh's Web Design course.

You can view the Github here and you can email me at nlutz@mit.edu if you have more questions! There are instructions to download all of the code here too.

Closed caption videos, glossaries, slides and more are available here

We will be hosting office hours on 3/24 and 3/31 for these sessions or for any Gallaudet students that want to come by and ask questions about resaerch in HCI, creative coding, or careers in this space. Info is below! ASL interpretation will be provided!

3/24 12pm-2pm Zoom ID: 978 3958 3579

3/31 6pm-8pm Zoom ID: 945 4745 7131

Session 1: Sketching and Weather

Related reading for this session: A Year (2020) of (almost) Daily (code) Sketches

Closed caption videos, glossaries, slides and more are available here

This session will cover building weather web apps by retrieving data via an API call into p5.js sketches and visualizing data using object oriented programming and computer graphics. Through this, we will also go more in the process of sketching and iteration of design.

An example of iterative sketches

All of these sketches were created from the following code with less than 5 lines of change between them. Feel free to copy/paste this code below and play with it in an online editor if you want.

//Set up 2 global vars
var r=200; //radius
var s=0.002; //speed

function setup() {
   createCanvas(600, 600);
   background(255)
}
function draw() {
   background(255, 150);
   translate(width/2, height/2);
   let n=360;
   for (let i=0; i<n; i++) {
      let t1= i*2*PI/n;
      let t2= s*t1;
      let x1= r*cos(t1);
      let y1= r*sin(t1);
      let x2= r*cos(t2);
      let y2= r*sin(t2);
      stroke(i%255, 50)
      strokeWeight(0.2)
      noFill();
      line(x1, y1, x2,y2);
   }
   s+=0.002;
   fill(0);
}

Some example weather apps

All of these were made from the same weather API, which makes a weather object in our code. This is done by making a GET request to the server and creating a weather object in our code, illustrated in the diagrams below.

The example video of a weather app with this API is here and the functions that the API supports is here in this PDF.

Homework for this session

Use this template (download the whole folder) and use the instructions in the .txt file to make 3 weather apps that are iterations on each other (so not changing a lot of code but changing some). Get as creative as you want!

There are instructions to download all of the code here too.

Session 2: Video on the Web and in Creative Coding

This session will cover how recording and streaming video on websites for both desktops and mobile devices works. As well as covering advances in computer vision on the web, such as developments with the Mediapipe release from Google and Tensorflow. We will also go over the process of user research for websites.

Video over the web works very similarly to other data and APIs for other websites.

In p5.js, there is a server spun up that creates a capture object which grabs frames from the webcam and allows you to display them in the canvas. This is documented well on the p5.js website here.

We also cover computer vision in this module, which allows computers to understand our visual world through math. Computers use landmarks to break objects into understandable chunks.

OpenCV and Mediapipe are both very popular computer vision toolkits. Mediapipe does amazing things over the web for hands and faces.

This sketch covers integrating mediapipe into p5.js and draws with the user's hands in p5.js-- feel free to duplicate it or download it in the p5.js editor and play around!

We also covered user testing and A Counting: Sign, a project that Nina is producing at the MIT Media Lab for her thesis work, which is actively accepting participation.

Misc sketch tips

Every p5.js sketch has a setup() and draw() loop. Setup runs once. Draw runs consistently and just keeps looping. If you don't want to have a looping drawing, just leave your draw() loop empty and put all your code in setup()

In general, once you have that structure, you can kind of do whatever you want. The key things to remember in general are:

    In general, don't worry if your code is sloppy or not perfect! It is ok. You really don't need to be a production ready software developer to do creative coding -- that is why we have tools.
    Give credit where credit is due. Often my sketches start as a rift from a few lines of something I see so I usually put on the top of the code file "rift from [source]". This also shows that no one does anything really in a vacuuum or alone!
    p5.js is a Javascript library but has a lot of built in functions so always check their reference!
    p5.js uses a Cartesian, computer graphics coordinate system. By default the UPPER LEFT is 0, 0. x increases to the right, y increases as you go down. This is common across all of computer graphics and there are a lot of math reasons we do this, read more here.
    RGB is the default color space, but you can mess around with it with lots of color functions.
    Remember that p5.js has default alignments for primative shapes like rectangles. Check these if they are throwing off your sketches.
    In general, you are probably going to want either a polar coordinate system or rectangular/Cartesian system for sketches. So think about this in your design. Here's an example converting between.
    Object oriented programming can be super powerful in p5.js and create beautiful things and particle systems and force fields. Example 8 has an example of a Particle object if you haven't written objects in JS before.
    While p5.js does support 3D, in general, OpenGL/WebGL implementations are going to just give you more. p5.js has a 3D mode on top of these, but it's just something to keep in mind. I primarily don't do a lot of 3D things so I have very few tips for this.

Some of my favorite...functions/hacks

    background(255, 50) this makes a semi transparent background that gives your code a "ghost" appearance. 255 would be a white background like Example 1.
    p5.js has a built in noise() function so you don't have to write matrices to get noise! Reference here.
    I love creating movement with sinusodial functions and also creating weird shapes with tau (2*pi). Example 5 shows an example of tau being used to create weird, noisy curves. Example 4 shows some cool sin/cos curves.
    map(value, start1, stop1, start2, stop2) is a function that allows you to linearly interpolate between value ranges. Used for color + coordinates. lerpColor(startColor, endColor, percent) lets you do this a bit easier for colors.
    createVector() a lot of my sketches are vector based and what's great about creating a vector is you can do vector math like adding forces and velocities. Reference here.
    curveVertex() allows you to build more complex geometries overall. Refence here. Example 4 also has an example of this.
    blendMode() is a super easy way to start playing with color and pixel blending without writing any code. Reference for all the modes is here . I personally love using EXCLUSION to get a glitchy look.

Code/Downloads from this module

All of the code of the examples on this site is in this repo. More documentation will be added soon.

Resources/Help/Documentation

The p5.js website really is the best place to get help/documentation. In particular they have a learn page with more code explanations and also a documentation/reference page for quick lookups and an examples page.

There are a lot of amazing people in the space that do tutorials and content for this work and code alongs and have lots of tutorials. Daniel Shiffman is a great person to look at for these.

There is also just an amazing community around this space, which I encourage people to look into. There are creative coding clubs and scenes all over the world, and a lot of them are welcoming to everyone and newcomers!

We have created glossaries of relevant terms here, including some in handout forms.