Author: Uinseann Mould

This blog will be a reflection on the project, as well as an explanation of why the final project is quite different from what I wanted.

First, here is the piece in an audio-only form as well as a video form.

My biggest regret with this project has to be the spectrograms. While you can see a few of them if you look at the audio’s spectrogram, the video itself isn’t actually made of spectrograms. This was due to a massive mistake on my part because I had spent around 4 days exporting the images to audacity one at a time only to discover that they weren’t really what I was looking for at all.

This is an example of what they looked like after being reconstructed.

But my mistakes didn’t stop there. After downloading a program called “Audio Paint”, it took around 3 days to create all of the audio bites again; however, this time only the first few worked as I had to close the program early on, which reset settings that I didn’t know could reset, the being what information is stored in each channel. This meant that I had spent around 3 days exporting silence.

Even recreating the effect in a video editor took time because I didn’t have a video editing program that could do it so i had to call around, for a long time.

Despite all my follies, I genuinely am happy with the sound portion of the project. This is probably the first time I have ever done something like this and for the first time, I think it went pretty well, especially some of the sound effects (which I was quite worried about because I’m not that good at them).

If I was to do this project again, I would learn and rely on coding. through all my research, coding came up again and again but I did not go down that route because I haven’t learned anything more than just a bit of pure data patches and even when I attempted it, it was awfully confusing. On top of that, I had no way to troubleshot because I didn’t know anything…

But I know next year will have a lot more coding in it, which I’m looking forward to. This style is still something I haven’t been able to fully realise but I do believe I’ve come a step closer to doing that, and at the end of the day, I am content with respect to the fact that I answered the brief in a way I would respond to it and not just a way that is just easy to do.

What is a spectrogram?

“A spectrogram is a visual representation of the spectrum of frequencies of a signal as it varies with time.”

I’ve been interested in spectrograms for a long time but I always did wonder why I couldn’t find anything on animations using spectrogram images.

I understand why now.

Usually, when creating sound out of images to get that image represented in a spectrogram, the process is quite simple just make sure your image is either a bitmap or raw image data (or any lossless format I believe). The problems start occurring when you try to make the spectrogram into an image.

Now that the process has become going from image – audio – image, the duration of the created audio clip matters a lot more. Depending on the program, the audio created may squash or expand the spectrogram which in turn changes the size of the final image. I found that there weren’t many programs where you could dictate the “aspect ratio of the audio file” and was a big problem for me as I needed the images to be a certain size.

But that problem was only multiplied by the fact that I was trying to make a video out of spectrograms. To get a well-proportioned spectrogram, the sound bites I created were around 5 seconds long; however, in my video, a single image was 1/30th of a second. This meant that the idea I had before (that I haven’t mentioned) to make an animation in an audio file was not feasible as the resolution and speed of a spectrogram just weren’t high enough.

This doesn’t seem to pose a problem in my project, but one thing I wanted to do was edit the visual aspects using audio effects. Looking at the spectrograms I made, it was clear to me that editing a video using audio effects would have unique quirks to them that one could not replicate usually video effects. A solution would be to create a noise track that is affected by the effects, turn that into a spectrogram and then superimpose the spectrogram images of my video onto that track.

Another hurdle was exporting spectrograms as images. Out of all the programs I downloaded for this project, the only one that could turn a sound bite into a spectrogram image was Sonic Visualiser. But that restriction to certain software is a but part of the biggest problem I’ve faced in this project.

Batch processing.

Without enough knowledge in coding, it was going to be hard to make a video around 5000 frames into audio bites of the right size and then into spectrogram images but I sorely underestimated how long it would take to export them all.

Here’s an example.

That image was me trying to work out if I could use audacity to convert images into sound. Each one would take about a minute for the first conversion and 2 for the second one. It was also the first time I had created a video frame from a audio file, and it has an incredibly interesting effect.

Here’s what I use as drone in my piece. The images are stills from the video I made (1 frame for every 180) and then I just lined them up in audacity once they were sound bites. The image was actually 5x-6x longer than that image above just I couldn’t post that because of site restrictions. Also, the audio generated by the stills was really interesting to listen to and definitely something I want to look into more for a future project. Despite being around 20-30ish photos, it took around 1:30-2 hours to do that part, time I don’t have.

But why do I not want to just recreate the filter in a video editing program? I want the piece to be authentic and fresh, on top of that, it’s incredibly hard to recreate. However, if I do find myself short on time it might be wise to just recreate the look as while the piece loses something important, it is way more important that there is a visual aspect.

In this blog, I going to go through my intentions and plans for the main sound aspect of my project, which is the musical part.

In the piece, the way I deliver my lines is through rap. I wish I could have sung a song instead but my order of operations has made it so I can come up with anything for the rap, as long as is in time (I’ve filmed it at this point). Another reason I decided on rap is that it’s a lot easier to come up with something like a spoken word piece which is arguably atonal verses singing a predetermined lyric and tune.

The 3 sections of the piece are Soundscape; The Noise; The Melting Pot.

The first section is there to show off some skills I’ve gained in the year via sound design/effects. The way I plan to show that is by using the referencing of different sounds as an excuse to synthesise sounds instead of sampling them. Example – When I say “then the birds, and it’s third” a synthetic bird sound will play, and then it’s third.

The second section is meant to mostly be comical. Musically, it’s the least defined section and it doesn’t necessarily show off my skill but I did think that it would be wrong not to mention noise on a sound art course. Also, I want to sell that uncanny valley feeling and peel back the curtain with it as it strangely breaks the fourth wall (in some respect)

The third part is personally the most important part and integral to the piece itself. “The Melting Pot” is in reference to how I express my creativity in many different ways and the way I’ve dealt with being interested in so many different fields. This project is no different. As an audio-visual piece, no side is complete without the other and abstractly represents me in some way.

The way I’m going to tackle the music is by layering the visual with a click track and then improvising over it. As someone who watches a lot of YouTuber content, you hear a lot of royalty-free music which is the main inspiration for the different style of songs in the piece. I wanted to use the jazz-inspired track as the main part because it gives a homely and is common in vlog videos.

Spectrograms also lie at the heart of the project, which is why I’ve decided to layer everything else with a the “spectrogram track”. While it can’t really be animated like a video, it’ll feature frames from the video meaning that if you viewed the sound in something like Sonic Visualiser, you should see parts of the video in the sound (and not just the video).

My next blog will be focused purely on the details of the biggest problem I’m facing in the project, the spectrogram.

Before I can start, I need a script. I’m planning to use the video audio for the vocals in my sound piece so a script is needed because I can’t improvise it.

Here’s one I made earlier: 

“(Jazz) 74 

Hey Everybody! Welcome back to the show! Its the yearly review of all the highs and the lows.  

As you all know, this year I was studying sound art, but I’ve only got 5 mins so I suppose I’ll make a start. 

So, I could separate the year into 3 main veins, what I learnt, what was interesting and how my craft has changed. 

But, that’s a little dry so I’ll give this a try and I’ll separate this video into these 3 topics 

First we’ve got soundscapes (tweet tweet tweet) 

And then we’ve got the noise (silence or low buzz longer than it should be) 

And lastly we have what I like the melting pot (dodo, dodo, dodo, dodo do) 

(Epic music, like a pads and lower tempo) 60 

It was in the first term, the year, still fresh. 

We had a task called a sound walk, to close your eyes and, walk 

Hark, what is that? The sound of the breeze 

The rustle of leaves 

The buzzing of bees 

And then the birds, and its third, and maybe a plane 

And because we live in london, there’s also a car and a train 

(Jazz) 74 

But I’ll leave it at that because it’s a task of few joys, so here’s the next topic, I think I called it the (noise) 

(Rap) 84 

It was fun to expand my skill set, to rip the safety net, to learn from artists met, even if you can’t see it. 

‘cuz I know now how to sculpt white noise into pink, make track using ink, and plug in a modular sink, synth. 

Something I can’t forget as a sound artist you have to have met, the work of John Cage, 433 moments of silent neglect  

(Silence) 

(Cough) 

Brilliant 

(Jazz) 74 

But really this is just the lead up to my last part, an Amalgamation of my skills, the melting pot, of my art.  

I couldn’t say I’m a sound artist, because before this course. I was making art, music, videos and all differnt sorts 

and if i was forced to answer, what is a sound artist to you? hmm, Well, I could probably show you through 

this, that, that and that this. 

If those are random things, sound arts are the links. 

To round things off, this year was nice, I furthered my craft, even if at great price 

and hope things go well, for I don’t know how to program, or make a rap, but that’s enough of that 

“ 

As one can tell, I am neither a linguist nor a rapper; however, I thought this was a decently comical script while also being okay (even if quite amateuristic) musically. 

What I’m trying to do with the script is be interesting. I could easily write something that portrays more of what I’m trying to go for or have more on-the-nose jokes but the premise is funny enough and it’s better to not say everything well than say everything badly. 

I haven’t really made it clear in the script, but what I’m going for is a rap and the numbers next to the “themes” of each part represent the bpm I’m rapping at. As something who hasn’t really rapped before, it will be interesting what the product is. 

Why was the last blog post relevant to this project? Well, recently one of my friends told me that they’re going to Japan to study for a week and asked if to edit some vlog videos for him. I thought it would be great fun to do; however, all I thought about was making a vlog that would be popular, with all the YouTuber troupe included. While it’s fine to do that, it did make me debate why I couldn’t just make something without an ulterior motive, and that sent me on the thought spiral in the last post.

This is why the audio-visual piece I want to make is going to be a riff on youtube culture, being hopefully unsettling and pleasing at the same time, and manipulated in a certain way. The way I’ll do that is by making a video that is a musical that talks about everything I learn this year, the main reason for it being a musical is because the nonsensical premise of speaking through song could be used to reference formulaic youtube vlog videos are and how one has to suspend belief to a certain extent in musicals, implying that one should do the same for youtube videos (which many don’t). But honestly, I just thought it would be a lot funnier to make a musical.

But that certain way? Sonification, or specifically sonifiying a video I make, then using that to then create an image sequence of spectrograms and finally animation of those spectrograms. This process was something I tried to do 2 years ago for my art foundation course, but it was the year that covid struck and I didn’t have the tools or knowledge to follow through with my idea.

https://www.behance.net/gallery/95455971/My-Soundscape

This is a link to my attempt at this idea but I didn’t get to the part of animating a video using the audio of the original images/video. I know a lot more now but I know I still don’t know enough to animate a whole video using spectrograms in an efficient way.

That is the biggest problem I’m going to face with this project, Time. Will I have enough of it?
Well, I first have to make a script for my video with directions for the editing, what I say, and a basic idea of what the music and sound effects will be like.
Then, I have to make most of the sound aspect of the piece, making sure it’s in time with the video.
Next, I have to convert the video into sound, which is a big problem because I know of a few ways to do that but they can all only be done as images. This means I would have to make an image sequence of the whole video (I’ll make it in 30fps) and convert them individually into sound bites.
Honestly, that’s the simple part (kind of) because I then have to convert all of the spectrograms into images again which complicates this process and makes it a lot harder than just turning an image into a sound. That would be around 7200 different sound bites and thus frames, all of which I have to individually capture (I know of ways to do this in FFmpeg; however, I am not that learned when it comes to coding and it becomes a lot more complicated to make something when I’m going from video to sound to images and then back to video).
Lastly, I would compile the images into a video again and add the “soundtrack” and also a representation of the sound of the video spectrograms as a drone throughout the piece.
I also want to manipulate the video using the audio effects from the soundtrack, so I need to work that out as well.

I don’t really need to do this, I could just make a sound piece in the same vein. Nevertheless, I’ve chosen to make something like this because the whole reason I talk about what I’ve learnt during my year in sound arts is that it has improved my practice outside of sound arts (video editing and music). I want to a piece only I could conceive, something original to me, and my way of doing that is making a piece that is an amalgamation of all my creative disciplines.

Before I did sound arts, I was, and still am, a fine artist and musician. Through this piece, I want that aspect of myself to be shown because I do believe it’s the links between different disciplines that elevate my work.

At the start of any project, I usually start off with making a blog post about idea generation and while I could do that again, I’ve decided against it.

That’s mainly down to how I don’t really have a good idea for this final project (other than the idea I’m pursuing) and knowing the scope of my idea, I really just have to press on, start theory crafting and working through the problems I know I’ll face.

So, what’s my idea for this project? In my notes, it’s an “audio-visual piece using pure-data as the main audio engine and Ableton for its convolution reverb and a few other effects and maybe FL studios.”.
At the start of my supporting text, it’s this
“As the brief was practically about a culmination of all the skills we have accrued over the year, the concept of the piece was a musical version of a stylised youtube vlog which talks about have I’ve learnt and what that’s actually informed about my work during the year and in the future.

The visual aspect of the piece is a video to go with it; however, the video will actually be a spectrogram animation as in the video is played as a spectrogram. on top of this, the video as an audio file is put through the same effects as the audio track as well as superimposed on to the acoustics sounds of the video or the acoustic sounds of instruments in the video.”

In short, an audio-visual piece where the main crux of the idea is predicated on spectrograms and turning images into sound.

While I haven’t fully fleshed out the content of the piece, I do know that I want to touch upon that “YouTuber vlog” feel, the uncanny valley of making a psudo-satire piece on that sub-culture, some sense of musicality and behind that a semi-honest reflection on what the year in sound arts has actually done for me.

Youtube and a lot of stuff in the curated, performed space is a great source of entertainment but has come at the cost of restraining creativity (kind of) and cultivating a generation that has a really strange relationship with what they watch/mental health issues for money.
First, on youtube, there are certain formats and things that work and get the most amount of exposure to grow a channel and thus make more money. One could then argue that this actually restrains creativity because it’s harder to be creative on youtube and stay that way if you aren’t making enough money.
Youtube acts more like a planned/command economy now than it did in its hayday when it was more like a free market. If it is dictated that certain pieces of media are made more valuable than others based on the will of a small body or group that only cares about money and is partisan in its politics, that will is reflective in the art that is produced.
To me, art is many things. A way to make sense of the world, a way to understand and argue with the ideas proposed in the current day, to create stories and marvels that transport people to a different universe and makes them reflect on how they interact with their world. Money is just a means to an end, not the end.
But on youtube, when the popularity of your video is based on an algorithm that only cares about what companies like and not what consumers want, one has to be careful when they become a “YouTuber”. One always has to think, is the money for the art or is the art for the money.

I’m not saying it’s bad to make art that makes money because I would love to live a life just making all the art I want, but I would say someone who lives exclusivity making art for the sake of money isn’t a true artist. To me, it’s about intentions and what one actually wants out of their art that decides what is true art. I say that because movies are amazing, but they also make a lot of money. I would call quite a lot of movies “true art” because even if they make a lot of money, the intent behind it isn’t for the money; however the money is a by-product.

Honestly, it is a cutthroat definition that is quite flawed and not one I would use as a sweeping definition of art; however, I do think that there needs to be a push back of mega-corporation dictating culture and thus dictating art. I believe that’s a big problem that we as artists have to face because in a world where again a lot of cultures are created by companies that want money more than anything else, but also one where it’s so easy to be a “good artist”, make art, and be creative, there is so much art that one has to debate “what is actually art”, and that could be seen as very telling on the state of art contemporaneously.

There’s also that aspect with youtube and parasocial relations/interactions that everyone talks about but also is silent about how that makes money. I would say it’s a bad thing because let’s be honest, it has an adverse effect on people, especially children, and is one of the greatest propaganda tools that the world has ever seen. I also don’t think that that’s a good thing.

All of this is only relevant because youtube is the video platform of the world (other than in Asia I believe) and monopolises culture in that space, and I think it’s very important to talk about, but more importantly, make a joke of. Well, at least a joke of the youtube culture.

How I’m going to that, why anything in this blog is relevant, what the project means to me as well as why I thought about this in the first place are all things I will divulge in my next blog. On top of that, I’ll talk about the main idea behind the style or process of the piece, mentioning more about the video into spectrogram idea.

I thought it would be interesting to articulate my thoughts and tangents in this blog post because for this project I would rather delve deeply into what I’m trying to touch upon with my piece than just use some buzz words to give “meaning” to my art.
One could say that I would rather my art be a “deep sea” than a “lagoon”