Category: python

Behind the Scenes of “Rawkintrevo’s House of Real Time IoT Analytics, An AIoT platform MVP Demo”

by rawkintrevo, posted in big data for noobs, Engineering, flink, How-Tos, IoT, K8s, openwhisk, python, React, video

Woo, that’s a title- amirigh!?

It’s got everything- buzzwords, a corresponding YouTube video, a Twitter handle conjugated as a proper noun.

Introduction

Just go watch the video– I’m not trying to push traffic to YouTube, but it’s a sort of complicated thing and I don’t do a horrible job of explaining it in the video. You know what, I’m just gonna put it in line.

Ok, so Now you’ve see that.  And you’re wondering? How in the heck?!  Well good news- because you’ve stumbled to the behind the scenes portion where I explain how the magic happened.

There’s a lot of magic going on in there, and some you probably already know and some you’ve got no idea. But this is the story of my journey to becoming a full stack programmer.  As it is said in the Tao of Programming:

There once was a Master Programmer who wrote unstructured programs. A novice programmer, seeking to imitate him, also began to write unstructured programs. When the novice asked the Master to evaluate his progress, the Master criticized him for writing unstructured programs, saying, “What is appropriate for the Master is not appropriate for the novice. You must understand Tao before transcending structure.”

I’m not sure if I’m the Master or the Novice- but this program is definitely unstructured AF. So here is a companion guide that maybe you can learn a thing or two / Fork my repo and tell your boss you did all of this yourself.

Table of Contents

Here’s my rough outline of how I’m going to proceed through the various silliness of this project and the code contained in my github repo .

  1. YOU ARE HERE. A sarcastic introduction, including my dataset, WatsonIoT Platform (MQTT). Also we’ll talk about our data source- and how we shimmed it to push into MQTT, but obviously could (should?) do the same thing with Apache Kafka (instead). I’ll also introduce the chart- we might use that as a map as we move along.
  2. In the second post, I’ll talk about my Apache Flink streaming engine- how it picks up a list of REST endpoints and then hits each one of them.  In the comments of this section you will find people telling me why my way was wrong and what I should have done instead.
  3. In this post I’ll talk about my meandering adventures with React.js, and how little I like the Carbon Design System. In my hack-a-thon submission,  I just iFramed up the Flink WebUI and Kibana, but here’s where I would talk about all the cool things I would have made if I had more time / Carbon-React was a usable system.
  4. In the last post I’ll push this all on IBM’s K8s. I work for IBM, and this was a work thing. I don’t have enough experience on any one else’s K8s (aside from microK8s which doesn’t really count) to bad mouth IBM. They do pay me to tell people I work there, so anything to rude in the comments about them will most likely get moderated out. F.u.

Data Source

See README.md and scroll down to Data Source. I’m happy with that description.

As the program is currently, right about here the schema is passed as a string. My plan was to make that an argument so you could submit jobs from the UI.  Suffice to say, if you have some other interesting data source- either update that to be a command line parameter (PRs are accepted) or just change the string to match your data.  I was also going to do something with Schema inference, but my Scala is rusty and I never was great at Java, and tick-tock.

Watson IoT Platform

I work for IBM, specifically Watson IoT, so I can’t say anything bad about WatsonIoT.  It is basically based on MQTT, which is a pub-sub thing IBM wrote in 1999 (which was before Kafka by about 10 years, to be fair).

If you want to see my hack to push data from the Divvy API into Watson IoT Platform, you can see it here. You will probably notice a couple of oddities.  Most notably, that only 3 stations are picked up to transmit data.  This is because the Free account gets shut down after 200MB of data and you have to upgrade to a $2500/mo plan bc IBM doesn’t really understand linear scaling. /shrug. Obviously this could be easily hacked to just use Kafka and update the Flink Source here.

The Architecture Chart

That’s also in the github, so I’m going to say just look at it on README.md.

Coming Up Next:

Well, this was just about the easiest blog post I’ve ever written.  Up next, I may do some real work and get to talking about my Flink program which picks up a list of API endpoints every 30 seconds, does some sliding window analytics, and then sends each record and the most recent analytics to each of the end points that were picked up, and how in its way- this gives us dynamic model calling. Also- I’ll talk about the other cool things that could/should be done there that I just didn’t get to. /shrug.

See you, Space Cowboy.

 

 

 

Pavlov’s Sandman Pt 2: Background and Detecting Snores

by rawkintrevo, posted in Engineering, How-Tos, python

Welcome to the long anticipated follow on post to Pavlov’s Sandman: Pt 1.  In that post I admitted that I had a problem with snoring, and laid out a basic strategy and app to stop snoring.  They say that admitting you have a problem is half of the battle.  In this case that is untrue- hacking the shock collar was half of the battle, and the other half of the battle was detecting snores.

I had a deadline for completing this project, ODSC East in Boston, at the beginning of May.  What the talk ended up being about (and in turn what this blog post will now be about) is how in “data science” we usually start with a problem, that based on some passing knowledge seems very solvable and then the million little compromises we make with ourselves on our way to solving the problem/completing a product.

This blog post and the one or two that follow will be an exposition / journal of my run-and-gun data science as I start with assuming a problem is easy, have some trouble but assume I have plenty of time, realize time is running out, panic, be sloppy with experiments, make a critical break through in the 11th hour, and then deliver an OK product / talk / blog post series.

A brief history of the author as a snorer.

As best as I can tell, I began snoring in Afghanistan.  This isn’t suprising, the air in Kabul was so bad the army gave me a note saying that if I ever had respitory issues, it was probably their fault (in spite of the fact that everyone was smoking a pack per day of Luckys / Camel Fulls / Marlboro Reds ).  This is to say nothing of the burn pits I sat next to to keep warm while on gate duty from December until March, or the five thousand years of sheep-poo-turned-to-moon-dust always blowing around in the country side west of Kabul City.

As a brief aside, do you know what’s really fun about trying to “research” when you started snoring? Making a list of ex’s and then contacting each of them out of the blue with a “hey, weird question but…”, it’s like a much more fun version of the “Who gave me the STD?” game.

After Afghanistan, girls I was sleeping with would occasionally complain of my snoring.  This came to a head with my ex though.  She would complain, elbow me, sleep on the couch, etc. But I was more concerned when the issue came up with my new girl friend (a very light sleeper, or I’ve gotten much worse about snoring).  This was especially concerning, because I had tried every snoring “remedy” on the internet and had no success.

Break throughs on other fronts.

I have a puppy named Apache, and was at the trainers.  They convinced me that I should start using a wireless collar ( a shock collar ).  The guy who trained me taught me that you don’t want to hurt the dog, you want to deliver the lightest shock they can feel and just keep tapping them with that until they stop doing what they should be doing.  The shock should be uncomfortable, not painful.

One of the “remedies” I had tried for my snoring before was an app called Sleep as Android. In this app there was an “anti-snoring” function where the phone would buzz or make a noise when you were snoring- this had no effect, but I had always wished I could rig it out to a shock collar.

Finally, in November of last year- I discovered that you can buy a shock collar which is controlled via Bluetooth on Amazon for about $50. (PetSafe).

I have done some device hacking, I figured I could figure out the shock collar easily enough. Detecting snores also seemed easy enough.  So I wrote a paper proposal for ODSC and started working on the issue. (And wrote the last blog post which recorded me snoring).

Snore Detection and the Data Science Process of the Author

Traditionally when I start on a project, I try to come up with an exceptionally simple-enough-to-explain-to-a-5-year-old type of algorithm just so I have a baseline to test against. The benefits to this are 1) having a baseline to train against, but 2) to become “familiar with the data”.

My first attempt at a “simple snore detector” was to attempt to fit a sin curve to the volume of the recorded noises.  This got me used to working with Python audio controls and sound files.  I also learned right away this wasn’t going to work because the “loud” part of the snore happens then there is a much longer quiet portion. That is to say we don’t breath evenly.  I don’t have sleep apnea (that is to say I don’t stop breathing), so the snores are relatively evenly spaced apart, but there are also “other noises” and various other reasons, the sin wave curve fitting just wasn’t ideal.

At this point I went back and read some academic literature on snore detection. There isn’t a lot, but there is a bit.

Automatic Detection of Snoring Events Using Gaussian Mixture Models by Dafna et. al.

Automatic detection, segmentation and assessment of snoring from ambient acoustic data by Duckitt et. al.

An efficient method for snore/nonsnore classification of sleep sounds Cavusoglu et. al.

My Synopsis

Dafna reconstructed when the patient was snoring by looking at the entire night of data and looking at how volume compared to the average.  Following his method and converting it to “real-time” detection however, was going to be problematic.

Duckitt created a Hidden Markov Model (mid 2000s speak for LSTM) (yes I know they’re not that same) with the states snoring, not-snoring, other-noises, breathing, duvet-noise.  An interesting idea, one I might visit for a “real version”.

Cavusoglu looked at subband energy distributions, inter and intra individual spectra energy distributions, some principal component analysis, in other words- MATH.  I liked this guys approach and decided to mimic it next.

PyAudioAnalysis

pyAudioAnalysis is a package created and maintained(ish) by Theodoros Giannakopoulos.  It will break audio files down into 32 features, including the ones used by Cavusoglu.  From there I tried some simple logistic regression, random forrest classification, and K-nearest-neighbors classification.

The results weren’t bad, but I was VERY opposed to false positives (e.g. getting shocked when I didn’t snore.  The numbers I was getting on validation just didn’t inspire me (though looking back I think I could probably have been ok.)

Screen Shot 2018-05-22 at 1.59.18 PM

Back to Basics

A quick note on the “equipment” I am using to record, it is a laptop mic, which is basically trash.  Lots of back ground noise.  At this point, I had been playing with audio files for a while.  I decided to see if I could isolate the frequency bands of my snoring.

In short I found that I normally snore at 1500-2000Hz, 5khz-7khz, and occasionally at 7khz-15khz.  I decided to revisit the original loud noises idea, but this time, I would filter the original recording (for the last 5 seconds) into 1500-2khz and 5-7khz. If there was a period which was over the average + 1 standard deviation for the clip, which lasted longer than 0.4 seconds, but less than 1.2 seconds, then there was a pause in which the intensity (volume at that frequency band) was less than the threshold (mean + 1.5 stdevs) for 2.2-4.4 seconds and then another period where the intensity was above the threshold for 0.4 to 1.2 seconds, then we would be in a state of snoring.

This worked exceptionally well, except when I deployed it, I accidentally set the bands at 1500-2khz and 5khz-7khz, which missed a lot of snores.  I will be updating shortly.

Screen Shot 2018-05-22 at 2.13.08 PM.png

In the upper image above, we see the original audio file intensity (blue) over time, and in orange is the intensity on the 5-7khz band. The black line is the threshold. This would have been classified as a snore (except it wouldn’t because I wasn’t watching 5-7khz on accident).

Conclusions

So that is the basics of detecting snores in real time.  Record a tape of the last 5 seconds of audio, analyze it- and if thresholds are surpassed, then we have a “snore”. Fire the shock. but oh, firing the shock and hacking the shock collar- that was a whole other adventure.  To be continued…

 

Pavlov’s Sandman pt. 1

by rawkintrevo, posted in Engineering, How-Tos, python

I snore. I’m perfect in many most ways, but this is the one major defect I am aware of. Like any good engineer, upon learning of a defect I set out to correct or at least patch it. The name of this little project stems from Ivan Pavlov’s experiments with conditioning as well as Operation Sandman, a CIA program for sleep deprivation torture of detainees at Gitmo.

The naming convention is evident when one looks at the strategy I am taking to correct my unpleasant snoring habit.  I have an app on my phone (Sleep as Android) that tracks my sleep, is a cool alarm, and among other things, tracks my snoring.   From this app I know:

  1. It is possible to “detect” snoring.
  2. I don’t rip logs all night, but in bursts.

I have tried a number of things to correct this throughout the last year including mouth piece, a jaw strap, a shirt with a tennis ball sewn in the back, video recording myself sleeping to see if I can detect a position where snoring occurs, essential oils/other alternative medicines, etc.  Failing all of these, I now begrudgingly turn to “Data Science” the form of mysticism reserved for the exceptionally desperate.

The plan of attack on this endeavor is as follows:

  1. Create a program that detects loud noises (preprocessing).
  2. Differentiate between snoring and other nighttime noises (dog, furnace, coughs, etc).
  3. When snoring is detected administer a small shock via a Bluetooth controlled shock collar for dogs which I will be wearing as an arm band.
  4. Video record results of me electrocuting myself while trying to sleep and post to YouTube, elevating me to stardom.
  5. Possibly train myself to stop snoring.

The title of this project should now be apparent, as I am hoping to “train” myself to not snore, and if I were developing this commercially, I’m fairly confident that any beta-testing I did would (rightly) classify me as a war criminal.

In part one of this series (I promise that all the time and have a bad habit of not following through), I present the code and methods I have developed for detecting loud noises / building my dataset.

In future parts, I hope to do some cool things with respect to signal processing and Bluetooth device hacking with Python.

GitHub Repo

Loud Noises

The first step is to record sounds. The code presented is fairly elegant and easy to follow (for now).

We have a class AudioHandler which contains some variables and a few methods.

In addition to the alsaaudio handler, we have:

  • rawData a list for holding caching the audio recorded
  • volume which will be used to create a csv of  timestamp, volume data
  • and various thresholds to prevent getting multiple shocks in unison, a warm-up period, volume threshold for recording, etc.

The methods are:

setThreshold

This action listens to the mic for a number of seconds and attempts to dynamically set the threshold. It’s not great- for my first night of use I ended up doing it manually by observation and laying in bed and making some breathing / fake snoring sounds and seeing where it hit.

dumpData

This method writes the csv and audio to disk

executeAction

This is a place holder, and later will be used to call the “shock”

run

This is where all the fun happens. In short it

  1. Attempts to set a baseline threshold considering mic sensativity, background noise, etc.
  2. In a while loop it then
    1. Listens to the mic
    2. If the volume is above the threshold:
      1. Set a recordingActive flag to True if it wasn’t already
      2. If it wasn’t already, timestamp when this recording started
      3. Determine how long the current recording has been going on.
      4. If it has been going on for some duration, call the executeAction method and dump the recording to disk.
    3. If recordingActive is true, add the raw audio and volume levels to rawData and volume

And that’s about it. Again, look at the code for specifics, but all in all pretty straight forward.

Last night I recorded.

Building a Training Set

As a priest of the mystic art of data science, the first part of any ceremonial ritual is to create a training / testing data set.

This was a very tedious part of my day.  I went through the recordings and seperated them into two folders “snore” and “non-snore”.  Well, I did this for about 30 minutes, and got approx 80 samples of each. Then I moved the rest into an “unlabled” folder… you know for testing purposes, not because I was super bored.  Perhaps if I had an intern, this would have been a more robust set.

Finally I wrote a little python script that will copy the csvs over appropriately to all of the wav files you sorted out into the proper directories.

Stay tuned for part 2, where we’ll do some signal processing to differentiate the snores from the noises!