In March 2016 IBM announced an internal contest for IBMers. Contest goal was to create a fun IoT Gadget. At the time I had recently joined the company, so that contest was the perfect excuse to get hands-on experience with IBM cognitive technologies. It was a chance to build something with Artificial Intelligence, Cloud Computing, Internet of Things and whichever other technology I could find in IBM Bluemix (now called IBM Cloud).

This article discusses the project I created in response to that contest. A voice-based candy machine that I call ‘Cognitive Candy,’ or just ‘Candy,’ for short.

The Idea for voice-enabled candy machine

The idea was to create a machine that people could talk to and ask for candy. However, to make things more interesting, the machine would not simply give out candy. I wanted there to be a conversation aspect to it. I want people to engage with the machine as if they were talking to a person.

For that, I added a ‘small talk’ element to Candy, where she would make an observation about the current weather conditions looking for user’s response. E.g., Looks like it’s raining outside. Did you bring an umbrella?

Additionally, I wanted to introduce a business aspect to this candy machine. That was not a contest requirement, but I thought that would be a nice touch. I decided that the machine would collect some information about the user before giving out the candy. In essence, Candy was a polling machine, that gave out a prize in exchange for user information.

To keep things in a fun spirit, Candy asks the person for their favorite color. The initial idea was to ask “are you voting for Trump or Clinton,” but I decided to stay away from politics. Yes, I’m smart.

The technology

Cognitive Candy Architecture

Per the architecture diagram below, we have “internet” components that run in the cloud and “thing” components that are sitting on the desk. Let’s start covering the hardware & local software (CandyOS).

Hardware: Candy uses a Raspberry Pi 3 (built-in Wifi). It has a custom designed 3D printed enclosure, split in a ‘head’ (photo below) and ‘dispenser’ (not in photo). Electronic components: sound amplifier, speaker, microphone and a circuit board for buttons, and LEDs.

CandyOS is the operating system inside Candy. It manages the user interaction logic and integrations with the outer world. It’s a node.js program written in Node-RED. There are shell scripts to interact with microphone and speaker.

Speech Recognition & Synthesis: Candy relies on external Speech libraries. I experimented with free open source libraries like PocketSphinx & eSpeak but ended up going with the IBM Watson Speech to Text.

IBM Cloud services used:

IBM Speech to Text
IBM Text to Speech
IBM Watson IoT Platform
IBM Blockchain
Blumix Runtime: Node-Red + Cloudant
Real-Time Insights
Watson IoT Dashboard

CandyCloud: cloud application is providing remote management, inbound connections, etc.

Candy Dev Tools includes the Candy Brain Editor and Brain Tap (so you can type instead of speaking to Candy).

For now, the technology section is short on content as this article is a draft. If you want me to expand on any aspect, please leave a comment. Meanwhile, you can read more about my project this interview

The demo

Here’s a video of Candy in the Austin Maker Faire May 2016. It’s a short, must-watch video.

A couple of observations from that video – and the Maker Faire experiment

Having a chair in from of the booth was key. I gave people the notion that only one person at a time should interact with the machine. If I took the chair away, there would be 5+ kids asking for candy at the same time. I’d lose control over the demonstration.
The background noise at the fair was bad. So bad I had to turn off actual recognition and be the ‘Mechanical Turk’ operating the machine and punching in the user responses. In fact, that’s a big challenge for speech to text technology.

If you have more time to spare, you can watch the contest submission video: https://vimeo.com/174682798

Other materials on YouTube:

Final thoughts:

Cognitive Candy was ‘Staff Pick’ at Austin Maker Faire 2016
Cognitive Candy won IBM fun IoT gadget contest ( 2nd place – $800 prize!)
Cognitive Candy became a popular project inside of IBM:
- Read Igor’s Interview to IBM Internet of Things blog
- I was invited to fly to Las Vegas and showcase Candy at the World of Watson 2016 – IBM most important conference. Stay tuned post on this.

I listed Cognitive Candy on Kickstarter – a popular crowd-funding website – with the goal to keep the project alive, however, I found I just did not have the time to support it. Further, right after I created Candy similar open-source projects – with much better staffing – started popping up such as ReSpeaker by Seeed Studio. Check them out.
My work on Cognitive Candy has led to a few ‘innovation disclosures.’ Few patent applications being filed. I’ll post details here when these become public.

In closing, it was gratifying to work on Cognitive Candy. I learn a lot of new technologies and new user interface. I had lots of fun demoing it for kids and geeks, meeting many smart folks along the way. I got few awards, interviews, talks and even few patents out of it. Success!

Thanks for reading!

Igor