Microsoft AR Hackathon

I attended Microsoft + Kal Academy AR hackathon  hoping to get more experience in Hololens and mixed reality. The event was sponsored by Microsoft, Amazon, and Kal Academy.

We won the first place and received Amazon Echo devices for the prize.

This was a very short Hackathon with only 6 hours in total. One main problem of Hololens and its productivity related applications is that the basic input methods(Gaze, touch, gesture) are not appropriate for text searching or typing phrase. Considering the possibilities of Amazon Alexa, we chose to work on AR shopping experience by integrating Hololens and Alexa. While Hololens has its own speech recognition and VUI features, it is no match to Amazon Alexa in terms of features and accuracy. Therefore, we wanted to bridge the gap between two unique devices, hoping to provide a visual UI for Amazon Alexa through Hololens, because Hololens is quite limited with precision-based inputs. 

We created Alexa Skill & Lamda Function that captures and sends key phrases from Amazon Echo Dot. A temporary web server was built to automatically refresh by inserting search word into the targeted search field. Then the resulting page was opened in a custom-made Hololens application using web-view module. Below is the simple procedure how you would speak to Alexa to search items on an online shopping website.

Here is a short clip of our demo. There is an Echo Dot on the podium, but we didn't speak into the mic. The mirror screen of Hololens had some delays, even though it was connected via USB.