What is the Juno chrome extension?

The Juno chrome extension was created to allow users to seamlessly interact with profiles they have either selected or created in Juno. The Juno chrome extension automatically syncs with Juno allowing for a smooth transition between both platforms.

Interaction is conducted using a assigned shortcut. To interact the user holds down the shortcut, which starts speech recognition within the browser. To end a session the user can click the assigned shortcut once.

Speech Recognition

Once the shortcut is held, speech recognition begins. Speech recognition entails audio recording via Web Audio API via microphone input. The recorded data is then sent to Google Cloud Speech-to-Text API, in which the user's speech is returned as a string.

Intent Recognition

The user's intent is deteced using Azure's Conversational language understanding service which takes a query in the form of text (in our case the user's speech) and returning the top intent, top entity, and confidence level.

For example, given the user's speech: "Open Google"

The top intent would be: "Open Webpage"
The top entity would be: "Google"
The confidence level would be: .9 (in which 1 indicates 100% confidence)

Large language model interaction

After the user's speech has been retrieved, the user's speech is used to query the large language model. As well, the user's selected prompt via their selected profile is used to provide the foundational context and background information to the large language model. Specifcally the prompt specifies the large language model's personality, context, interaction guidelines, background, and temperature (variability in responses)

Text-to-Speech using Elevenlabs

After the large language model's response is retrieved, Elevenlabs text-to-speech is performed on it using the user's selected voice in their profile.

User Interface

The user interface was intended to be clean and minamilsitic in design. Shadcn was used for many of the components such as the combo box, breadcrumb, and card display. From the main page the user can easily browser through and select their profile, as well as assign their shortcut. Which are the two main functionalities intended for the chrome extension.