Vivint Smart Home is blurring the line between two emerging technologies – audio analytics and predictive analytics. The Utah-based company, ironically famous for selling security and home-automation systems door-to-door, has filed a patent application for identifying guests by their unique sound signatures, including how they knock on doors. Through artificial intelligence (AI), a home-control system could determine how to respond to these sound events.
“System and methods for correlating sound events to security and/or home automation system operations,” was filed in October 2015 and published April 20, 2017. The patent application describes the limitations of smart doorbells and facial-recognition systems in identifying guests, and suggests a method of measuring sound signatures to determine who’s knocking or loitering.
A “knock,” according to the patent application, might include:
a rapping, slapping, pounding, or kicking on the door, doorframe, or windows or walls surrounding the door, or may further include clapping, ringing a bell, unlocking a door, turning or jiggling a doorknob, walking up steps, calling out or whistling, approaching in a car, or any other detectable audio signal.
The sound patterns would be matched to a user’s database of known visitors, and could elicit different responses from a smart-home system.
For example, “[T]he user may input a preference indicating that, where the FedEx deliveryman is identified as knocking at the door, a recorded voice message should be broadcasted at the front door, telling the deliveryman to please leave the package on the bench by the front door.”
Sound patterns with similar characteristics as the FedEx guy’s, might alert homeowners that some kind of delivery person has arrived.
Audio that matches known offenders could trigger the lights in the home to turn on, and a siren to blare. And, although not mentioned in the patent application, those sound fingerprints might be shared among neighbors to alert them of potential miscreants.
Audio analytics is nothing new, and intelligent sound detection systems are becoming more advanced. Indeed, we referred to several pioneers in the field when we named the category among the Top 5 Home Technology Trends and Opportunities for 2015.
Vivint has multple patents and applications related to triggering an alarm system based on certain inputs or events such as geo-location and sleep-sensing. Sound-sensing would be just another event.
Highlights from the Vivint patent application (emphasis added by author):
BACKGROUND
[0001] The present disclosure, for example, relates to a security and/or automation system, and more particularly to implementing at least one home automation system operation based, at least in part, on comparing a received monitored sound event with a received input regarding at least one home automation system operation associated with the sound event.
[0002] Home automation systems are widely deployed to provide various types of communication and functional features such as monitoring, communication, notification, and/or others. These systems may be capable of supporting communication with a user through a communication connection or a system management action.
[0003] Typically, when a guest approaches a front door and knocks, rings the doorbell, or otherwise announces his presence, the homeowner is unaware of the identity of the knocking guest, and has only two options for responding to the knock: opening the door, or not opening the door. The homeowner may check a peephole to learn the identity of the knocking guest, but in order to do so, the homeowner must approach the door, and likely must reveal his presence to the knocking guest. Additionally, where the homeowner is not home at the time of the guest's knock, the homeowner may not know who came to the door, and may not have any means by which to respond to the knocking guest's approach.
[0004] Existing guest identification means may provide for video cameras positioned at the front door, and in some cases may provide facial recognition technologies, to identify the knocking guest. Yet these methods may be unreliable in accurately identifying the guest, and may still require direct homeowner response to the knock. Existing voice recognition systems may also provide for identification of guests, but may be limited to identifying known visitors. Accordingly, it may be desirable to provide a means by which the identity of the knocking guest may be determined using audio detection systems not limited to voice detection, and by which the user may be notified of the knocking guest's identity. Further, it may be desirable to implement one or more functionalities of a home automation system on the basis of the knocking guest's identity.
[0005] Accordingly, in one embodiment, a method for security and/or automation systems is provided. In one embodiment, the method may comprise detecting a first sound event at a home entry point using one or more sensors. The method may further comprise receiving input to associate at least one home automation system operation with the first sound event, and may further comprise storing the first sound event. The method may further comprise initiating the at least one home automation system operation associated with the first sound event.
[0006] In some embodiments, a detected knocking event may be compared with a precompiled library of known audio signals stored at the home automation system in order to identify and authenticate the knock or other sound event. Thus, the library may contain a plurality of sound clips and/or sound “fingerprints” associated with one or more known sound events and the guest who produces them. For example, the library may contain clips of knocks for the homeowner's family, friends, neighbors, and regular delivery people, such that when any of those guests knocks or produces an audio signal corresponding with the sound clips stored in the library, that guest may be authenticated and identified by the home automation system. The sound “fingerprints” may include data related to any of knock or other audio input intensity, source, location, duration, pattern, or the like. Thus, sound characteristics may be identified in three primary categories, including temporal composition (such as the timing of a knock event pattern), spectral composition (such as amplitude and frequency characteristics produced by the striking surface materials), and spectrotemporal composition (for enhanced signal discrimination to distinguish deliberate sound events from ambient noise).
[0007] A “knock” may include any audio signal produced by a guest that is intended to call the homeowner to the door. For example, a “knock” may include a rapping, slapping, pounding, or kicking on the door, doorframe, or windows or walls surrounding the door, or may further include clapping, ringing a bell, unlocking a door, turning or jiggling a doorknob, walking up steps, calling out or whistling, approaching in a car, or any other detectable audio signal. The audio signal may be detected by an audio and/or video monitoring or recording device positioned at one or more entry and exit point of the home. For example, a video monitoring device may be positioned above a front door in some embodiments, or in other embodiments a microphone may be integrated with a doorbell. In some embodiments, the audio detecting means may be continuously operational and “listening”; in other embodiments, the audio detecting means may be initiated based on occupancy detection data, such as detected motion, light, vibration, or the like.
[0008] Upon authenticating and identifying the guest, the home automation system may, in some embodiments, communicate the identity of the knocking guest to the homeowner, for example at a dedicated application on the homeowner's smartphone, or at a control panel. In some embodiments, the communicated guest's identity may be accompanied by a video feed or still photograph of the guest at the front door, for example recorded by a video monitor positioned at the front door. In the event that the guest cannot be identified based on the detected knock, the mere detection of the knock may nevertheless trigger a communication to the homeowner, accompanied in some cases by a picture or video of the guest and notifying the homeowner that the knocking guest is unidentified.
[0009] In other embodiments, upon authenticating and identifying the guest, the home automation system may compare the identity of the guest with inputted user preferences regarding home automation system operations to implement for that identified guest. For example, the user may input a preference indicating that, where the FedEx deliveryman is identified as knocking at the door, a recorded voice message should be broadcasted at the front door, telling the deliveryman to please leave the package on the bench by the front door. In other embodiments, the user may input a preference indicating that, where the user himself is identified by the unique audio associated with the sound of the user's car engine approaching the house, the garage door should be opened and the door leading from the garage to the house should be unlocked. In examples in which the knocking guest is unidentified, the user preferences with respect to an unidentified guest may be implemented, such as locking doors or initiating a video monitor. Any combination of identified guests or users and preferred home automation system operation instructions are envisioned.
[0010] In some embodiments, the home automation system may implement various system operations to suggest that the home is occupied upon detecting an attempted break-in. For example, interior lights may be turned on, stereos or televisions turned on, or the like, in response to the detection of glass break, excessive vibration or force against a door, excessive manipulation of a door lock, or the like.
[0011] Over time, the home automation system may adaptively learn user preferences in response to guests knocking, and may automatically derive home automation system operation instructions based on observed user patterns. For example, where an unidentified guest knocks on the door or produces some auditory signal, the home automation system may observe that the homeowner consistently inputs an instruction to broadcast a message to leave the package near the door in response to the received knock. Thus, the home automation system may infer that the knocking guest is a delivery person, and may accordingly derive an operation instruction corresponding with this guest such that, when the guest comes to the door in the future, the system may implement the operation instruction. In some embodiments, the home automation system may communicate a confirmation request to the user, for example at a dedicated application on the user's smartphone, indicating the proposed operation instruction and awaiting user confirmation before implementing the operation instruction. In other embodiments, the home automation system may communicate a notification to the user after the operation has been implemented.
[0012] In addition, the home automation system may adaptively learn over time to recognize guests based on detected patterns in their knocking styles. For example, the home automation system may detect that a particular guest consistently knocks on the door with an evenly spaced three-rap knuckle knock in the middle of the door. Using other identification techniques, such as voice recognition or facial recognition, the home automation system may identify the guest as Sally, and may add the sound clip and/or sound fingerprint of Sally's knock to the library of known audio signals. In some embodiments, the home automation system may adaptively learn to recognize guests based on detected temporal patterns, such as identifying that Sally visits the home at the same time every Tuesday to take her piano lesson. Thus, when Sally knocks in the future, the home automation system may communicate Sally's identity to the user notifying him of her presence at the door, or in addition or alternatively, may implement one or more home automation system operations corresponding with user inputted preferences or derived patterns specific to Sally, such as unlocking the door.
[0013] In some embodiments, the guest may be identified solely based on the detected sound event, whereas in other embodiments, the guest may be identified by any combination of a sound event, facial recognition, voice recognition, or the like. In this way, the likelihood of a false positive for a recognized guest may be reduced by corroborating the identification means.
[0014] In further embodiments, identified sound events may be associated with the homeowner(s) himself. For example, a sound event such as the sound of the homeowner's car engine as he pulls into the driveway, or the sound of the garage door opening, may be identified by one or more sensors. The home automation system may then compare the detected sound event with inputted user preferences regarding actions to be taken, such as unlocking the front door, turning on interior lights, disarming alarm systems, adjusting the thermostat, issuing a personalized auditory or visual greeting, or the like. In some embodiments, a homeowner may input a preference to silence alarms and auditory notifications when the home automation system detects that the homeowner has returned home after a certain hour at night, so as to avoid awakening other members of the household.
[0015] In some embodiments, particular sound events may be associated with a child returning home alone. For example, a child may use a particular input code, may have a discernible walk or knocking pattern, or may speak a predetermined auditory command, which may indicate to the home automation system that the child is returning home alone. Using a plurality of occupancy sensors positioned throughout the home, the home automation system may further identify that there are no other occupants in the home. A particular “home alone” home automation system operation profile may accordingly be implemented on the basis of determining that the child is home alone. For example, the home automation system may communicate an instruction to the security system to arm all point of entry sensors, while disarming interior motion sensors, and may further communicate an instruction to the exterior door locking mechanisms to lock all exterior doors. According to user-inputted preferences, the home automation system may further derive an instruction to disable doorbell sounds, and instead communicate any visitor identifications to the homeowners at their mobile devices, so as to avoid the child's attempting to answer the door. Other user inputted preferences associated with individual home occupants are also envisioned.
If you enjoyed this article and want to receive more valuable industry content like this, click here to sign up for our digital newsletters!