Michael P Rubinstein

US Patent:

20130202205, Aug 8, 2013

Filed:

Feb 6, 2012

Appl. No.:

13/367139

Inventors:

Ce Liu - Arlington MA, US
Michael Rubinstein - Somerville MA, US

Assignee:

Microsoft Corporation - Redmond WA

International Classification:

G06K 9/68
G06K 9/46

US Classification:

382170, 382195

Abstract:

Techniques for semantically annotating images in a plurality of images, each image in the plurality of images comprising at least one image region. The techniques include identifying at least two similar images including a first image and a second image, identifying corresponding image regions in the first image and the second image, and assigning, using at least one processor, annotations to image regions in one or more images in the plurality of images by using a metric of fit indicative of a degree of match between the assigned annotations and the corresponding image regions. The metric of fit may depend on at least one annotation for each image in a subset of the plurality of images and the identified correspondence between image regions in the first image and the second image.

US Patent:

20140072190, Mar 13, 2014

Filed:

Mar 26, 2013

Appl. No.:

13/850717

Inventors:

Michael Rubinstein - Somerville MA, US
Eugene Inghaw Shih - Brookline MA, US
John V. Guttag - Lexington MA, US
Frederic Durand - Somerville MA, US
William T. Freeman - Acton MA, US

Assignee:

Massachusetts Institute of Technology - Cambridge MA

International Classification:

G06T 7/00

US Classification:

382128

Abstract:

In one embodiment, a method of amplifying temporal variation in at least two images comprises examining pixel values of the at least two images. The temporal variation of the pixel values between the at least two images can be below a particular threshold. The method can further include applying signal processing to the pixel values.

US Patent:

20140072228, Mar 13, 2014

Filed:

Sep 7, 2012

Appl. No.:

13/607173

Inventors:

Michael Rubinstein - Somerville MA, US
Neal Wadhwa - Cambridge MA, US
Fredo Durand - Boston MA, US
William T. Freeman - Acton MA, US

Assignee:

Massachusetts Institute of Technology - Cambridge MA

International Classification:

G06K 9/48

US Classification:

382197

Abstract:

In one embodiment, a method of amplifying temporal variation in at least two images includes converting two or more images to a transform representation. The method further includes, for each spatial position within the two or more images, examining a plurality of coefficient values. The method additionally includes calculating a first vector based on the plurality of coefficient values. The first vector can represent change from a first image to a second image of the at least two images describing deformation. The method also includes modifying the first vector to create a second vector. The method further includes calculating a second plurality of coefficients based on the second vector.

US Patent:

20140072229, Mar 13, 2014

Filed:

Dec 6, 2012

Appl. No.:

13/707451

Inventors:

Michael Rubinstein - Somerville MA, US
Frederic Durand - Boston MA, US
William T. Freeman - Acton MA, US
Hao-Yu Wu - Cambridge MA, US
Eugen Inghaw Shih - Brookline MA, US
John V. Guttag - Lexington MA, US

Assignee:

Massachusetts Institute of Technology - Cambridge MA

International Classification:

G06K 9/48

US Classification:

382197

Abstract:

In one embodiment, a method of amplifying temporal variation in at least two images includes converting two or more images to a transform representation. The method further includes, for each spatial position within the two or more images, examining a plurality of coefficient values. The method additionally includes calculating a first vector based on the plurality of coefficient values. The first vector can represent change from a first image to a second image of the at least two images describing deformation. The method also includes modifying the first vector to create a second vector. The method further includes calculating a second plurality of coefficients based on the second vector.

US Patent:

20200335121, Oct 22, 2020

Filed:

Nov 21, 2018

Appl. No.:

16/761707

Inventors:

- Mountain View CA, US
Michael Rubinstein - Natick MA, US
Ariel Ephrat - Efrat, IL
William Freeman - Acton MA, US
Oran Lang - Givatayim, IL
Kevin William Wilson - Cambridge MA, US
Tali Dekel - Arlington MA, US
Avinatan Hassidim - Petah Tikva, IL

International Classification:

G10L 21/10
G10L 21/18
G10L 15/16
G06K 9/00
G06K 9/62

Abstract:

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for audio-visual speech separation. A method includes: obtaining, for each frame in a stream of frames from a video in which faces of one or more speakers have been detected, a respective per-frame face embedding of the face of each speaker; processing, for each speaker, the per-frame face embeddings of the face of the speaker to generate visual features for the face of the speaker; obtaining a spectrogram of an audio soundtrack for the video; processing the spectrogram to generate an audio embedding for the audio soundtrack; combining the visual features for the one or more speakers and the audio embedding for the audio soundtrack to generate an audio-visual embedding for the video; determining a respective spectrogram mask for each of the one or more speakers; and determining a respective isolated speech spectrogram for each speaker.

US Patent:

20180096482, Apr 5, 2018

Filed:

Nov 21, 2017

Appl. No.:

15/819791

Inventors:

- Cambridge MA, US
Frederic Durand - Somerville MA, US
Tianfan Xue - Cambridge MA, US
Michael Rubinstein - Somerville MA, US
Neal Wadhwa - Mountain View CA, US

International Classification:

G06T 7/20
G02B 27/54
G01P 5/00

Abstract:

An apparatus according to an embodiment of the present invention enables measurement and visualization of a refractive field such as a fluid. An embodiment device obtains video captured by a video camera with an imaging plane. Representations of apparent motions in the video are correlated to determine actual motions of the refractive field. A textured background of the scene can be modeled as stationary, with a refractive field translating between background and video camera. This approach offers multiple advantages over conventional fluid flow visualization, including an ability to use ordinary video equipment outside a laboratory without particle injection. Even natural backgrounds can be used, and fluid motion can be distinguished from refraction changes. Embodiments can render refractive flow visualizations for augmented reality, wearable devices, and video microscopes.

US Patent:

20180047160, Feb 15, 2018

Filed:

Sep 29, 2017

Appl. No.:

15/721375

Inventors:

- Cambridge MA, US
Michael Rubinstein - Somerville MA, US
Eugene Inghaw Shih - Brookline MA, US
John V. Guttag - Lexington MA, US
Frederic Durand - Somerville MA, US
William T. Freeman - Acton MA, US
Neal Wadhwa - Mountain View CA, US

Assignee:

Quanta Computer, Inc. - Kuei Shan Hsiang

International Classification:

G06T 7/00

Abstract:

In an embodiment, a method converts two images to a transform representation in a transform domain. For each spatial position, the method examines coefficients representing a neighborhood of the spatial position that is spatially the same across each of the two images. The method calculates a first vector in the transform domain based on first coefficients representing the spatial position, the first vector representing change from a first to second image of the two images describing deformation. The method modifies the first vector to create a second vector in the transform domain representing amplified movement at the spatial position between the first and second images. The method calculates second coefficients based on the second vector of the transform domain. From the second coefficients, the method generates an output image showing motion amplified according to the second vector for each spatial position between the first and second images.

US Patent:

20150195430, Jul 9, 2015

Filed:

Jan 8, 2015

Appl. No.:

14/592211

Inventors:

- Cambridge MA, US
Michael Rubinstein - Newton MA, US
Frederic Durand - Somerville MA, US
William T. Freeman - Acton MA, US

International Classification:

H04N 5/14
H04N 5/213
H04N 5/20

Abstract:

Some embodiments are directed to a method, corresponding system, and corresponding apparatus for rendering a video and/or image display to amplify small motions through video magnification. Some embodiments include a new compact image pyramid representation, the Riesz pyramid, that may be used for real-time, high-quality phase-based video magnification. Some embodiments are less overcomplete than even the smallest two orientation, octave-bandwidth complex steerable pyramid. Some embodiments are implemented using compact, efficient linear filters in the spatial domain. Some embodiments produce motion magnified videos that are of comparable quality to those using the complex steerable pyramid. In some embodiments, the Riesz pyramid is used with phase-based video magnification. The Riesz pyramid may phase-shift image features along their dominant orientation, rather than along every orientation like the complex steerable pyramid.