Hao Cheng

US Patent:

20210326751, Oct 21, 2021

Filed:

May 22, 2020

Appl. No.:

16/882296

Inventors:

- Redmond WA, US
Hao Cheng - Bellevue WA, US
Yu Wang - Bellevue WA, US
Jianfeng Gao - Woodinville WA, US
Weizhu Chen - Kirkland WA, US
Pengcheng He - Sammamish WA, US
Hoifung Poon - Bellevue WA, US

Assignee:

Microsoft Technology Licensing, LLC - Redmond WA

International Classification:

G06N 20/00
G06N 3/08
G06F 40/284
G06K 9/62

Abstract:

This document relates to training of machine learning models. One example method involves providing a machine learning model having one or more mapping layers. The one or more mapping layers can include at least a first mapping layer configured to map components of pretraining examples into first representations in a space. The example method also includes performing a pretraining stage on the one or more mapping layers using the pretraining examples. The pretraining stage can include adding noise to the first representations of the components of the pretraining examples to obtain noise-adjusted first representations. The pretraining stage can also include performing a self-supervised learning process to pretrain the one or more mapping layers using at least the first representations of the training data items and the noise-adjusted first representations of the training data items.