WO2022052997 - METHOD AND SYSTEM FOR TRAINING NEURAL NETWORK MODEL USING KNOWLEDGE DISTILLATION
National phase entry:
Publication Number
WO/2022/052997
Publication Date
17.03.2022
International Application No.
PCT/CN2021/117532
International Filing Date
09.09.2021
Title **
[English]
METHOD AND SYSTEM FOR TRAINING NEURAL NETWORK MODEL USING KNOWLEDGE DISTILLATION
[French]
PROCÉDÉ ET SYSTÈME D'ENTRAÎNEMENT DE MODÈLE DE RÉSEAU NEURONAL UTILISANT LA DIFFUSION DE CONNAISSANCES
Applicants **
HUAWEI TECHNOLOGIES CO.,LTD.
Huawei Administration Building
Bantian, Longgang District
Shenzhen, Guangdong 518129, CN
Inventors
PASSBAN, Peyman
Apt.217
55 rue Molière
Montreal, Québec H2R 1N7, CA
WU, Yimeng
401
3454 Aylmer Street
Montreal, Québec H2X 2B6, CA
REZAGHOLIZADEH, Mehdi
6225 Place Northcrest
Apt. 5
Montreal, Québec H3S 2T5, CA
Priority Data
63/076,335
09.09.2020
US
17/469,573
08.09.2021
US
Application details
| Total Number of Claims/PCT | * |
| Number of Independent Claims | * |
| Number of Priorities | * |
| Number of Multi-Dependent Claims | * |
| Number of Drawings | * |
| Pages for Publication | * |
| Number of Pages with Drawings | * |
| Pages of Specification | * |
| * | |
| * | |
International Searching Authority |
CNIPA
* |
| Applicant's Legal Status |
Legal Entity
* |
| * | |
| * | |
| * | |
| * | |
| Entry into National Phase under |
Chapter I
* |
| Translation |
|
Recalculate
* The data is based on automatic recognition. Please verify and amend if necessary.
** IP-Coster compiles data from publicly available sources. If this data includes your personal information, you can contact us to request its removal.
Quotation for National Phase entry
| Country | Stages | Total | |
|---|---|---|---|
| China | Filing | 1535 | |
| EPO | Filing, Examination | 9106 | |
| Japan | Filing | 590 | |
| South Korea | Filing | 607 | |
| USA | Filing, Examination | 3710 |
Total: 15548 USD
The term for entry into the National Phase has expired. This quotation is for informational purposes only
Abstract[English]
An agnostic combinatorial knowledge distillation (CKD) method for transferring trained knowledge of neural model from a complex model (teacher) to a less complex model (student) is described. In addition to training the student to generate a final output that approximates both the teacher's final output and a ground truth of a training input, the method further maximizes knowledge transfer by training hidden layers of the student to generate outputs that approximate a representation of a subset of teacher hidden layers are mapped to each of the student hidden layers for a given training input.[French]
Un procédé de diffusion de connaissances combinatoire agnostique (CKD) pour transférer des connaissances entraînées de modèle neuronal entre un modèle complexe (enseignant) et un modèle moins complexe (étudiant) est décrit. En plus de l'apprentissage de l'étudiant pour générer une sortie finale qui s'approche à la fois de la sortie finale de l'enseignant et d'une réalité de terrain d'une entrée d'apprentissage, le procédé maximise en outre le transfert de connaissances en entraînant des couches cachées de l'étudiant afin de générer des sorties qui se rapprochent d'une représentation d'un sous-ensemble de couches cachées de l'enseignant mises en correspondance avec chacune des couches cachées de l'étudiant pour une entrée d'apprentissage donnée.