公开(公告)号：GB2586559A

公开(公告)日：2021-02-24

申请号：GB202019054

申请日：2019-05-30

Applicant: IBM

Inventor： SILVIA MELITTA MUELLER ,  ANKUR AGRAWAL ,  BRUCE FLEISCHER ,  KAILASH GOPALAKRISHNAN ,  DONGSOO LEE

IPC: G06F7/483 ,  G06F7/499

Abstract: Techniques for operating on and calculating binary floating-point numbers using an enhanced floating-point number format are presented. The enhanced format can comprise a single sign bit, six bits for the exponent, and nine bits for the fraction. Using six bits for the exponent can provide an enhanced exponent range that facilitates desirably fast convergence of computing-intensive algorithms and low error rates for computing-intensive applications. The enhanced format can employ a specified definition for the lowest binade that enables the lowest binade to be used for zero and normal numbers; and a specified definition for the highest binade that enables it to be structured to have one data point used for a merged Not-a-Number (NaN)/infinity symbol and remaining data points used for finite numbers. The signs of zero and merged NaN/infinity can be "don't care" terms. The enhanced format employs only one rounding mode, which is for rounding toward nearest up.

公开(公告)号：GB2586559B

公开(公告)日：2021-07-14

申请号：GB202019054

申请日：2019-05-30

Applicant: IBM

Inventor： SILVIA MELITTA MUELLER ,  ANKUR AGRAWAL ,  BRUCE FLEISCHER ,  KAILASH GOPALAKRISHNAN ,  DONGSOO LEE

IPC: G06F7/483 ,  G06F7/499

Abstract: Techniques for operating on and calculating binary floating-point numbers using an enhanced floating-point number format are presented. The enhanced format can comprise a single sign bit, six bits for the exponent, and nine bits for the fraction. Using six bits for the exponent can provide an enhanced exponent range that facilitates desirably fast convergence of computing-intensive algorithms and low error rates for computing-intensive applications. The enhanced format can employ a specified definition for the lowest binade that enables the lowest binade to be used for zero and normal numbers; and a specified definition for the highest binade that enables it to be structured to have one data point used for a merged Not-a-Number (NaN)/infinity symbol and remaining data points used for finite numbers. The signs of zero and merged NaN/infinity can be “don't care” terms. The enhanced format employs only one rounding mode, which is for rounding toward nearest up.

公开(公告)号：GB2582232A

公开(公告)日：2020-09-16

申请号：GB202009717

申请日：2018-11-30

Applicant: IBM

Inventor： CHIA-YU CHEN ,  ANKUR AGRAWAL ,  DANIEL BRAND ,  KAILASH GOPALAKRISHNAN ,  JUNGWOOK CHOI

IPC: G06N3/08 ,  G06N3/04

Abstract: Embodiments of the present invention provide a computer-implemented method for adaptive residual gradient compression for training of a deep learning neural network (DNN). The method includes obtaining, by a first learner, a current gradient vector for a neural network layer of the DNN, in which the current gradient vector includes gradient weights of parameters of the neural network layer that are calculated from a mini-batch of training data. A current residue vector is generated that includes residual gradient weights for the mini-batch. A compressed current residue vector is generated based on dividing the residual gradient weights of the current residue vector into a plurality of bins of a uniform size and quantizing a subset of the residual gradient weights of one or more bins of the plurality of bins. The compressed current residue vector is then transmitted to a second learner of the plurality of learners or to a parameter server.