<section data-tools="135ç¼è¾å¨" data-id="98856" data-mpa-powered-by="yiban.io"><section><section style="margin: 10px auto;text-align: center;"><section style="display: flex;flex-direction: column;justify-content: center;align-items: center;"><section data-tools="135ç¼è¾å¨" data-id="95952" style="text-align: justify;"><section style="margin-right: auto;margin-left: auto;"><section style="padding: 2px 10px;display: flex;justify-content: flex-end;"><section style="margin-right: 10px;width: 50px;height: 50px;border-width: 2px;border-style: solid;border-color: rgb(0, 0, 0);border-radius: 100%;background-image: url("https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZ1F7icKAWGqxkM3zy2Dzmia9UwuBJYpMp9dP4XgNlXTCFiazruuTYOU0cw/640?wx_fmt=jpeg");background-position: center center;background-size: cover;background-repeat: no-repeat;background-attachment: initial;background-origin: initial;background-clip: initial;"><img data-ratio="1" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZ1F7icKAWGqxkM3zy2Dzmia9UwuBJYpMp9dP4XgNlXTCFiazruuTYOU0cw/640?wx_fmt=jpeg" data-type="jpeg" data-w="750" data-width="100%" style="width: 100%;border-radius: 50%;opacity: 0;" title="" /></section></section><section style="margin: 5px;border-width: initial;border-style: none;border-color: initial;"><section style="margin-left: 454.391px;width: 0px;border-bottom: 0.6em solid rgb(0, 0, 0);border-top-color: rgb(0, 0, 0);height: 10px;color: inherit;overflow: hidden;border-left: 0.7em solid transparent !important;border-right: 0.7em solid transparent !important;"><br /></section><section style="margin-top: -8px;margin-left: 454.391px;width: 0px;border-bottom: 0.6em solid rgb(254, 254, 254);border-top-color: rgb(254, 254, 254);height: 10px;color: inherit;float: left;overflow: hidden;border-left: 0.7em solid transparent !important;border-right: 0.7em solid transparent !important;"><br /></section><section style="margin-right: auto;margin-bottom: -2px;margin-left: auto;text-align: center;border-width: 2px;border-style: solid;border-color: rgb(0, 0, 0);border-radius: 5px;"><section data-autoskip="1" style="padding: 1em;font-size: 14px;text-align: justify;letter-spacing: 1.5px;line-height: 1.75em;color: rgb(62, 62, 62);"><p hm_fix="374:371" style="text-align: left;"><span style="font-family: 微软é é», "Microsoft YaHei";"><span style="color: rgb(0, 0, 0);text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 0, 0);font-size: 15px;outline: none 0px !important;">å¸é¢ä¸æ²¡çå°</span><strong style="color: rgb(0, 0, 0);text-align: justify;max-inline-size: 100%;cursor: text;font-size: medium;caret-color: rgb(0, 0, 0);font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">å¾æç»åçhuggingface transformersæ¡æ¶bertæºç 解说</span></strong><span style="color: rgb(0, 0, 0);text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 0, 0);font-size: 15px;outline: none 0px !important;">ï¼ç¬è ä¹å¨çæºç ï¼æ以顺æèªå·±å个ç¬è®°ã</span></span></p></section></section></section></section></section><section data-tools="135ç¼è¾å¨" data-id="101126"><section style="margin: 10px auto;text-align: center;"><section style="background: rgb(0, 0, 0);padding: 1em;margin-left: 8px;margin-right: 8px;"><section style="display: flex;justify-content: space-between;align-items: center;" hm_fix="353:417"><section data-role="outer" label="Powered by 135editor.com"><section data-tools="135ç¼è¾å¨" data-id="101126"><section style="margin: 10px auto;text-align: center;"><section style="background: #000;padding: 1em;"><section style="display: flex;justify-content: space-between;align-items: center;" hm_fix="353:417"><section style="width: 30px;"><img data-ratio="0.6585365853658537" data-src="https://mmbiz.qpic.cn/mmbiz_png/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZv49X4QS9RpUCDgXXicZZhS2N52yzyzOkEMYVkVicIibwEsPE4XELRHia2Q/640?wx_fmt=png" data-type="png" data-w="41" data-width="100%" style="width: 100%;display: block;" /></section><section data-brushtype="text" style="font-size: 16px;letter-spacing: 1.5px;padding: 4px 1em;color: #fff;font-weight: bold;background: #007aaa;"><p style="line-height: 1.75em;margin-left: 8px;margin-right: 8px;"><span style="font-size: 17px;">åºç¨èå´</span></p></section></section><section data-autoskip="1" style="text-align: justify;line-height: 1.75em;letter-spacing: 1.5px;font-size: 14px;color: rgb(255, 255, 255);margin-top: 2em;margin-bottom: 1em;"><p style="line-height: 1.75em;margin-left: 8px;margin-right: 8px;"><span style="font-size: 15px;">BERTå¯åºç¨å¨æç´¢å¼æï¼æºè½å®¢æï¼è天æºå¨äººï¼è¯é³å©æï¼æ¨èç³»ç»ï¼ç¥è¯å¾è°±çNLPä¸æ¸¸ä»»å¡å¦ï¼é 读ç解ï¼ç¥è¯æ½åï¼ææ¬åç±»ï¼è¯è¨çæ</span></p></section></section></section></section></section></section></section><section data-autoskip="1" style="text-align: justify;line-height: 1.75em;letter-spacing: 1.5px;font-size: 14px;color: rgb(255, 255, 255);margin-top: 2em;margin-bottom: 1em;"><section style="line-height: 1.75em;margin-left: 8px;margin-right: 8px;"><br /></section></section></section></section><p style="font-weight: bold;color: rgb(0, 122, 170);font-size: 60px;"><span style="font-family: 微软é é», "Microsoft YaHei";">1</span></p><section data-width="15%" style="margin-top: -35px;width: 15%;height: 1px;background: rgb(0, 122, 170);overflow: hidden;"><br /></section></section><section data-brushtype="text" hm_fix="318:337" style="padding: 8px;font-size: 16px;font-weight: bold;letter-spacing: 1.5px;background: rgb(255, 255, 255);transform: rotate(0deg);"><span style="font-family: 微软é é», "Microsoft YaHei";">æç« å 容</span></section></section></section></section><section data-role="paragraph"><section style="margin-right: 8px;margin-left: 8px;max-inline-size: 100%;cursor: text;line-height: 1.75em;outline: none 0px !important;"><span style="font-family: 微软é é», "Microsoft YaHei";"><strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: 15px;outline: none 0px !important;">ä»å®è§å°å¾®è§è®²è§£huggingface transformersæ¡æ¶çbertæºç </span></strong><span style="max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: 15px;">ï¼é¨åå·¥ç¨ç»èå¦éä½è®ç»å å使ç¨çutilï¼ä¸äºåæ°å¯é项é½ä¸ä¼æ¶åï¼åªè®²ä¸»å¹²ã</span></span></section></section><section data-tools="135ç¼è¾å¨" data-id="98856"><section><section style="margin: 10px auto;text-align: center;"><section style="display: flex;flex-direction: column;justify-content: center;align-items: center;"><section data-original-title="" title="" data-num="2" style="font-weight: bold;color: rgb(0, 122, 170);font-size: 60px;"><span style="font-family: 微软é é», "Microsoft YaHei";">2</span></section><section data-width="15%" style="margin-top: -35px;width: 15%;height: 1px;background: rgb(0, 122, 170);overflow: hidden;"><br /></section></section><section data-brushtype="text" hm_fix="341:328" style="padding: 8px;font-size: 16px;font-weight: bold;letter-spacing: 1.5px;background: rgb(255, 255, 255);transform: rotate(0deg);"><span style="font-family: 微软é é», "Microsoft YaHei";">BERTæ¦è¦</span></section></section></section></section><section data-role="paragraph"><section style="margin: 1.4em 8px;caret-color: rgb(255, 0, 0);max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="font-family: 微软é é», "Microsoft YaHei";"><span style="max-inline-size: 100%;cursor: text;color: rgb(0, 122, 170);"><strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;outline: none 0px !important;">BERTæ¯Googleå¨2018å¹´10ææåºæ¥çä¸ä¸ªææ¬è¯è¨é¢è®ç»æ¹æ³</span></strong></span><span style="max-inline-size: 100%;cursor: text;font-size: 15px;">ï¼ä»æ¯åºäºTransformerçå®ç°ãå¨bertä¹åè¿æ诸å¦GlovãWord2vecãULMFiTãElmoãGPTçé¢è®ç»æ¹æ³ï¼ä½æ¯bertå¨åç§ä»»å¡ä¸ç表ç°æçæ¾èæåï¼å¾®è°å³å¯å¨ä¸äºä»»å¡ä¸è½»æ¾è¾¾å°state of the artï¼å¯¹å·¥ä¸çNLPè½å°æçæ大çæ¨å¨ä½ç¨ã</span></span></section><section style="margin: 1.4em 8px;caret-color: rgb(255, 0, 0);max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">深度å¦ä¹ é常éè¦é¢è®ç»æ¨¡åãæäºå¥½çé¢è®ç»æ¨¡åï¼å¤§å®¶å¯ä»¥æ´ä¸æ³¨äºèªå·±çå®éªï¼èä¸æ¯å å»å¾å°è¯ä¹è¡¨å¾åå»åä¸æ¸¸ä»»å¡ã</span></section><section style="margin: 1.4em 8px;caret-color: rgb(255, 0, 0);max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">åºè¯å¤ªå¤äºâ¦â¦å¼æ</span></section><section data-tools="135ç¼è¾å¨" data-id="98856"><section><section style="margin: 10px auto;text-align: center;"><section hm_fix="339:315" style="display: flex;flex-direction: column;justify-content: center;align-items: center;"><section data-original-title="" title="" data-num="3" style="font-weight: bold;color: rgb(0, 122, 170);font-size: 60px;"><span style="font-family: 微软é é», "Microsoft YaHei";">3</span></section><section data-width="15%" style="margin-top: -35px;width: 15%;height: 1px;background: rgb(0, 122, 170);overflow: hidden;"><br /></section></section><section data-brushtype="text" style="padding: 8px;font-size: 16px;font-weight: bold;letter-spacing: 1.5px;background: rgb(255, 255, 255);transform: rotate(0deg);"><span style="font-family: 微软é é», "Microsoft YaHei";">BERTçå ¨è²</span></section></section></section></section><figure data-size="small" style="margin-top: 1.4em;margin-bottom: 1.4em;max-inline-size: 100%;clear: both;color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-backh="777" data-backw="578" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZjnEWIdoogDibQuJepFyiaxVER8CJ6FicZIStYqFq4TLkYKaPNsMsw7zdA/640?wx_fmt=jpeg" data-type="jpeg" width="1303" style="margin-right: auto;margin-left: auto;max-inline-size: 100%;width: 578px;display: block;cursor: zoom-in;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="1.3444444444444446" data-w="720" /><section style="margin-top: 0.66667em;margin-right: 8px;margin-left: 8px;padding-right: 1em;padding-left: 1em;text-align: center;max-inline-size: 100%;cursor: text;font-size: 0.9em;color: rgb(153, 153, 153);line-height: 1.75em;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 12px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">transformerçencoderï¼å¾çæ¥èªtransformeråæ</span></section></figure><section style="margin: 1.4em 8px;max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="font-family: 微软é é», "Microsoft YaHei";"><strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;outline: none 0px !important;">红è²æ¹æ¡å°±æ¯bertç主è¦æ¶æ</span></strong><span style="max-inline-size: 100%;cursor: text;font-size: 15px;">ï¼æ们çè¾å ¥å°±æ¯å¥åï¼è¾åºçæ¯è¿ä¸ªå¥å对åºçè¯ä¹è¡¨ç¤ºçåéã对äºä¸ä¸ªææ¬å¥åï¼æ们éè¦æå¥ååè¯ï¼ç¶åå转æ¢æåéè¾å ¥å°bertï¼bertçåè¯åºæ¬åçæ¯byte pair encoding(BPE)ï¼å¾å°çtokenå«wordpieceï¼æ¯ä¸ªtoken对åºä¸ä¸ªidï¼ä¸ä¸ªå¥åæ¯ä¸ä¸ªäºç»´ç©éµ(vocab_size,sentence_len)ã</span></span></section><section style="margin: 1.4em 8px;padding-left: 1em;max-inline-size: 100%;cursor: text;color: rgb(100, 100, 100);border-left: 3px solid rgb(211, 211, 211);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 12px;color: rgb(127, 127, 127);font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">Input:å¥åçwordpieceçid ï¼Output:å¥åçè¯ä¹è¡¨å¾åé</span></section><section style="margin: 1.4em 8px;max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="font-family: 微软é é», "Microsoft YaHei";"><strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;outline: none 0px !important;">ä¸é¢ç代ç çæ¶æã</span></strong></span></section><figure data-size="normal" style="margin-top: 1.4em;margin-bottom: 1.4em;max-inline-size: 100%;color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-backh="145" data-backw="578" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZc2Jmyeh5VMFa5yGtrDalutyCKnFOicJyFyzgxF5xyBl3PrtMTRGyaVA/640?wx_fmt=jpeg" data-type="jpeg" width="3797" style="margin-right: auto;margin-left: auto;max-inline-size: 100%;width: 100%;display: block;cursor: zoom-in;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="0.2513888888888889" data-w="720" /></figure><section style="margin: 1.4em 8px;max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">åå§åæä¸ä¸ªåï¼<strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;">BertEmbeddings</strong>ï¼<strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;">BertEncoder</strong>ï¼<strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;">BertPooler</strong>ï¼åéä¹æ¯ä¾æ¬¡å¾ä¸ä¼ éã</span></section><section style="margin-top: 1em;margin-bottom: 1em;max-inline-size: 100%;color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><pre style="padding: 0.88889em;overflow-wrap: normal;max-inline-size: 100%;font-size: 0.9em;word-break: normal;overflow: auto;background: rgb(246, 246, 246);border-radius: 4px;outline: none 0px !important;"><section style="margin-right: 8px;margin-left: 8px;max-inline-size: 100%;cursor: text;line-height: 1.75em;outline: none 0px !important;"><code style="max-inline-size: 100%;border-radius: 0px;font-size: inherit;background-color: inherit;font-family: Menlo, Monaco, Consolas, "Andale Mono", "lucida console", "Courier New", monospace;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 12px;color: rgb(127, 127, 127);font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">class BertModel(BertPreTrainedModel): def __init__(self, config): super().__init__(config) self.config = config self.embeddings = BertEmbeddings(config) self.encoder = BertEncoder(config) self.pooler = BertPooler(config) self.init_weights()</span></code></section></pre></section><section style="margin: 1.4em 8px;max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">对åº</span></section><figure data-size="small" style="margin-top: 1.4em;margin-bottom: 1.4em;max-inline-size: 100%;clear: both;color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-backh="210" data-backw="186" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZHSvB7P35sicSAcpZ8NKBvCSREYaTWVCu0z4bKAthOczBCQXu6icrRVDQ/640?wx_fmt=jpeg" data-type="jpeg" width="1011" style="margin-right: auto;margin-left: auto;max-inline-size: 100%;width: 100%;display: block;cursor: zoom-in;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="1.1291666666666667" data-w="720" /></figure><section style="margin: 1.4em 8px;max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">BertPooleråæ¯åäºCLS tokençhidden statesåtanhï¼ä¸é¢æ们ä¸ä¸ªä¸ä¸ªè®²è§£ã</span></section><figure data-size="small" style="margin-top: 1.4em;margin-bottom: 1.4em;max-inline-size: 100%;clear: both;color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-backh="76" data-backw="186" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZCFA2ZGfRicibplia2oIUq8NuglT1KBWtxEoYREpiaXQwNzdCk41BJvgjZg/640?wx_fmt=jpeg" data-type="jpeg" height="76" width="186" style="margin-right: auto;margin-left: auto;max-inline-size: 100%;width: 578px;display: block;cursor: zoom-in;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="0.4111111111111111" data-w="720" /><section style="margin-top: 0.66667em;margin-right: 8px;margin-left: 8px;padding-right: 1em;padding-left: 1em;text-align: center;max-inline-size: 100%;cursor: text;font-size: 0.9em;color: rgb(153, 153, 153);line-height: 1.75em;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 12px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">pooler</span></section></figure></section><section data-tools="135ç¼è¾å¨" data-id="101124"><section style="margin: 10px auto;text-align: center;"><section style="padding: 1em;border-width: 1px;border-style: solid;border-color: rgb(0, 122, 170);"><section style="display: inline-block;"><section data-brushtype="text" style="font-size: 16px;letter-spacing: 1.5px;padding: 6px 1.5em;color: #fff;font-weight: bold;background: #007aaa;border-radius: 30px;"><span style="font-family: 微软é é», "Microsoft YaHei";"><strong style="caret-color: rgb(0, 122, 170);font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;"><span style="font-size: 17px;font-family: 微软é é», "Microsoft YaHei";">BertEmbeddings</span></strong></span></section></section><section style="margin-right: 1em;margin-left: 1em;"><section data-autoskip="1" style="text-align: justify;line-height: 1.75em;letter-spacing: 1.5px;font-size: 14px;margin-top: 1.5em;"><section style="margin-top: 1em;margin-bottom: 1em;color: rgb(18, 18, 18);font-size: medium;background-color: rgb(255, 255, 255);font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;"><pre style="padding: 0.88889em;font-size: 0.9em;word-break: normal;overflow-wrap: normal;overflow: auto;background: rgb(246, 246, 246);border-radius: 4px;"><code style="border-radius: 0px;font-size: inherit;background-color: inherit;font-family: Menlo, Monaco, Consolas, "Andale Mono", "lucida console", "Courier New", monospace;"><span style="color: rgb(127, 127, 127);font-size: 12px;font-family: 微软é é», "Microsoft YaHei";">class BertEmbeddings(nn.Module):<br /> """Construct the embeddings from word, position and token_type embeddings.<br /> """<br /> def __init__(self, config):<br /> super().__init__()<br /> self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)<br /> self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)<br /> self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)<br /><br /> self.LayerNorm = BertLayerNorm(config.hidden_size, eps=config.layer_norm_eps)<br /> self.dropout = nn.Dropout(config.hidden_dropout_prob)<br /><br /> self.register_buffer("position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)))<br /><br /> def forward(self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None):<br /> #ä¸ä¸ªembedding使ç¨äºå æ³<br /> embeddings = inputs_embeds + position_embeddings + token_type_embeddings<br /> embeddings = self.LayerNorm(embeddings)<br /> embeddings = self.dropout(embeddings)<br /> return embeddings</span></code></pre></section><figure data-size="small" style="margin-top: 1.4em;margin-bottom: 1.4em;clear: both;color: rgb(18, 18, 18);font-size: medium;background-color: rgb(255, 255, 255);font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;"><figure data-size="small" style="margin-top: 1.4em;margin-bottom: 1.4em;clear: both;"><img data-ratio="1.4333333333333333" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZLDic8gywvzxQiaLncjBfNGxDYpbMnIJ32tiaYiboCh0PUzkLH9M6OyCNDQ/640?wx_fmt=jpeg" data-type="jpeg" data-w="720" style="margin-right: auto;margin-left: auto;width: 100%;display: block;cursor: zoom-in;background-color: transparent;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;" width="952" /></figure><figure data-size="normal" style="margin-top: 2.24em;margin-bottom: 1.4em;"><img data-ratio="0.3486111111111111" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZl9J889Ql7icsHDskfvppDPiaOTY6FOMjPTufHQKrzYyhdx9vJVkPBUFQ/640?wx_fmt=jpeg" data-type="jpeg" data-w="720" style="margin-right: auto;margin-left: auto;width: 100%;display: block;cursor: zoom-in;background-color: transparent;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;" width="2532" /></figure><section style="margin: 1.4em 8px;caret-color: rgb(0, 122, 170);max-inline-size: 100%;cursor: text;line-height: 1.75em;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">ä»ä¸å¾å¯ä»¥çå°ï¼embeddingç主è¦å 个embeddingï¼<strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;">为ä»ä¹éè¦è¿äºembeddingï¼ä¸ºä»ä¹è¦ç¨å æ³ï¼</strong></span></section><section style="margin: 1.4em 8px;caret-color: rgb(0, 122, 170);max-inline-size: 100%;cursor: text;line-height: 1.75em;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">ä»ä»£ç å¯ä»¥çå°è¿éç¨äºä¸ä¸ªembeddingå±(transformeråæçposition encodingç¨çæ¯æ£ä½å¼¦å½æ°)ï¼æåä¸ä¸ªç¸å ã</span></section></figure><section style="margin-top: 1em;margin-bottom: 1em;color: rgb(18, 18, 18);font-size: medium;background-color: rgb(255, 255, 255);font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;"><pre style="padding: 0.88889em;font-size: 0.9em;word-break: normal;overflow-wrap: normal;overflow: auto;background: rgb(246, 246, 246);border-radius: 4px;"><code style="border-radius: 0px;font-size: inherit;background-color: inherit;font-family: Menlo, Monaco, Consolas, "Andale Mono", "lucida console", "Courier New", monospace;"><span style="font-size: 12px;color: rgb(127, 127, 127);font-family: 微软é é», "Microsoft YaHei";">def forward(self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None):<br /> #ä¸ä¸ªembedding使ç¨äºå æ³<br /> embeddings = inputs_embeds + position_embeddings + token_type_embeddings<br /> embeddings = self.LayerNorm(embeddings)<br /> embeddings = self.dropout(embeddings)<br /> return embeddings</span></code></pre></section><section style="margin: 1.4em 8px;background-color: rgb(255, 255, 255);caret-color: rgb(0, 122, 170);max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="font-family: 微软é é», "Microsoft YaHei";"><strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;outline: none 0px !important;">ç</span></strong><span style="max-inline-size: 100%;cursor: text;font-size: 15px;">ï¼å¨encoderä¹åï¼æ们éè¦ç¨ä¸äºä¿¡å·æ¥å¢å inputæºå¸¦çä¿¡æ¯ï¼å ³äºä¸ºä»ä¹ç¨å æ³è¿ä¸ªåçå¾ä¸éï¼æ¦æ¬å°±æ¯<strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;">ä»å£°æ³¢é¢ççä¸åèèè¿ä¸ªé®é¢</strong>ï¼å å çè¿äºä¿¡æ¯ï¼æ¨¡åå¯ä»¥æ ¹æ®ç®æ é®é¢ç¨ä¸åç滤波æ¹æ³å¯¹ä¿¡å·æ»¤æ³¢ï¼æ¥è·å¾æ³è¦çæè æ¾æ§çç¹å¾ã</span></span></section><section style="margin: 1.4em 8px;background-color: rgb(255, 255, 255);caret-color: rgb(0, 122, 170);max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="font-family: 微软é é», "Microsoft YaHei";"><strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;outline: none 0px !important;">ä¸ä¸ª embedding æºå¸¦äºä»ä¹ä¿¡å·å¢ï¼</span></strong></span></section><section style="margin: 1.4em 8px;background-color: rgb(255, 255, 255);caret-color: rgb(0, 122, 170);max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;color: rgb(0, 122, 170);font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;"><strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;outline: none 0px !important;">token embeddingï¼æ¯ä¸ªtokençè¯è¡¨ç¤ºä¿¡æ¯<br style="max-inline-size: 100%;outline: none 0px !important;" />segment embeddingï¼åºåå¥åï¼bertçNSP<br style="max-inline-size: 100%;outline: none 0px !important;" />position embedding ï¼åºåæ¯ä¸ªè¯çå¨å¥åçä½ç½®</span></strong></span></section><p style="margin-right: 8px;margin-left: 8px;background-color: rgb(255, 255, 255);color: rgb(255, 255, 255);caret-color: rgb(0, 122, 170);line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;"><span style="color: rgb(18, 18, 18);font-size: 15px;font-family: 微软é é», "Microsoft YaHei";">ä¸é¢å°encoderé¨å</span></p><p style="margin-right: 8px;margin-left: 8px;background-color: rgb(255, 255, 255);color: rgb(255, 255, 255);caret-color: rgb(0, 122, 170);line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;"><br /></p></section></section></section></section></section><section data-tools="135ç¼è¾å¨" data-id="101124"><section style="margin: 10px auto;text-align: center;"><section style="padding: 1em;border-width: 1px;border-style: solid;border-color: rgb(0, 122, 170);"><section style="display: inline-block;"><section data-brushtype="text" style="font-size: 16px;letter-spacing: 1.5px;padding: 6px 1.5em;color: #fff;font-weight: bold;background: #007aaa;border-radius: 30px;"><span style="font-family: 微软é é», "Microsoft YaHei";"><strong style="caret-color: rgb(0, 122, 170);font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;"><span style="font-size: 17px;font-family: 微软é é», "Microsoft YaHei";">encoder</span></strong></span></section></section><section style="margin-right: 1em;margin-left: 1em;"><figure data-size="small" style="margin-top: 1.4em;margin-bottom: 1.4em;color: rgb(18, 18, 18);font-size: medium;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);text-align: left;max-inline-size: 100%;clear: both;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZ9Xic5ApMPqRdYWE1xm3vnWKqVah4CiaErypyjQZYhdpEf2bVk1ia4aD1g/640?wx_fmt=jpeg" data-type="jpeg" width="600" data-backw="279" data-backh="511" style="margin-right: auto;margin-left: auto;max-inline-size: 100%;width: 100%;display: block;cursor: zoom-in;background-color: transparent;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="1.8316666666666668" data-w="600" /></figure><section style="margin: 1.4em 8px;color: rgb(18, 18, 18);font-size: medium;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);text-align: left;max-inline-size: 100%;cursor: text;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="font-family: 微软é é», "Microsoft YaHei";"><strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;outline: none 0px !important;">红è²æ¹æ¡å°±æ¯bertçencoder</span></strong><span style="max-inline-size: 100%;cursor: text;font-size: 15px;">ï¼å¯¹äºBertEncoderï¼æ们éè¦å ä¼ å ¥å®çè¶ åæ°ï¼å¯ä»¥çå°ç¬¬ä¸ä¸ªè¢«ç¨å°çå°±æ¯ï¼</span></span></section><section style="margin-top: 1em;margin-bottom: 1em;color: rgb(18, 18, 18);font-size: medium;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);text-align: left;max-inline-size: 100%;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><pre style="padding: 0.88889em;overflow-wrap: normal;max-inline-size: 100%;font-size: 0.9em;word-break: normal;overflow: auto;background: rgb(246, 246, 246);border-radius: 4px;outline: none 0px !important;"><section style="margin-right: 8px;margin-left: 8px;max-inline-size: 100%;cursor: text;line-height: 1.75em;outline: none 0px !important;"><code style="max-inline-size: 100%;border-radius: 0px;font-size: inherit;background-color: inherit;font-family: Menlo, Monaco, Consolas, "Andale Mono", "lucida console", "Courier New", monospace;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 12px;color: rgb(127, 127, 127);font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">class BertEncoder(nn.Module): def __init__(self, config): super().__init__() self.config = config #BertLayerå°±æ¯ä¸ä¸ªencoderï¼è¿éç¨nn.ModuleListæ¾è¿äºnum_hidden_layers个åå§åçencoder self.layer = nn.ModuleList([BertLayer(config) for _ in range(config.num_hidden_layers)])</span></code></section></pre></section><section style="margin: 1.4em 8px;padding-left: 1em;font-size: medium;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);text-align: left;max-inline-size: 100%;cursor: text;color: rgb(100, 100, 100);border-left: 3px solid rgb(211, 211, 211);line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 12px;color: rgb(127, 127, 127);font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">config.num_hidden_layer</span></section><section style="margin: 1.4em 8px;color: rgb(18, 18, 18);font-size: medium;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);text-align: left;max-inline-size: 100%;cursor: text;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">è¿ä¸ªåæ°å°±æ¯å¾ä¸çNï¼è¿éè¦åºå«äºmulti head self attentionçhï¼è¿éçNåªæ¯è¡¨ç¤ºè¦encodeå 次ï¼æ¯æ¬¡encodeçè¾å ¥è¾åºåéå½¢ç¶ä¿æä¸åï¼å¦æN=3ï¼å</span></section><section style="margin: 1.4em 8px;padding-left: 1em;font-size: medium;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);text-align: left;max-inline-size: 100%;cursor: text;color: rgb(100, 100, 100);border-left: 3px solid rgb(211, 211, 211);line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 12px;color: rgb(127, 127, 127);font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">hs > encoder> hs > encoder > hs > encoder > hs</span></section><section style="margin: 1.4em 8px;color: rgb(18, 18, 18);font-size: medium;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);text-align: left;max-inline-size: 100%;cursor: text;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">encodeäºä¸æ¬¡ï¼ç代ç ï¼å¯ä»¥çå°æä¸ä¸ªModuleListè£ äºãnum_hidden_layers个åå§åè¿çBertLayerï¼forwardå°±æ¯ä¸é¢è¯´çhs > encoderçä¼ éè¿ç¨ã</span></section></section></section></section></section><section data-tools="135ç¼è¾å¨" data-id="101124"><section style="margin: 10px auto;text-align: center;"><section style="padding: 1em;border-width: 1px;border-style: solid;border-color: rgb(0, 122, 170);"><section style="display: inline-block;"><section data-brushtype="text" style="font-size: 16px;letter-spacing: 1.5px;padding: 6px 1.5em;color: #fff;font-weight: bold;background: #007aaa;border-radius: 30px;"><span style="font-family: 微软é é», "Microsoft YaHei";"><strong style="font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;"><span style="font-size: 17px;font-family: 微软é é», "Microsoft YaHei";">encoderæ解</span></strong></span></section></section><section style="margin-right: 1em;margin-left: 1em;"><section style="margin: 1.4em 8px;text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 122, 170);color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">ä¸é¢æ们è¦å解encoderçå é¨ç»æï¼çbertlayeréæå¥ï¼å¯ä»¥çå°æBertAttentionï¼BertIntermediateã</span></section><section style="margin-top: 1em;margin-bottom: 1em;text-align: justify;max-inline-size: 100%;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><pre style="padding: 0.88889em;overflow-wrap: normal;max-inline-size: 100%;font-size: 0.9em;word-break: normal;overflow: auto;background: rgb(246, 246, 246);border-radius: 4px;outline: none 0px !important;"><section style="margin-right: 8px;margin-left: 8px;max-inline-size: 100%;cursor: text;line-height: 1.75em;outline: none 0px !important;"><code style="max-inline-size: 100%;border-radius: 0px;font-size: inherit;background-color: inherit;font-family: Menlo, Monaco, Consolas, "Andale Mono", "lucida console", "Courier New", monospace;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 12px;color: rgb(127, 127, 127);font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">class BertLayer(nn.Module): def __init__(self, config): super().__init__() self.attention = BertAttention(config) self.intermediate = BertIntermediate(config)</span></code></section></pre></section><section style="margin: 1.4em 8px;text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 122, 170);color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">åå«å¯¹åº</span></section><figure data-size="small" style="margin-top: 1.4em;margin-bottom: 1.4em;text-align: justify;max-inline-size: 100%;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);clear: both;color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-backh="193" data-backw="186" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZZuosUWSC9myXQQK2WZcKQhwJMUjjYg6oUeF5QKUA6gMkbzYauIDB4w/640?wx_fmt=jpeg" data-type="jpeg" width="1394" style="margin-right: auto;margin-left: auto;max-inline-size: 100%;width: 100%;display: block;cursor: zoom-in;background-color: transparent;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="1.0402777777777779" data-w="720" /></figure><h3 style="margin: 1.90909em 8px 1.27273em;font-weight: bold;font-size: 1.1em;text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 122, 170);font-variant-numeric: inherit;font-variant-east-asian: inherit;font-stretch: inherit;clear: left;color: rgb(18, 18, 18);line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">BertAttention(Multi-Head Self-Attention)</span></h3><section style="margin: 1.4em 8px;text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 122, 170);color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">å ¶ä¸ç<span style="max-inline-size: 100%;cursor: text;font-weight: 600;">BertAttention</span>å°±æ¯è®ºæä¸ç<span style="max-inline-size: 100%;cursor: text;font-weight: 600;">Multi-Head Self-Attention</span></span></section><figure data-size="normal" style="margin-top: 1.4em;margin-bottom: 1.4em;text-align: justify;max-inline-size: 100%;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-backh="369" data-backw="532" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZRIWD1sK2icvzoSHIP32F0tAcGmslxzDwnC3cxRpiaUib9wTPXWE9lXoLw/640?wx_fmt=jpeg" data-type="jpeg" width="1930" style="margin-right: auto;margin-left: auto;max-inline-size: 100%;width: 532px;display: block;cursor: zoom-in;background-color: transparent;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="0.6930555555555555" data-w="720" /><section style="margin-top: 0.66667em;margin-right: 8px;margin-left: 8px;padding-right: 1em;padding-left: 1em;text-align: center;max-inline-size: 100%;cursor: text;font-size: 0.9em;color: rgb(153, 153, 153);line-height: 1.75em;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-family: 微软é é», "Microsoft YaHei";font-size: 12px;outline: none 0px !important;">Self-Attention</span></section></figure><figure data-size="normal" style="margin-top: 2.24em;margin-bottom: 1.4em;text-align: justify;max-inline-size: 100%;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-backh="163" data-backw="532" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZEMG0UoTM6A6hslgqIsLp0xOP0esaoaTuEJ46y9c87sKnLn8upe8WlQ/640?wx_fmt=jpeg" data-type="jpeg" width="2508" style="margin-right: auto;margin-left: auto;max-inline-size: 100%;width: 532px;display: block;cursor: zoom-in;background-color: transparent;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="0.30694444444444446" data-w="720" /><section style="margin-top: 0.66667em;margin-right: 8px;margin-left: 8px;padding-right: 1em;padding-left: 1em;text-align: center;max-inline-size: 100%;cursor: text;font-size: 0.9em;color: rgb(153, 153, 153);line-height: 1.75em;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-family: 微软é é», "Microsoft YaHei";font-size: 12px;outline: none 0px !important;">Multi-Head Attention</span></section></figure><h3 style="margin: 1.90909em 8px 1.27273em;font-weight: bold;font-size: 1.1em;text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 122, 170);font-variant-numeric: inherit;font-variant-east-asian: inherit;font-stretch: inherit;clear: left;color: rgb(18, 18, 18);line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">BertIntermediate(Position-wise Feed-Forward Networks)</span></h3><section style="margin: 1.4em 8px;text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 122, 170);color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-weight: 600;">BertIntermediate</span>å°±æ¯è®ºæä¸ç<span style="max-inline-size: 100%;cursor: text;font-weight: 600;">Position-wise Feed-Forward Networks</span></span></section><figure data-size="normal" style="margin-top: 1.4em;margin-bottom: 1.4em;text-align: justify;max-inline-size: 100%;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZTVd4Cjnb32AnZW6ricdiammLibrcnPW3sibEF88FkMtynHRFyCqWQriam0Q/640?wx_fmt=jpeg" data-type="jpeg" width="2376" style="margin-right: auto;margin-left: auto;max-inline-size: 100%;width: 2376px;display: block;cursor: zoom-in;background-color: transparent;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="0.3388888888888889" data-w="720" /><section style="margin-top: 0.66667em;margin-right: 8px;margin-left: 8px;padding-right: 1em;padding-left: 1em;text-align: center;max-inline-size: 100%;cursor: text;font-size: 0.9em;color: rgb(153, 153, 153);line-height: 1.75em;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">Position-wise Feed-Forward Networks</span></section></figure><section style="margin: 1.4em 8px;text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 122, 170);color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">æ们æ¥çBertAttention</span></section><section style="margin-top: 1em;margin-bottom: 1em;text-align: justify;max-inline-size: 100%;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><pre style="padding: 0.88889em;overflow-wrap: normal;max-inline-size: 100%;font-size: 0.9em;word-break: normal;overflow: auto;background: rgb(246, 246, 246);border-radius: 4px;outline: none 0px !important;"><section style="margin-right: 8px;margin-left: 8px;max-inline-size: 100%;cursor: text;line-height: 1.75em;outline: none 0px !important;"><code style="max-inline-size: 100%;border-radius: 0px;font-size: inherit;background-color: inherit;font-family: Menlo, Monaco, Consolas, "Andale Mono", "lucida console", "Courier New", monospace;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;color: rgb(127, 127, 127);font-family: 微软é é», "Microsoft YaHei";font-size: 12px;outline: none 0px !important;">class BertAttention(nn.Module): def __init__(self, config): super().__init__() self.self = BertSelfAttention(config) self.output = BertSelfOutput(config) self.pruned_heads = set()</span></code></section></pre></section><section style="margin: 1.4em 8px;text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 122, 170);color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">å ¶å®å°±æ¯ä¸é¢è¿ä¸ªå¾äºï¼</span></section><figure data-size="normal" style="margin-top: 1.4em;margin-bottom: 1.4em;text-align: justify;max-inline-size: 100%;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-backh="294" data-backw="532" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZldrMknUaZ6gfVicaImCGty3dRo4ZzjtfwlxHk5Lt6khFGS8GPjnXkqQ/640?wx_fmt=jpeg" data-type="jpeg" width="2032" style="margin-right: auto;margin-left: auto;max-inline-size: 100%;width: 100%;display: block;cursor: zoom-in;background-color: transparent;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="0.5527777777777778" data-w="720" /></figure><section style="margin: 1.4em 8px;text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 122, 170);color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">æ们主è¦è®²BertSelfAttentionï¼è¿æ¯bertæ主è¦çç¹å¾æ½åé¨å</span></section><section style="margin-top: 1em;margin-bottom: 1em;text-align: justify;max-inline-size: 100%;caret-color: rgb(0, 122, 170);background-color: rgb(255, 255, 255);color: rgb(18, 18, 18);font-size: medium;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><pre style="padding: 0.88889em;overflow-wrap: normal;max-inline-size: 100%;font-size: 0.9em;word-break: normal;overflow: auto;background: rgb(246, 246, 246);border-radius: 4px;outline: none 0px !important;"><section style="margin-right: 8px;margin-left: 8px;max-inline-size: 100%;cursor: text;line-height: 1.75em;outline: none 0px !important;"><code style="max-inline-size: 100%;border-radius: 0px;font-size: inherit;background-color: inherit;font-family: Menlo, Monaco, Consolas, "Andale Mono", "lucida console", "Courier New", monospace;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;color: rgb(127, 127, 127);font-size: 12px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">class BertSelfAttention(nn.Module): def __init__(self, config): super().__init__() if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"): raise ValueError( "The hidden size (%d) is not a multiple of the number of attention " "heads (%d)" % (config.hidden_size, config.num_attention_heads) ) self.num_attention_heads = config.num_attention_heads self.attention_head_size = int(config.hidden_size / config.num_attention_heads) self.all_head_size = self.num_attention_heads * self.attention_head_size self.query = nn.Linear(config.hidden_size, self.all_head_size) self.key = nn.Linear(config.hidden_size, self.all_head_size) self.value = nn.Linear(config.hidden_size, self.all_head_size) self.dropout = nn.Dropout(config.attention_probs_dropout_prob) def transpose_for_scores(self, x): new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size) x = x.view(*new_x_shape) return x.permute(0, 2, 1, 3) def forward( self, hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, output_attentions=False, ): mixed_query_layer = self.query(hidden_states) # If this is instantiated as a cross-attention module, the keys # and values come from an encoder; the attention mask needs to be # such that the encoder's padding tokens are not attended to. if encoder_hidden_states is not None: mixed_key_layer = self.key(encoder_hidden_states) mixed_value_layer = self.value(encoder_hidden_states) attention_mask = encoder_attention_mask else: mixed_key_layer = self.key(hidden_states) mixed_value_layer = self.value(hidden_states) query_layer = self.transpose_for_scores(mixed_query_layer) key_layer = self.transpose_for_scores(mixed_key_layer) value_layer = self.transpose_for_scores(mixed_value_layer) # Take the dot product between "query" and "key" to get the raw attention scores. attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2)) attention_scores = attention_scores / math.sqrt(self.attention_head_size) if attention_mask is not None: # Apply the attention mask is (precomputed for all layers in BertModel forward() function) attention_scores = attention_scores + attention_mask # Normalize the attention scores to probabilities. attention_probs = nn.Softmax(dim=-1)(attention_scores) # This is actually dropping out entire tokens to attend to, which might # seem a bit unusual, but is taken from the original Transformer paper. attention_probs = self.dropout(attention_probs) # Mask heads if we want to if head_mask is not None: attention_probs = attention_probs * head_mask context_layer = torch.matmul(attention_probs, value_layer) context_layer = context_layer.permute(0, 2, 1, 3).contiguous() new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,) context_layer = context_layer.view(*new_context_layer_shape) outputs = (context_layer, attention_probs) if output_attentions else (context_layer,) return outputs</span></code></section></pre></section><section style="margin: 1.4em 8px;text-align: justify;max-inline-size: 100%;cursor: text;caret-color: rgb(0, 122, 170);color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;">讲éçï¼å°±æ¯è®ºæçå ¬å¼ï¼ä½ 们ç»åï¼å¨å®ç°çæ¶åç¹å«è¦æ³¨æä¸ä¸<strong style="max-inline-size: 100%;cursor: text;outline: none 0px !important;">shapeååçç»è</strong>ï¼permuteï¼transposeï¼matmulï¼Softmaxçä½ç½®è¦éªè¯æ¸ æ¥ï¼å¦æä¸æ¸ æ¥å¯ä»¥çè¿ç¯è¯å¶æã</span></section></section></section></section></section><section data-role="paragraph"><section style="margin: 1.4em 8px;max-inline-size: 100%;cursor: text;color: rgb(18, 18, 18);font-size: medium;line-height: 1.75em;font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><span style="max-inline-size: 100%;cursor: text;font-size: 15px;font-family: 微软é é», "Microsoft YaHei";outline: none 0px !important;"><span style="color: rgb(18, 18, 18);font-family: 微软é é», "Microsoft YaHei";font-size: 15px;">æå</span>ï¼æ们åçä¸æ¬¡æ¶æå¾ä½ä¸ºæ»ç»å§ã</span></section><figure data-size="normal" style="margin-top: 1.4em;margin-bottom: 1.4em;max-inline-size: 100%;color: rgb(18, 18, 18);font-size: medium;background-color: rgb(255, 255, 255);font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Microsoft YaHei", "Source Han Sans SC", "Noto Sans CJK SC", "WenQuanYi Micro Hei", sans-serif;outline: none 0px !important;"><img data-backh="145" data-backw="578" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/AJkYaTAujIxnncYia9efBib5j4ia3yRX2WZc2Jmyeh5VMFa5yGtrDalutyCKnFOicJyFyzgxF5xyBl3PrtMTRGyaVA/640?wx_fmt=jpeg" data-type="jpeg" width="3797" style="max-inline-size: 100%;width: 100%;display: block;cursor: zoom-in;background-color: transparent;animation: 0.5s ease-in 0s 1 normal none running fxRichTextFadeIn;box-sizing: border-box !important;outline: none 0px !important;" data-ratio="0.2513888888888889" data-w="720" /></figure><section data-role="paragraph" style="max-inline-size: 100%;border-width: 0px;border-style: none;border-color: initial;z-index: 0;outline: none 0px !important;"><section style="margin-right: 8px;margin-left: 8px;text-align: center;line-height: 1.75em;"><br /></section><hr style="border-style: solid;border-right-width: 0px;border-bottom-width: 0px;border-left-width: 0px;border-color: rgba(0, 0, 0, 0.1);transform-origin: 0px 0px;transform: scale(1, 0.5);" /><section style="margin-right: 8px;margin-left: 8px;text-align: center;line-height: 1.75em;"><br /></section><section style="margin-right: 8px;margin-left: 8px;text-align: center;line-height: 1.75em;"><span style="font-size: 15px;color: rgb(0, 0, 0);font-family: 微软é é», "Microsoft YaHei";"><strong>#2020ï¼å«æçå½æµªè´¹å¨PPTä¸ï¼å»åé ä»·å¼#</strong><br /></span></section><section style="margin-right: 8px;margin-left: 8px;text-align: center;letter-spacing: 0.54px;min-height: 1em;line-height: 1.75em;"><br /></section><section style="margin-right: 8px;margin-left: 8px;text-align: center;line-height: 1.75em;"><img data-type="png" data-s="300,640" data-src="https://mmbiz.qpic.cn/mmbiz_png/AJkYaTAujIx4ElfytiaoGBKLyptvqklwdgJkaI2n0WsBFiaRYwfs1HZ7SAORAKgoNYn4wo5CcusJXAD7XcwZq5Xw/640?wx_fmt=png" style="width: 250px !important;visibility: visible !important;" data-ratio="1.4916666666666667" data-w="1080" /></section><section style="margin-right: 8px;margin-left: 8px;text-align: center;letter-spacing: 0.54px;min-height: 1em;line-height: 1.75em;"><span style="color: rgb(178, 178, 178);font-size: 12px;font-family: 微软é é», "Microsoft YaHei";"><br />ââ END ââ<br /><br /></span></section><section style="margin-right: 8px;margin-left: 8px;text-align: center;letter-spacing: 0.54px;min-height: 1em;line-height: 1.75em;"><span style="font-size: 12px;font-family: 微软é é», "Microsoft YaHei";">çä½ æ³åç<br /></span></section><section style="margin-right: 8px;margin-left: 8px;text-indent: 0em;text-align: center;letter-spacing: 0.54px;background-color: rgb(255, 255, 255);line-height: 1.75em;"><br /></section><section style="margin-right: 8px;margin-left: 8px;text-indent: 0em;text-align: left;letter-spacing: 0.54px;background-color: rgb(255, 255, 255);line-height: 1.75em;"><span style="font-size: 12px;font-family: 微软é é», "Microsoft YaHei";"><strong>åå¸ï¼</strong>| <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831649&idx=1&sn=191239418c15b0c6f395c843415da5fd&chksm=8351793eb426f028970ac64031955167bfde997cdc96375bfb3bce25b5076f1458e11be2be27&token=217203186&lang=zh_CN&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">å¾å·</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831649&idx=4&sn=3a8acfb05820bbd38cd7b15a58e92ec9&chksm=8351793eb426f028dfd96ec6ffc5239a84aaf25b6643813aa6b4a597b9a3215e7ab25e2b4eaf&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">宿è¿</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831649&idx=3&sn=1fcdc66fac79d5ba9504bc24314a14f9&chksm=8351793eb426f02829eef9ba1dfcb8c3e91edb05d3c1d34c548b82f618928834f1da5907d2e7&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">æ·®å</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831649&idx=2&sn=18d65f24543e71ff6bc70eb1cb0a7bde&chksm=8351793eb426f028b98805f0f7555bfc57e3aefe6c76e7ca8df0369ad5dcc4a593a011a9b029&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">åå±±</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831649&idx=5&sn=9f550e5d1f6bd3becc92df723adedd61&chksm=8351793eb426f0289523b13ecd4fcb3d227a3cb259a09e504d388ace5a5eb25ece3aecfe8d46&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">天津</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831491&idx=4&sn=4c4fb5ce9d25c1fe6012d719c5878c82&chksm=8351799cb426f08a8a6b9f35c8db84ceac56401b6da4a6429d1bc5e3339251db372a5045c242&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">æ·±æ±</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831603&idx=5&sn=c2d8bae147d4a2d86ec992c50c39ddad&chksm=8351796cb426f07a50b1acbe92593aabba12c3fb95e9966d93ecad99918dbcf654b9ce767405&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">åå°æ»¨</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831491&idx=3&sn=1e787f725d62f227dbc6c33d0b9ea3eb&chksm=8351799cb426f08accd8896c160b37e7a4436d21e1fbf2cc596763115db5ef6e04dc914ce478&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">å °å·</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831603&idx=4&sn=048088acd651254d77cc238d71b1af04&chksm=8351796cb426f07a7c36e33460f244db68a856502151c73820b314470b7e24412b9a8b6dcd5b&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">常德</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831491&idx=2&sn=b6295a149a38a026e886d81014b568f3&chksm=8351799cb426f08a8a8d7adbd8721956fd91c59be8bd1ec35d4f9963a11e7e67bee72cd49405&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">åå®</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831603&idx=8&sn=0efaff09c26e604d7aa5c29376c129ad&chksm=8351796cb426f07abb37dd11a0cfe941868be2b543feebf288460418759254327292658947ae&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">æ¡æ</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831603&idx=3&sn=12f08ffb076544edcbc24e91bc3d87d7&chksm=8351796cb426f07a1cd4530a9033d462d4732c500dcdda723bdc3a87f4d55f1c62f657a4cb77&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">æ³å·</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831603&idx=6&sn=0a89c4ab17039cec25754aa111b87308&chksm=8351796cb426f07a6fdc14a1c0e877928f2fa46798a8b592b2b2023ad1c65081bbc80b2fafad&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">èå </a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831603&idx=7&sn=1bc72e377ffc3b621c2886d68c437515&chksm=8351796cb426f07adc8c50a3e1409197892c508ad9c27a6b52f8e4dde12740baf5674f5a7fa6&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">åè¥</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831491&idx=1&sn=10e3d481f7fd0ff46dafe2648cd3ddfb&chksm=8351799cb426f08a5118705d6b85487fa97dd013a2aee842be719ee13086aab8dee370b54185&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">å京</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831603&idx=1&sn=9ffda0c7d9667d6700966a8bab12eccf&chksm=8351796cb426f07a4e94a2471b2d59264ed4460e0567c63095fad9c5704541a546d7bc23584a&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">ä¸æµ·ä¸´æ¸¯</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831065&idx=1&sn=2ffb60d98d02ebcab655c55d34d0acbf&chksm=83517746b426fe509cc721e0e3acf05fd1f60b8695dc3b66c08c0a7aa917e7471b013b02c2c2&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">太å</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830804&idx=1&sn=f571d8218b897b21571a6b2f497d3a1b&chksm=8351744bb426fd5d509591673f4082563a02342283f53dfda255ad33304e6746d6448051b9ea&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">温å·</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830454&idx=1&sn=4e7c3289190e2a69c5fb20f2ae8a6d94&chksm=835175e9b426fcffa5f65cafe5274b1c296caf1a51e13ef4c7920c5a18847bef3c7b13c2bfd0&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">ææ</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830295&idx=1&sn=f680ffa2ddf52783e9a7730971520d48&chksm=83517248b426fb5e0efc1e5c62d15b173e4d23ab7aafaab82c6ce861e98d96271fe5dc7db798&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">åå ´</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830278&idx=1&sn=8441ac780477cd9def8d4e9d7866f143&chksm=83517259b426fb4f93ff50f1c07d7b830f16ec4d1e0c23f64ef15a78a056ae301274ffbb8ec3&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">éåº</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830196&idx=1&sn=7a599cf338734c1d41c6fa16005b1641&chksm=835172ebb426fbfdda6873adbeddc124ae670de675a7c36d11977106150a1d924572ea70f299&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">æé½</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830221&idx=1&sn=f5994dbfe9fa00ffce4b5be1a69f3244&chksm=83517292b426fb8419e771cf48ea94de01fc7dc909b4e8c2759c6423b8be6080bc5bd1c248cc&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">æ·±å³</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649829995&idx=1&sn=cd78421e4f2e724f552f8eefc65e0236&chksm=835173b4b426faa26aed46ed566bc1f8ffa11498370fed14fa9d0690795a3eff5a186c63a03d&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">ä¸æµ·</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649829512&idx=1&sn=421f2b766d75009082de8fc1e81cd8d2&chksm=83517157b426f84197c93fd852a3e3f93019c9d81e099784ad0aa09f326ca9dd33a0edb7cfd7&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">ç ä¸è§</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649829480&idx=1&sn=c77baf137c109f1017170a708548d37b&chksm=835171b7b426f8a1d942708988b8f4bd875366f5928d1db99470ccad6958aa0c8cd47b292c23&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">éå®æ°åº</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=401964235&idx=1&sn=8839375241e6a7e69f49aec108ca1ad6&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">æ é¡</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=203026648&idx=1&sn=466dca69ff3eb746e2e5c21831d57f0f&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">å京</a> | <br /></span></section><section style="margin-right: 8px;margin-left: 8px;text-indent: 0em;text-align: left;letter-spacing: 0.54px;background-color: rgb(255, 255, 255);line-height: 1.75em;"><span style="font-size: 12px;font-family: 微软é é», "Microsoft YaHei";"><strong>çç¹ï¼</strong>| <a href="http://mp.weixin.qq.com/s?__biz=MzIxMTUxMzE2Mw==&mid=2247485505&idx=1&sn=90a4482d0ec14f8a4d441f77c653806c&chksm=97557265a022fb7385559a6648c54242c721582b3811ad664360ea1e6f562f0f62264852c20c&scene=21#wechat_redirect" target="_blank" data-linktype="2" hasload="1" data-itemshowtype="0" style="cursor: pointer;">æ·±å³åæ</a><strong> </strong>| <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831957&idx=1&sn=a889b497d60264ef90aea585e4d4556f&chksm=83517bcab426f2dc5e2b3778bc8e8834b453c4eac98c490ebeb525c4a5b01093fe602418b8fb&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">ç¾å½å°äº§</a> | </span><a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831073&idx=1&sn=b9e813dced68e2e1397bd2cc2b6d05c7&chksm=8351777eb426fe68993e996d9312897701ad0e93a7703c5d719dc20ebf09a1b0c6715f502e4d&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;font-size: 12px;text-decoration: underline;font-family: 微软é é», "Microsoft YaHei";">é¿ç§å ¬å¯</a><span style="font-size: 12px;font-family: 微软é é», "Microsoft YaHei";"> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830578&idx=1&sn=73f5149b299d5ce19a433c365eedee83&chksm=8351756db426fc7b173558a1765830766ec5f7fb387e652250f10a106ef6490821f809722cfb&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">ä¸ç¾è´¸ææ</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830503&idx=1&sn=2fae9eec64a6ed71f0c4ee9e4e9bafe7&chksm=835175b8b426fcaec41bb2a484ef939ac78474c77acde2d96a6f7f8b61a0f4ade630c13e7f90&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">粤港澳</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830398&idx=1&sn=ffd40b9cb7730bd712d8f053280d6fa9&chksm=83517221b426fb37e97b45db61547227724f5a09e6a2a0484a55475066ac41fc1b0165954d0d&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">ä¸ç§åä¸</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830360&idx=1&sn=3cea24af7ebf848fde74c52538f111e5&chksm=83517207b426fb11a1ae1c301117e9626ba09012aa9d8b4d8f722e45423ecb7e63987f50bec5&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">2018å¹´èµ°å¿</a> |<br /></span></section><section style="margin-right: 8px;margin-left: 8px;text-indent: 0em;text-align: left;letter-spacing: 0.54px;background-color: rgb(255, 255, 255);line-height: 1.75em;"><span style="font-size: 12px;font-family: 微软é é», "Microsoft YaHei";"><strong>ä¸ä¸ï¼</strong>| <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831987&idx=1&sn=8553537eb64de48a9b7eb16564fe8988&chksm=83517becb426f2fa0c63ef59ace65085334609b45795331170f0edb7f455d154c06d53aa85f7&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">ç©¿è¶çå¸</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831371&idx=1&sn=9b15fe316680aabc64e2b7c37e5767d8&chksm=83517614b426ff0229de8ab490c6a7fb33f9e7de34bae145abd3565818c5965c1eca5a4b0029&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">å°äº§ç æ¥</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831103&idx=1&sn=6ae2c7ba214abd19c76ed907399b77b7&chksm=83517760b426fe767422812a59d53a334c00664a2cc146d59a076bc0e59a062f518c7df00861&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;">æèµ</a> |<a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649831043&idx=1&sn=dcae4bb0d9c69a6aefeb48f7447dff10&chksm=8351775cb426fe4a32f43b1fe3a11ebffb7d381291adc0746d6d0ed9f3efb882617e0407aa10&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;"> æè</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830765&idx=1&sn=d21dbefeb5e1aea1599bcec8fae7a7d4&chksm=835174b2b426fda4b66776ffb3da368f2b71e44233e17581d647b246e52aa53255140e3adac0&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">客ç </a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830698&idx=1&sn=27ffa7f097500e9e7fb23f99fdce51ef&chksm=835174f5b426fde3854d2eec6dc6fd1c0c182faa0218d1b0ce833e50c860c23b062b88f914e0&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">é«å¨è½¬</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830656&idx=1&sn=dc61474f9d09133fcce2934476ea10c1&chksm=835174dfb426fdc9575a9a316aef5c25d051f574d0cd96ae3d1cda189bca7c7b0b7447e403e4&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">æ¿ä»·é¢æµ</a> | <a href="http://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830457&idx=1&sn=37a5c454028fb849c2c5fce3b3575756&chksm=835175e6b426fcf07c9d8fdee249e33edf0ae7f5684be2290c3145624ad430d880a10a654ebd&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">2018æ¿å°äº§æ¥å</a> | <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649830201&idx=1&sn=1816623486b879233513443f1df6ec59&chksm=835172e6b426fbf07aa0f023c3ff79ca1cf11606fc2f2b66cfe773222e3fa1df7f92affa90b6&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">è¿éå®ä½ </a>| <a href="https://mp.weixin.qq.com/s?__biz=MzAxMDEzOTgyNA==&mid=2649829710&idx=1&sn=58d632f4b07dc0796127ef5df605bb36&chksm=83517091b426f9871fa25aed40ade8d757557347ecf505d8b9510645b5adcc6d703caed90b31&scene=21#wechat_redirect" target="_blank" data-linktype="2" style="cursor: pointer;outline: currentcolor none 0px;">æ¶å¿èå¾</a> |</span></section></section><p><br /></p></section><p><br /></p>