[{"data":1,"prerenderedAt":292},["ShallowReactive",2],{"content-query-f5s8MevEVT":3},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"date":10,"cover":11,"type":12,"category":13,"body":14,"_type":286,"_id":287,"_source":288,"_file":289,"_stem":290,"_extension":291},"/technology-blogs/en/3283","en",false,"","MindSpore-Powered Emotion Detection of BERT","This blog introduces a use case of BERT for emotion detection.","2024-08-06","https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2025/01/17/df2c0677056446b2ba5f4952ba020ecb.png","technology-blogs","Practices",{"type":15,"children":16,"toc":283},"root",[17,25,31,36,41,46,59,64,69,74,79,89,94,102,107,133,138,143,148,153,158,166,171,183,191,196,204,209,221,229,234,239,247,252,257,265,270,275],{"type":18,"tag":19,"props":20,"children":22},"element","h1",{"id":21},"mindspore-powered-emotion-detection-of-bert",[23],{"type":24,"value":8},"text",{"type":18,"tag":26,"props":27,"children":28},"p",{},[29],{"type":24,"value":30},"Author: JeffDing Source: MindSpore Community Forum",{"type":18,"tag":26,"props":32,"children":33},{},[34],{"type":24,"value":35},"01 Model Introduction",{"type":18,"tag":26,"props":37,"children":38},{},[39],{"type":24,"value":40},"Bidirectional Encoder Representations from Transformers (BERT) is a new language model developed and released by Google at the end of 2018. It plays a crucial role in various natural language processing tasks, including question answering (QA), named entity recognition (NER), natural language inference (NLI), and text classification. The model is built on the transformer's encoder and features a bidirectional structure.",{"type":18,"tag":26,"props":42,"children":43},{},[44],{"type":24,"value":45},"The primary innovation of the BERT model lies in its pre-training method, which employs the Masked Language Model (MLM) and Next Sentence Prediction (NSP) techniques to capture word-level and sentence-level representations, respectively.",{"type":18,"tag":26,"props":47,"children":48},{},[49,51,57],{"type":24,"value":50},"When the MLM method is used to train BERT, 15% of words in the corpus are randomly masked. This masking operation is divided into three cases: 80% of the words are replaced with ",{"type":18,"tag":52,"props":53,"children":54},"strong",{},[55],{"type":24,"value":56},"[Mask]",{"type":24,"value":58},", 10% are replaced with a different word, and 10% remain unchanged.",{"type":18,"tag":26,"props":60,"children":61},{},[62],{"type":24,"value":63},"For QA and NLI tasks, NSP is added to help the model understand the relationship between two sentences. Compared with MLM, NSP is simpler. Two sentences A and B are input to BERT for training, and there is a half probability that the model predicts the sentence B as the next sentence of sentence A.",{"type":18,"tag":26,"props":65,"children":66},{},[67],{"type":24,"value":68},"After BERT is pre-trained, its embedding table and 12-layer transformer weights (BERT-BASE) or 24-layer transformer weights (BERT-LARGE) are saved. The pre-trained BERT model can be used to fine-tune downstream tasks, such as text classification, similarity calculation, and reading comprehension.",{"type":18,"tag":26,"props":70,"children":71},{},[72],{"type":24,"value":73},"Emotion Detection (EmoTect) focuses on identifying user emotion in a conversation scenario. For user text in the conversation scenario, an emotion type of the text is automatically determined, and a corresponding confidence level is provided. The emotion type is classified into positive, negative, and neutral. Conversational EmoTect is applicable to various scenarios, including chat and customer service. It helps enterprises better control the conversation, improve user interaction experience, analyze the customer service quality, and reduce manual inspection costs.",{"type":18,"tag":26,"props":75,"children":76},{},[77],{"type":24,"value":78},"02 MindNLP Installation",{"type":18,"tag":80,"props":81,"children":83},"pre",{"code":82},"pip install mindnlp\n",[84],{"type":18,"tag":85,"props":86,"children":87},"code",{"__ignoreMap":7},[88],{"type":24,"value":82},{"type":18,"tag":26,"props":90,"children":91},{},[92],{"type":24,"value":93},"The following uses a text sentiment classification task as an example to describe how to use BERT.",{"type":18,"tag":80,"props":95,"children":97},{"code":96},"import os\n\nimport mindspore\nfrom mindspore.dataset import text, GeneratorDataset, transforms\nfrom mindspore import nn, context\n\nfrom mindnlp._legacy.engine import Trainer, Evaluator\nfrom mindnlp._legacy.engine.callbacks import CheckpointCallback, BestModelCallback\nfrom mindnlp._legacy.metrics import Accuracy\n\n# prepare dataset\nclass SentimentDataset:\n    \"\"\"Sentiment Dataset\"\"\"\n\n    def __init__(self, path):\n        self.path = path\n        self._labels, self._text_a = [], []\n        self._load()\n\n    def _load(self):\n        with open(self.path, \"r\", encoding=\"utf-8\") as f:\n            dataset = f.read()\n        lines = dataset.split(\"\\n\")\n        for line in lines[1:-1]:\n            label, text_a = line.split(\"\\t\")\n            self._labels.append(int(label))\n            self._text_a.append(text_a)\n\n    def __getitem__(self, index):\n        return self._labels[index], self._text_a[index]\n\n    def __len__(self):\n        return len(self._labels)\n",[98],{"type":18,"tag":85,"props":99,"children":100},{"__ignoreMap":7},[101],{"type":24,"value":96},{"type":18,"tag":26,"props":103,"children":104},{},[105],{"type":24,"value":106},"03 Dataset Preparation",{"type":18,"tag":26,"props":108,"children":109},{},[110,112,117,119,124,126,131],{"type":24,"value":111},"A labeled chatbot dataset that has been tokenized is provided here. The data consists of two columns separated by tabs (\\t). The first column is the emotion category (",{"type":18,"tag":52,"props":113,"children":114},{},[115],{"type":24,"value":116},"0",{"type":24,"value":118}," indicates negative, ",{"type":18,"tag":52,"props":120,"children":121},{},[122],{"type":24,"value":123},"1",{"type":24,"value":125}," indicates neutral, and ",{"type":18,"tag":52,"props":127,"children":128},{},[129],{"type":24,"value":130},"2",{"type":24,"value":132}," indicates positive). The second column is the Chinese text separated by spaces. The file is encoded in UTF-8 format. The following is an example:",{"type":18,"tag":26,"props":134,"children":135},{},[136],{"type":24,"value":137},"label–text_a",{"type":18,"tag":26,"props":139,"children":140},{},[141],{"type":24,"value":142},"0–谁骂人了？我从来不骂人，我骂的都不是人，你是人吗 ？",{"type":18,"tag":26,"props":144,"children":145},{},[146],{"type":24,"value":147},"1–我有事等会儿就回来和你聊",{"type":18,"tag":26,"props":149,"children":150},{},[151],{"type":24,"value":152},"2–我见到你很高兴谢谢你帮我",{"type":18,"tag":26,"props":154,"children":155},{},[156],{"type":24,"value":157},"This part includes dataset reading, data format conversion, data tokenization, and padding.",{"type":18,"tag":80,"props":159,"children":161},{"code":160},"wget https://baidu-nlp.bj.bcebos.com/emotion_detection-dataset-1.0.0.tar.gz -O emotion_detection.tar.gz\ntar xvf emotion_detection.tar.gz\n",[162],{"type":18,"tag":85,"props":163,"children":164},{"__ignoreMap":7},[165],{"type":24,"value":160},{"type":18,"tag":26,"props":167,"children":168},{},[169],{"type":24,"value":170},"04 Data Loading and Preprocessing",{"type":18,"tag":26,"props":172,"children":173},{},[174,176,181],{"type":24,"value":175},"Create a ",{"type":18,"tag":52,"props":177,"children":178},{},[179],{"type":24,"value":180},"process_dataset",{"type":24,"value":182}," function for data loading and preprocessing. The following is an example:",{"type":18,"tag":80,"props":184,"children":186},{"code":185},"import numpy as np\n\ndef process_dataset(source, tokenizer, max_seq_len=64, batch_size=32, shuffle=True):\n    is_ascend = mindspore.get_context('device_target') == 'Ascend'\n\n    column_names = [\"label\", \"text_a\"]\n\n    dataset = GeneratorDataset(source, column_names=column_names, shuffle=shuffle)\n    # transforms\n    type_cast_op = transforms.TypeCast(mindspore.int32)\n    def tokenize_and_pad(text):\n        if is_ascend:\n            tokenized = tokenizer(text, padding='max_length', truncation=True, max_length=max_seq_len)\n        else:\n            tokenized = tokenizer(text)\n        return tokenized['input_ids'], tokenized['attention_mask']\n    # map dataset\n    dataset = dataset.map(operations=tokenize_and_pad, input_columns=\"text_a\", output_columns=['input_ids', 'attention_mask'])\n    dataset = dataset.map(operations=[type_cast_op], input_columns=\"label\", output_columns='labels')\n    # batch dataset\n    if is_ascend:\n        dataset = dataset.batch(batch_size)\n    else:\n        dataset = dataset.padded_batch(batch_size, pad_info={'input_ids': (None, tokenizer.pad_token_id),\n                                                         'attention_mask': (None, 0)})\n\nreturn dataset\n",[187],{"type":18,"tag":85,"props":188,"children":189},{"__ignoreMap":7},[190],{"type":24,"value":185},{"type":18,"tag":26,"props":192,"children":193},{},[194],{"type":24,"value":195},"Static shapes are used for data preprocessing.",{"type":18,"tag":80,"props":197,"children":199},{"code":198},"from mindnlp.transformers import BertTokenizer\ntokenizer = BertTokenizer.from_pretrained('bert-base-chinese')\n\ntokenizer.pad_token_id\n\ndataset_train = process_dataset(SentimentDataset(\"data/train.tsv\"), tokenizer)\ndataset_val = process_dataset(SentimentDataset(\"data/dev.tsv\"), tokenizer)\ndataset_test = process_dataset(SentimentDataset(\"data/test.tsv\"), tokenizer, shuffle=False)\n\nprint(next(dataset_train.create_tuple_iterator()))\n",[200],{"type":18,"tag":85,"props":201,"children":202},{"__ignoreMap":7},[203],{"type":24,"value":198},{"type":18,"tag":26,"props":205,"children":206},{},[207],{"type":24,"value":208},"05 Model Build",{"type":18,"tag":26,"props":210,"children":211},{},[212,214,219],{"type":24,"value":213},"Use ",{"type":18,"tag":52,"props":215,"children":216},{},[217],{"type":24,"value":218},"BertForSequenceClassification",{"type":24,"value":220}," to build a BERT model for sentiment classification, load pre-trained weights, and set hyperparameters for sentiment classification with three categories. Then, perform automatic mixed precision operations on the model to improve the training speed. Next, instantiate an optimizer and evaluation metrics, and set a policy for saving weights for model training. Finally, build a trainer and start model training.",{"type":18,"tag":80,"props":222,"children":224},{"code":223},"from mindnlp.transformers import BertForSequenceClassification, BertModel\nfrom mindnlp._legacy.amp import auto_mixed_precision\n\n# set bert config and define parameters for training\nmodel = BertForSequenceClassification.from_pretrained('bert-base-chinese', num_labels=3)\nmodel = auto_mixed_precision(model, 'O1')\n\noptimizer = nn.Adam(model.trainable_params(), learning_rate=2e-5)\n\nmetric = Accuracy()\n# define callbacks to save checkpoints\nckpoint_cb = CheckpointCallback(save_path='checkpoint', ckpt_name='bert_emotect', epochs=1, keep_checkpoint_max=2)\nbest_model_cb = BestModelCallback(save_path='checkpoint', ckpt_name='bert_emotect_best', auto_load=True)\n\ntrainer = Trainer(network=model, train_dataset=dataset_train,\n                  eval_dataset=dataset_val, metrics=metric,\n                  epochs=5, optimizer=optimizer, callbacks=[ckpoint_cb, best_model_cb])\n# start training\ntrainer.run(tgt_columns=\"labels\")\n",[225],{"type":18,"tag":85,"props":226,"children":227},{"__ignoreMap":7},[228],{"type":24,"value":223},{"type":18,"tag":26,"props":230,"children":231},{},[232],{"type":24,"value":233},"06 Model Verification",{"type":18,"tag":26,"props":235,"children":236},{},[237],{"type":24,"value":238},"Add the validation dataset to the trained model to check the effect of the model on the validation data. The evaluation metric is \"accuracy\".",{"type":18,"tag":80,"props":240,"children":242},{"code":241},"evaluator = Evaluator(network=model, eval_dataset=dataset_test, metrics=metric)\nevaluator.run(tgt_columns=\"labels\")\n",[243],{"type":18,"tag":85,"props":244,"children":245},{"__ignoreMap":7},[246],{"type":24,"value":241},{"type":18,"tag":26,"props":248,"children":249},{},[250],{"type":24,"value":251},"07 Model Inference",{"type":18,"tag":26,"props":253,"children":254},{},[255],{"type":24,"value":256},"Traverse the inference dataset and map the results and labels.",{"type":18,"tag":80,"props":258,"children":260},{"code":259},"dataset_infer = SentimentDataset(\"data/infer.tsv\")\n\ndef predict(text, label=None):\n    label_map = {0: \"negative,\" 1: \"neutral,\" 2: \"positive\"}\n\n    text_tokenized = Tensor([tokenizer(text).input_ids])\n    logits = model(text_tokenized)\n    predict_label = logits[0].asnumpy().argmax()\n    info = f\"inputs: '{text}', predict: '{label_map[predict_label]}'\"\n    if label is not None:\n        info += f\" , label: '{label_map[label]}'\"\n    print(info)\n\nfrom mindspore import Tensor\n\nfor label, text in dataset_infer:\n    predict(text, label)\n",[261],{"type":18,"tag":85,"props":262,"children":263},{"__ignoreMap":7},[264],{"type":24,"value":259},{"type":18,"tag":26,"props":266,"children":267},{},[268],{"type":24,"value":269},"08 Custom Inference Dataset",{"type":18,"tag":26,"props":271,"children":272},{},[273],{"type":24,"value":274},"Input inference data to test the generalization capability of the model.",{"type":18,"tag":80,"props":276,"children":278},{"code":277},"predict(\"家人们咱就是说一整个无语住了 绝绝子叠buff\")\n",[279],{"type":18,"tag":85,"props":280,"children":281},{"__ignoreMap":7},[282],{"type":24,"value":277},{"title":7,"searchDepth":284,"depth":284,"links":285},4,[],"markdown","content:technology-blogs:en:3283.md","content","technology-blogs/en/3283.md","technology-blogs/en/3283","md",1776506110978]