# Natural Language Processing (NLP) Application ## Overview Sentiment classification is a subset of text classification in NLP, and is the most basic application of NLP. It is a process of analyzing and inferencing affective states and subjective information, that is, analyzing whether a person's sentiment is positive or negative. > Generally, sentiments are classified into three categories: positive, negative, and neutral. In most cases, only positive and negative sentiments are used for training regardless of the neutral sentiments. The following dataset is a good example. [20 Newsgroups](http://qwone.com/~jason/20Newsgroups/) is a typical reference dataset for traditional text classification. It is a collection of approximately 20,000 news documents partitioned across 20 different newsgroups. Some of the newsgroups are very closely related to each other (such as comp.sys.ibm.pc.hardware and comp.sys.mac.hardware), while others are highly unrelated (such as misc.forsale and soc.religion.christian). As far as the network itself is concerned, the network structure of text classification is roughly similar to that of sentiment classification. After mastering how to construct the sentiment classification network, it is easy to construct a similar network which can be used in a text classification task with some parameter adjustments. In the service context, text classification is to analyze the objective content in the text discussion, but sentiment classification is to find a viewpoint from the text. For example, "Forrest Gump has a clear theme and smooth pacing, which is excellent." In the text classification, this sentence is classified into a "movie" theme, but in the sentiment classification, this movie review is used to explore whether the sentiment is positive or negative. Compared with traditional text classification, sentiment classification is simpler and more practical. High-quality datasets can be collected on common shopping websites and movie websites to benefit the business domains. For example, based on the domain context, the system can automatically analyze opinions of specific types of customers on the current product, analyze sentiments by subject and user type, and recommend products based on the analysis result, improving the conversion rate and bringing more business benefits. In special fields, some non-polar words also fully express a sentimental tendency of a user. For example, when an app is downloaded and used, "the app is stuck" and "the download speed is so slow" express users' negative sentiments. In the stock market, "bullish" and "bull market" express users' positive sentiments. Therefore, in essence, we hope that the model can be used to mine special expressions in the vertical field as polarity words for the sentiment classification system. Vertical polarity word = General polarity word + Domain-specific polarity word According to the text processing granularity, sentiment analysis can be divided into word, phrase, sentence, paragraph, and chapter levels. A sentiment analysis at paragraph level is used as an example. The input is a paragraph, and the output is information about whether the movie review is positive or negative. ## Preparation and Design ### Downloading the Dataset The IMDb movie review dataset is used as experimental data. > Dataset download address: The following are cases of negative and positive reviews. | Review | Label | |---|---| | "Quitting" may be as much about exiting a pre-ordained identity as about drug withdrawal. As a rural guy coming to Beijing, class and success must have struck this young artist face on as an appeal to separate from his roots and far surpass his peasant parents' acting success. Troubles arise, however, when the new man is too new, when it demands too big a departure from family, history, nature, and personal identity. The ensuing splits, and confusion between the imaginary and the real and the dissonance between the ordinary and the heroic are the stuff of a gut check on the one hand or a complete escape from self on the other. | Negative | | This movie is amazing because the fact that the real people portray themselves and their real life experience and do such a good job it's like they're almost living the past over again. Jia Hongsheng plays himself an actor who quit everything except music and drugs struggling with depression and searching for the meaning of life while being angry at everyone especially the people who care for him most. | Positive | Download the GloVe file and add the following line at the beginning of the file, which means that a total of 400,000 words are read, and each word is represented by a word vector of 300 latitudes. ```text 400000 300 ``` GloVe file download address: ### Determining Evaluation Criteria As a typical classification, the evaluation criteria of sentiment classification can be determined by referring to that of the common classification. For example, accuracy, precision, recall, and F_beta scores can be used as references. Accuracy = Number of accurately classified samples/Total number of samples Precision = True positives/(True positives + False positives) Recall = True positives/(True positives + False negatives) F1 score = (2 x Precision x Recall)/(Precision + Recall) In the IMDb dataset, the number of positive and negative samples does not vary greatly. Accuracy can be used as the evaluation criterion of the classification system. ### Determining the Network and Process Currently, MindSpore GPU supports the long short-term memory (LSTM) network for NLP. 1. Load the dataset in use and process data. 2. Use the LSTM network training data to generate a model. Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used for processing and predicting an important event with a long interval and delay in a time sequence. For details, refer to online documentation. 3. After the model is obtained, use the validation dataset to check the accuracy of model. ## Implementation ### Importing Library Files The following are the required public modules and MindSpore modules and library files. ```python import os import shutil import math import argparse import json from itertools import chain import numpy as np from config import lstm_cfg as cfg # Install gensim with 'pip install gensim' import gensim import mindspore.nn as nn import mindspore.context as context import mindspore.dataset as ds from mindspore.ops import operations as P from mindspore import Tensor from mindspore.common.initializer import initializer from mindspore.common.parameter import Parameter from mindspore.mindrecord import FileWriter from mindspore.train import Model from mindspore.nn.metrics import Accuracy from mindspore.train.serialization import load_checkpoint, load_param_into_net from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor ``` ### Configuring Environment Information 1. The `parser` module is used to transfer necessary information for running, such as storage paths of the dataset and the GloVe file. In this way, the frequently changed configurations can be entered during code running, which is more flexible. ```python parser = argparse.ArgumentParser(description='MindSpore LSTM Example') parser.add_argument('--preprocess', type=str, default='false', choices=['true', 'false'], help='Whether to perform data preprocessing') parser.add_argument('--mode', type=str, default="train", choices=['train', 'test'], help='implement phase, set to train or test') # Download dataset from 'http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz' and extract to 'aclimdb_path' parser.add_argument('--aclimdb_path', type=str, default="./aclImdb", help='path where the dataset is store') # Download GloVe from 'http://nlp.stanford.edu/data/glove.6B.zip' and extract to 'glove_path' # Add a new line '400000 300' at the beginning of 'glove.6B.300d.txt' with '40000' for total words and '300' for vector length parser.add_argument('--glove_path', type=str, default="./glove", help='path where the GloVe is store') # Specify the path to save preprocessed data parser.add_argument('--preprocess_path', type=str, default="./preprocess", help='path where the pre-process data is store') # Specify the path to save the CheckPoint file parser.add_argument('--ckpt_path', type=str, default="./ckpt", help='if mode is test, must provide\ path where the trained ckpt file') args = parser.parse_args() ``` 2. Before implementing code, configure necessary information, including the environment information, execution mode, backend information, and hardware information. ```python context.set_context( mode=context.GRAPH_MODE, save_graphs=False, device_target="GPU") ``` For details about the API configuration, see the `context.set_context`. ### Preprocessing the Dataset 1. Process the text dataset, including encoding, word segmentation, alignment, and processing the original GloVe data to adapt to the network structure. ```python # Read data from the specified directory def read_imdb(path, seg='train'): """ read imdb dataset """ pos_or_neg = ['pos', 'neg'] data = [] for label in pos_or_neg: files = os.listdir(os.path.join(path, seg, label)) for file in files: with open(os.path.join(path, seg, label, file), 'r', encoding='utf8') as rf: review = rf.read().replace('\n', '') if label == 'pos': data.append([review, 1]) elif label == 'neg': data.append([review, 0]) return data # Split records into words with spaces as separators def tokenizer(text): return [tok.lower() for tok in text.split(' ')] # Encode words into vectors def encode_samples(tokenized_samples, word_to_idx): """ encode word to index """ features = [] for sample in tokenized_samples: feature = [] for token in sample: if token in word_to_idx: feature.append(word_to_idx[token]) else: feature.append(0) features.append(feature) return features # Align the number of words in each record to 500 words def pad_samples(features, maxlen=500, pad=0): """ pad all features to the same length """ padded_features = [] for feature in features: if len(feature) >= maxlen: padded_feature = feature[:maxlen] else: padded_feature = feature while len(padded_feature) < maxlen: padded_feature.append(pad) padded_features.append(padded_feature) return padded_features # Crop GloVe raw data into a table which contains about 25,000 word vectors def collect_weight(glove_path, vocab, word_to_idx, embed_size): """ collect weight """ vocab_size = len(vocab) wvmodel = gensim.models.KeyedVectors.load_word2vec_format(os.path.join(glove_path, 'glove.6B.300d.txt'), binary=False, encoding='utf-8') weight_np = np.zeros((vocab_size+1, embed_size)).astype(np.float32) idx_to_word = {i+1: word for i, word in enumerate(vocab)} idx_to_word[0] = '' for i in range(len(wvmodel.index2word)): try: index = word_to_idx[wvmodel.index2word[i]] except KeyError: continue weight_np[index, :] = wvmodel.get_vector( idx_to_word[word_to_idx[wvmodel.index2word[i]]]) return weight_np def preprocess(aclimdb_path, glove_path, embed_size): """ preprocess the train and test data """ train_data = read_imdb(aclimdb_path, 'train') test_data = read_imdb(aclimdb_path, 'test') train_tokenized = [] test_tokenized = [] for review, _ in train_data: train_tokenized.append(tokenizer(review)) for review, _ in test_data: test_tokenized.append(tokenizer(review)) vocab = set(chain(*train_tokenized)) vocab_size = len(vocab) print("vocab_size: ", vocab_size) word_to_idx = {word: i+1 for i, word in enumerate(vocab)} word_to_idx[''] = 0 train_features = np.array(pad_samples(encode_samples(train_tokenized, word_to_idx))).astype(np.int32) train_labels = np.array([score for _, score in train_data]).astype(np.int32) test_features = np.array(pad_samples(encode_samples(test_tokenized, word_to_idx))).astype(np.int32) test_labels = np.array([score for _, score in test_data]).astype(np.int32) weight_np = collect_weight(glove_path, vocab, word_to_idx, embed_size) return train_features, train_labels, test_features, test_labels, weight_np, vocab_size ``` 2. Convert the dataset format to the `mindrecord` format for MindSpore to read. ```python def get_imdb_data(labels_data, features_data): data_list = [] for i, (label, feature) in enumerate(zip(labels_data, features_data)): data_json = {"id": i, "label": int(label), "feature": feature.reshape(-1)} data_list.append(data_json) return data_list # Convert the dataset to mindrecord dateset which is supported by MindSpore def convert_to_mindrecord(embed_size, aclimdb_path, proprocess_path, glove_path): """ convert imdb dataset to mindrecord """ num_shard = 4 train_features, train_labels, test_features, test_labels, weight_np, _ = \ preprocess(aclimdb_path, glove_path, embed_size) np.savetxt(os.path.join(proprocess_path, 'weight.txt'), weight_np) # write mindrecord schema_json = {"id": {"type": "int32"}, "label": {"type": "int32"}, "feature": {"type": "int32", "shape": [-1]}} writer = FileWriter(os.path.join(proprocess_path, 'aclImdb_train.mindrecord'), num_shard) data = get_imdb_data(train_labels, train_features) writer.add_schema(schema_json, "nlp_schema") writer.add_index(["id", "label"]) writer.write_raw_data(data) writer.commit() writer = FileWriter(os.path.join(proprocess_path, 'aclImdb_test.mindrecord'), num_shard) data = get_imdb_data(test_labels, test_features) writer.add_schema(schema_json, "nlp_schema") writer.add_index(["id", "label"]) writer.write_raw_data(data) writer.commit() print("============== Starting Data Pre-processing ==============") shutil.rmtree(args.preprocess_path) os.mkdir(args.preprocess_path) convert_to_mindrecord(cfg.embed_size, args.aclimdb_path, args.preprocess_path, args.glove_path) ``` 3. Create a dataset. ```python def create_dataset(base_path, batch_size, num_epochs, is_train): """Create dataset for training.""" columns_list = ["feature", "label"] num_consumer = 4 if is_train: path = os.path.join(base_path, 'aclImdb_train.mindrecord0') else: path = os.path.join(base_path, 'aclImdb_test.mindrecord0') dtrain = ds.MindDataset(path, columns_list, num_consumer) dtrain = dtrain.shuffle(buffer_size=dtrain.get_dataset_size()) dtrain = dtrain.batch(batch_size, drop_remainder=True) dtrain = dtrain.repeat(count=num_epochs) return dtrain ds_train = create_dataset(args.preprocess_path, cfg.batch_size, cfg.num_epochs, True) ``` ### Defining the Network 1. Initialize network parameters and status. ```python def init_lstm_weight( input_size, hidden_size, num_layers, bidirectional, has_bias=True): """Initialize lstm weight.""" num_directions = 1 if bidirectional: num_directions = 2 weight_size = 0 gate_size = 4 * hidden_size for layer in range(num_layers): for _ in range(num_directions): input_layer_size = input_size if layer == 0 else hidden_size * num_directions weight_size += gate_size * input_layer_size weight_size += gate_size * hidden_size if has_bias: weight_size += 2 * gate_size stdv = 1 / math.sqrt(hidden_size) w_np = np.random.uniform(-stdv, stdv, (weight_size, 1, 1)).astype(np.float32) w = Parameter( initializer( Tensor(w_np), [ weight_size, 1, 1]), name='weight') return w # Initialize short-term memory (h) and long-term memory (c) to 0 def lstm_default_state(batch_size, hidden_size, num_layers, bidirectional): """init default input.""" num_directions = 1 if bidirectional: num_directions = 2 h = Tensor( np.zeros((num_layers * num_directions, batch_size, hidden_size)).astype(np.float32)) c = Tensor( np.zeros((num_layers * num_directions, batch_size, hidden_size)).astype(np.float32)) return h, c ``` 2. Use the `cell` method to define the network structure. ```python class SentimentNet(nn.Cell): """Sentiment network structure.""" def __init__(self, vocab_size, embed_size, num_hiddens, num_layers, bidirectional, num_classes, weight, batch_size): super(SentimentNet, self).__init__() # Mapp words to vectors self.embedding = nn.Embedding(vocab_size, embed_size, embedding_table=weight) self.embedding.embedding_table.requires_grad = False self.trans = P.Transpose() self.perm = (1, 0, 2) self.encoder = nn.LSTM(input_size=embed_size, hidden_size=num_hiddens, num_layers=num_layers, has_bias=True, bidirectional=bidirectional, dropout=0.0) w_init = init_lstm_weight( embed_size, num_hiddens, num_layers, bidirectional) self.encoder.weight = w_init self.h, self.c = lstm_default_state(batch_size, num_hiddens, num_layers, bidirectional) self.concat = P.Concat(1) if bidirectional: self.decoder = nn.Dense(num_hiddens * 4, num_classes) else: self.decoder = nn.Dense(num_hiddens * 2, num_classes) def construct(self, inputs): # input:(64,500,300) embeddings = self.embedding(inputs) embeddings = self.trans(embeddings, self.perm) output, _ = self.encoder(embeddings, (self.h, self.c)) # states[i] size(64,200) -> encoding.size(64,400) encoding = self.concat((output[0], output[1])) outputs = self.decoder(encoding) return outputs embedding_table = np.loadtxt(os.path.join(args.preprocess_path, "weight.txt")).astype(np.float32) network = SentimentNet(vocab_size=embedding_table.shape[0], embed_size=cfg.embed_size, num_hiddens=cfg.num_hiddens, num_layers=cfg.num_layers, bidirectional=cfg.bidirectional, num_classes=cfg.num_classes, weight=Tensor(embedding_table), batch_size=cfg.batch_size) ``` ### Defining the Optimizer and Loss Function The sample code for defining the optimizer and loss function is as follows: ```python loss = nn.SoftmaxCrossEntropyWithLogits(is_grad=False, sparse=True) opt = nn.Momentum(network.trainable_params(), cfg.learning_rate, cfg.momentum) loss_cb = LossMonitor() ``` ### Training and Saving the Model Load the corresponding dataset, configure the CheckPoint generation information, and train the model using the `model.train` API. ```python model = Model(network, loss, opt, {'acc': Accuracy()}) print("============== Starting Training ==============") ds_train = create_dataset(args.preprocess_path, cfg.batch_size, cfg.num_epochs, True) config_ck = CheckpointConfig(save_checkpoint_steps=cfg.save_checkpoint_steps, keep_checkpoint_max=cfg.keep_checkpoint_max) ckpoint_cb = ModelCheckpoint(prefix="lstm", directory=args.ckpt_path, config=config_ck) model.train(cfg.num_epochs, ds_train, callbacks=[ckpoint_cb, loss_cb]) ``` ### Validating the Model Load the validation dataset and saved CheckPoint file, perform validation, and view the model quality. ```python print("============== Starting Testing ==============") ds_eval = create_dataset(args.preprocess_path, cfg.batch_size, 1, False) param_dict = load_checkpoint(args.ckpt_path) load_param_into_net(network, param_dict) acc = model.eval(ds_eval) print("============== Accuracy:{} ==============".format(acc)) ``` ## Experiment Result After 10 epochs, the accuracy on the training set converges to about 85%, and the accuracy on the test set is about 86%. **Training Execution** 1. Run the training code and view the running result. ```shell python main.py --preprocess=true --mode=train --ckpt_path=./ckpt ``` As shown in the following output, the loss value decreases gradually with the training process and reaches about 0.249. That is, after 10 epochs of training, the accuracy of the current text analysis result is about 85%. ```shell ============== Starting Data Pre-processing ============== vocab_size: 252192 ============== Starting Training ============== [INFO] ME(15831:140036161672960,MainProcess):2020-03-09-16:29:02.785.484 [mindspore/train/serialization.py:118] Execute save checkpoint process. [INFO] ME(15831:140036161672960,MainProcess):2020-03-09-16:29:03.658.733 [mindspore/train/serialization.py:143] Save checkpoint process finish. epoch: 1 step: 390 , loss is 0.6926409 ... [INFO] ME(15831:140036161672960,MainProcess):2020-03-09-16:32:18.598.405 [mindspore/train/serialization.py:118] Execute save checkpoint process. [INFO] ME(15831:140036161672960,MainProcess):2020-03-09-16:32:19.561.926 [mindspore/train/serialization.py:143] Save checkpoint process finish. epoch: 6 step: 390 , loss is 0.222701 ... epoch: 10 step: 390 , loss is 0.22616856 epoch: 10 step: 390 , loss is 0.24914627 ``` 2. Check the saved CheckPoint files. CheckPoint files (model files) are saved during the training. You can view all saved files in the file path. ```shell ls ckpt/ ``` The output is as follows: ```shell lstm-10_390.ckpt lstm-1_390.ckpt lstm-2_390.ckpt lstm-3_390.ckpt lstm-4_390.ckpt lstm-5_390.ckpt lstm-6_390.ckpt lstm-7_390.ckpt lstm-8_390.ckpt lstm-9_390.ckpt ``` **Model Validation** Use the last saved CheckPoint file to load and validate the dataset. ```shell python main.py --mode=test --ckpt_path=./ckpt/lstm-10_390.ckpt ``` As shown in the following output, the sentiment analysis accuracy of the text is about 86%, which is basically satisfactory. ```shell RegisterOperatorCreator:OperatorCreators init ============== Starting Testing ============== [INFO] ME(29963:140462460516096,MainProcess):2020-03-09-16:37:19.333.605 [mindspore/train/serialization.py:169] Execute load checkpoint process. [INFO] ME(29963:140462460516096,MainProcess):2020-03-09-16:37:20.434.749 [mindspore/train/serialization.py:200] Load checkpoint process finish. [INFO] ME(29963:140462460516096,MainProcess):2020-03-09-16:37:20.467.336 [mindspore/train/serialization.py:233] Execute parameter into net process. [INFO] ME(29963:140462460516096,MainProcess):2020-03-09-16:37:20.467.649 [mindspore/train/serialization.py:268] Load parameter into net process finish. ============== Accuracy:{'acc': 0.8599358974358975} ============== ``` ## Downloading Code Complete and executable code download address: - main.py: code file, including code for data preprocessing, network definition, and model training. - config.py: some configurations on the network, including the batch size and number of training epochs.