Enhanced Performance of Dynamic Neural Network Model using Wavelet Activation Functions

Both static and dynamic adaptive neural networks have been broadly utilized in mathematical modeling and numerical analysis. This study aimed to enhance the accomplishment of Dynamic Neural Networks (DNN) models by applying wavelet functions as activation functions. Research that models and forecasts the intensity of solar radiation in Mataram City shows that combining B-Spline and Morlet wavelet activation functions can significantly increase the DNN model performance. Wavelet-DNN (W-DNN) was modeled with an identical architecture; the best showed the increase in the model achievement (0.7596 points for in-sample and 0.8502 points for out-sample data). Mainly for out-sample data, the model's performance using the W-DNN + intervention model increased by 4.0492 points.


Introduction
Some aspects researchers consider as the basis for using the neural network model (NN) as an analytical tool include having adaptive abilities, learning algorithms by self, generalization abilities, and solving complex and complicated nonlinear problems [1].Based on the time variable, the applications of the NN models can be divided into two categories, namely, static and dynamic neural network (SNN and DNN) models.The SNN model is based on a fixed time (static), while the DNN model develops according to changes in time (dynamic).
In terms of modeling solar radiation intensity, many studies have investigated the scatter scheme of the intensity of solar radiation in different locations using various methods and approaches.The other modeling techniques have been implemented by a combination method of empirical techniques and machine learning [2], deep learning model [3], six machine learning algorithms for daily global solar radiation and air temperature [4], multivariate time series based on NN method [5], ensemble artificial intelligence [6], and deep learning approach [7].Meanwhile, those based on (static) neural network models include the ANN model with different learning algorithms [8], the modeling of solar radiation intensity in Mataram City, the hybrid wavelet-based model in Lombok Island, and SNN [9].
Introduced around 2000, the development and application of the DNN is a relatively new model after Warren McCulloch and Walter Pitts proposed the (static) neural network models in 1943 [10].In its development, the DNN model has been applied to different research to address problems, including weather data forecasting [11], Zika virus risk forecasting [12], Time-varying inner wall temperature prediction [13], VGF crystal growth process forecast [14], diagnostic and detection fault [15], and wastewater effluent quality prediction [16].
The implementation of a DNN model using a recurrent neural network (Elman and Jordan) has been carried out by [11] [17] using meteorological data in the form of wind speed, wind direction, air humidity, air pressure, and rainfall to model and predict the intensity of solar radiation in Mataram City, West Nusa Tenggara using the DNN model to produce significant forecasts.
Combining two or more methods and models into one model to solve a problem is called a hybrid model.In practice, several studies show that the hybrid model shows significant results in terms of effectiveness and efficiency.In particular, research in [9] showed that the Wavelet Neural Network Model (static) could effectively model and estimate solar intensity.Furthermore, [17] has successfully modeled and predicted solar radiation intensity using the DNN model.
Based on the advantages of the hybrid model, this research combines the wavelet method and the neural network model to develop a hybrid model called Wavelet-DNN (W-DNN) using the neural network model as the core model and the wavelet method as a tool to optimize it.The W-DNN model is used to model and predict the intensity of solar radiation as a health indicator, especially in Mataram City.

Data and Data Organization
This study used the solar radiation data of Mataram City from June 2018 until May 2019, derived from the Environment and Forestry Agency, West Nusa Tenggara Province (Figure 1).Meteorological information was utilized, including wind speed, humidity, air temperature, air pressure, and rainfall as predictor data.Next, solar radiation data was used in response to predictor data.Descriptively, the intensity of solar radiation data of Mataram City is given by several parameters, namely mean, max, min, and standard deviation of data, 55.3108, 135.3750, 0.00, and 14.2272, respectively.
Solar radiation intensity as an environmental parameter is affected by various meteorological specifics, including wind speed, air pressure, humidity, and rainfall.Solar radiation intensity can also be analyzed using time series data based on previous solar radiation intensities.Based on this situation, the intensity of solar radiation can be formulated as follows: ( )

The Architecture of the Wavelet-Dynamic Neural Network (W-DNN) Model
The W-DNN architecture we propose consists of 6 layers: two initial layers, three hidden layers, and one output layer (Figure 2).The initial layer shows the data pre-processing process using the normalization method.The hidden layer depicts the data aggregation and activation process, and the output layer represents the hidden layer output aggregated into the model output.This W-DNN architecture was developed in our previous research [17].

Activation Function
This research model predicts solar radiation intensity as the impact of various meteorological and rainfall variables.The DNN Model uses the wavelet activation function, called the Wavelet-Dynamic Neural Network (W-DNN).The wavelet functions used are the B-spline Wavelet and the Morlet Wavelet.
The successful application of a neural network in data analysis depends on selecting the activation function used in the model.There were two types of wavelet functions used in this study, namely the B-Spline Wavelet [18] and the Morlet Wavelet, shown respectively by the following equations for any variable x:

Feed-Forward W-DNN procedures
The W-DNN feed-forward procedure offered in this research is basically on the architecture in Figure 2 and is provided through the following steps.
The 1 st Layer: the input data is separated into two categories.The first input category, meteorological variables, including wind speed, temperature, humidity, air pressure, and rainfall, consists of m 1 data.The second input group comprises previous solar radiation data (lags) consisting of m 2 data.
The 2 nd Layer: the input data was transformed through normalization methods by the following rules: where where 1,2, , jC = using the equations as follows: for some constants 12 and .


The 6-th Layer: The terminal outputs of the W-DNN model are as the following: Type a: The optimizing procedure was accomplished using the partial differential formulas as follows: 44 Moreover, the procedure to improve weight used: where, ( determined by Equations ( 9)-(13).

Results and Discussion
The Wavelet-Dynamics Neural Network (W-DNN) model proposed in this research is divided into two types: the Type-a Model, the W-DNN Model with output aggregation using weighted coefficients, and the Type-b Model, the W-DNN Model with output aggregation using the maximum coefficient.

W-DNN Models with Weighted Coefficients (Type-a)
Referring to equation (7a) and numerical simulation using the trial-and-error method, data modeling of solar radiation intensity that is influenced by meteorological data and rainfall using the DNN Wavelet Model with weighted coefficients is calculated by the following equations:  Descriptively, the model implementation to the intensity of solar radiation data influenced by meteorological and rainfall data can be seen in Table 1.Based on Figure 3, it can be seen that the W-DNN model result curve tends to be close to the average and shows that the data pattern produced by the W-DNN model is generally similar to the characteristics of the actual data, especially in out-sample data.In this case, it can be seen that when the actual data graph increases or decreases, the model data also shows the same behavior.The statistics data in Table 1 reinforce that the average W-DNN model has a difference of 0.3663 points below the average actual data.By paying attention to the patterns formed, especially in the out-sample data, it can be seen that the model patterns are consistently below the actual data.If the constant factor can be viewed as an exogenous factor, and the addition of a constant value in equation ( 16) of 0.0850 points is a form of exogenous factor intervention, and other parameters are considered stable, then equation ( 16) given as below: 1.035 0.3342 Applying equation (17) to the testing data (out sample) increases the model's performance for forecasting the intensity of solar radiation; initially, the RMSE was 17,680 to 16.3169.Graphically, the graph quality of testing data (out sample) is shown in Figure 4.If observed specifically for the testing data, Figure 5 shows that the W-DNN Model given by equation ( 17), called the W-DNN + Model, with a green graph, is more accurate than the original W-DNN Model with a red graph.This is statistically proven by the RMSE indicators of 16.3160 and 17.6820, respectively.Referring to equation (7b) and numerical simulation using the trial-and-error method, data modeling of solar radiation intensity that is influenced by meteorological data and rainfall uses the DNN Wavelet Model with the maximum coefficient given by the following equation:    3), which gives results with RMSE- based accuracy of 13.5306 with a graph as shown in Figure 6.Referring to equation (18) with the same indicator, namely RMSE, the model accuracy for the testing data (out-sample) is obtained at 17.5422.

Figure 6. Graphical comparison of actual data on solar radiation intensity (blue) with the W-
DNN model (red) Descriptively, the application of the W-DNN model with maximum coefficients on solar radiation intensity data influenced by meteorological and rainfall data can be seen in Table 2. Figure 6 indicates that the yield curve of the W-DNN model with maximum coefficients tends to be close to the average and shows that the data pattern produced by the W-DNN model is generally similar to the characteristics of the actual data.In this case, it can be seen that when the actual data graph increases or decreases, the model data also shows the same behavior.This is reinforced by the statistical data in Table 2, which shows that the average W-DNN model has a difference of 0.0945 points above the average actual data.
Similar to the type-a model, paying attention to the patterns formed in the out-sample data shows that the model pattern is consistently below the actual data.If the constant factor is seen as an exogenous factor, and the addition of the constant value in equation ( 18) of 0.0850 points as a form of exogenous factor intervention, and other parameters are considered stable, equation (18) given as below:    19) to the testing data (out sample) increases the model's performance for forecasting the intensity of solar radiation, from an RMSE of 17.5422 to 14.3432.The increase in graphical performance on the testing data (out sample) is shown in Figure 7.If observed specifically for the testing data, Figure 8 shows that the W-DNN Model given by equation ( 19), called the W-DNN + model, with a green graph, is more accurate than the original W-DNN Model with a red graph.This is statistically proven by the RMSE indicators of 14.3432 and 17.5422, respectively.3).Performance comparison between the DNN and W-DNN models is shown in Table 3 below.Based on Table 3, it can be seen that using the activation function from the wavelet function family can improve the performance of the DNN model, both for training data and testing data.For insample data, the W-DNN model with output values calculated using input aggregation using weighted coefficients tends to be better than the aggregation model using the maximum coefficient.As for the out-sample data, the W-DNN model with output values calculated using input aggregation using maximum coefficients tends to be better than the aggregation model using weighted coefficients.For training data, RMSE Model W-DNN Type A and Type B are 13.5306 and 13.9380, respectively.As for the testing data, RMSE Model Type A and Type B were 17.6820 and 17.5422, respectively.These results align with research conducted by [9] and [17].
The performance of the W-DNN Model, especially for testing data, can be improved by providing intervention to the W-DNN Model, both Type A and Type B, by adjusting the constant values in the model.For Type A, adding a constant value of 0.0850 points causes an increase in model performance of 1.366, from an RMSE of 17.6820 to 16.3160.Whereas for Type b, adding a constant value of 0.1492 points causes an increase in model performance of 3.199, from an RMSE of 17.5422 to 14.3432.

Conclusion
Case studies of modeling and forecasting the intensity of solar radiation as the effect of several

Figure 1 .
Figure 1.The intensity of solar radiation data of Mataram City from June 2018 until May 2019

Figure 2 .
Figure 2. The proposed W-DNN architecture

N
.24843/LKJITI.2023.v14.i03.p03e-ISSN 2541-5832 Accredited Sinta 2 by RISTEKDIKTI Decree No. 158/E/KPT/2021 154 for fundamental constants a and b.2.5.Learning Parameters OptimizationOptimizing the parameter was executed in the backward procedures of the W-DNN model.The optimized parameters contained weights 1 W , 2 W , 31 W , 32 W , and 4 W . Optimizing the parameter utilizing the momentum gradient descent, minimizing the cost function: indicates the number of data, j y and d j y correspondingly shows the output of the offered model and the j-th objective data, j = 1,2, …, N.

2 ,
and the B-Spline Wavelet Equation (2), and ( ) p k t refer to the Morlet Wavelet Equation (3) give the result with an accuracy based on RMSE of 13.5306 with a graph as shown in Figure3.Referring to equation (16) with the same indicator, namely RMSE, the model's accuracy on testing data (out-sample) is 17.6820.

Figure 3 .
Figure 3.Comparison between actual data of solar radiation intensity (blue) and W-DNN model (red)

Figure 4 .
Figure 4. Graphical comparison between actual data of solar radiation intensity (blue) and W-DNN model (red) and improvement of W-DNN model curve on the testing data (green).

Figure 5 .
Figure 5. Graphic comparison of solar radiation intensity for testing data (out sample) between actual data (blue), W-DNN model data (red), and W-DNN+ intervention model (green)

2 ,
refer to the B-Spline wavelet function in equation (2), and ( ) p k t refer to the Morlet wavelet function in equation (

Figure 7 .
Figure 7. Graphical comparison of actual data between solar radiation intensity (blue) and the W-DNN model (red) and improvement of the W-DNN model curve on the testing data (green).

Figure 8 .
Figure 8. Graphical comparison of solar radiation intensity for testing data (out sample) between actual data (blue), W-DNN model data (red), and W-DNN + intervention model (green)Referring to the research conducted by[17], using the activation function from the wavelet function family improves the performance of the DNN model both for training data and testing meteorological and rainfall variables show that applying the wavelet function as an activation function in the Dynamic-Neural Network (DNN) Model can improve the models' performance.A performance increase occurs with both training and testing data.The W-DNN forecasting model performs better when constant parameters are adjusted as an exogenous factor.RISTEKDIKTI Decree No. 158/E/KPT/2021

LONTAR KOMPUTER VOL. 14, NO. 3 DECEMBER 2023 p-ISSN 2088-1541 DOI : 10.24843/LKJITI.2023.v14.i03.p03 e-ISSN 2541-5832 Accredited Sinta 2 by RISTEKDIKTI Decree No. 158/E/KPT/2021
and ' min X correspondingly, indicate the k -th, the minimum, and the maximum values from the primary data set.The number of neurons was identical to the 1 st Layer, namely 12 m m m =+ neuron.The 3-rd Layer: If C is the number of classes of the input data, then each input data

Table 1 .
The W-DNN model's performance on solar radiation intensity is determined by meteorological and rainfall variables.

Table 2 .
W-DNN model performance in modeling the intensity of solar radiation determined by meteorological and rainfall variables

Table 3 .
Comparison of the DNN Model and the W-DNN Model