I always do it in the following way:
plot(X)
set(gca, 'FontName', 'Arial')
set(gca, 'FontSize', 12)
ylabel('Label Y axis')
xlabel('Label X axis')
In this way, the axis and the label will have the requested font and size. It is important to put 'xlabel' and 'ylabel' after the 'set'. The order in this case matters.
There is other way to set the fonts for the xlabel, ylable, legend, plot as below; it may complement the upper answer:
% Define values
linewidthnumber = 2;
legendfontsize = 12;
labelfontsize = 12;
axisfontsize = 12;
custom_plot_handle_name = plot(x);
custom_legend_handle_name = legend();
custom_x_label_name = xlabel();
custom_y_label_name = ylabel();
% Set the parameters now to the given values
set(h, 'Linewidth', linewidthnumber);
set(custom_legen_handle_name,'FontSize', legendfontsize);
set(custom_x_label_name, 'FontSize', labelfontsize);
set(custom_y_label_name, 'FontSize', labelfontsize);
set(gca, 'FontSize', axisfontsize);