I have written a code to import a .csv file (containing numbers) into an excel file through openpyxl. It works, however, all the cells have written the numbers to the excel
You need to convert the value from the CSV file to what you need. All values in CSV files are strings.
ws.cell('%s%s'%(column_letter, (row_index + 1))).value = int(cell) ought to do it.
BTW. you might want to look at the ws.append() method.