lexer

Writing a lexer for a new programming language in python

白昼怎懂夜的黑 提交于 2020-08-20 10:36:15
问题 I have no idea how/where to start. I'm supposed to be using python, and more specifically, the ply library. So far, all I've done in create a list of tokens that will be part of the language. That list is given below: tokens = ( # OPERATORS # 'PLUS' , # + 'MINUS' , # - 'MULTIPLY', # * 'DIVIDE', # / 'MODULO', # % 'NOT', # ~ 'EQUALS', # = # COMPARATORS # 'LT', # < 'GT', # > 'LTE', # <= 'GTE', # >= 'DOUBLEEQUAL', # == 'NE', # # 'AND', # & 'OR', # | # CONDITIONS AND LOOPS # 'IF', # if 'ELSE', #

编译lineage,gcc的版本问题

自古美人都是妖i 提交于 2020-08-14 23:00:14
从9.3.0版本升级到10.1.0版本后,编译出错,具体信息如下: /usr/bin/ld: scripts/dtc/dtc-parser.tab.o:(.bss+0x50): multiple definition of `yylloc'; scripts/dtc/dtc-lexer.lex.o:(.bss+0x0): first defined here collect2: 错误:ld 返回 1 make[3]: *** [scripts/Makefile.host:127:scripts/dtc/dtc] 错误 1 make[2]: *** [kernel/xiaomi/msm8992/scripts/Makefile.build:455:scripts/dtc] 错误 2 make[2]: *** 正在等待未完成的任务.... HOSTLD scripts/mod/modpost make[1]: *** [kernel/xiaomi/msm8992/Makefile:525:scripts] 错误 2 make: *** [Makefile:130: sub-make] Error 2 make: Leaving directory 'kernel/xiaomi/msm8992' ninja: build stopped: subcommand failed. make:

《编译原理系列》——”揭开编译器前端的神秘面纱“

亡梦爱人 提交于 2020-08-13 16:26:43
------------恢复内容开始------------ ”揭开编译器前端的神秘面纱“——龙书附录读书总结 概述 ​ 历时10+h左右的时间,终于把课本附录的代码和解读都过了一遍。在这个过程,对编译器前端的认识也从最初的敬畏和稍微的恐惧,再随着逐渐掌握后,变为“不过如此”的感觉。个人觉得这部分掌握的难点应该在于,类的数量繁多,遇到“又忘记了”的属性和方法,要经常翻回去看。另外,对重要的类和方法做一定的自我总结,也是非常必要的。下面,就书上的一些重要的类,给出一些我个人的总结。 词法分析器 这部分的工作是分词,也就是把一串字符串识别为了一个个的词 Tag:定义了词法单元常量,例如Tag.AND=256 Token:就是一个int Word:把token的int和string绑起来了 Real:Tag.REAL(就是个数字)+ float Lexer:维护Word哈希表(字符串到TAG数值的一个映射),readch和read用来辅助一个一个字符地读,scan函数用于匹配对应词法单元。 符号表和类型 ------------恢复内容结束------------ 来源: oschina 链接: https://my.oschina.net/u/4347613/blog/4313892

SpringMVC日期格式属性自动转成时间戳实现源码分析

穿精又带淫゛_ 提交于 2020-08-11 09:11:11
背景介绍 SpringMVC搭建的微服务系统,后端数据库对时间类型的存储使用的是Long类型,而前端框架倾向于使用yyyy-MM-dd HH:mm:ss这种标准显示格式,前端JSON格式的请求报文与后台的接口交互都需要进行格式转换,这部分转换功能由后台实现。 使用时我们发现,前端定义的JSON请求,时间格式为yyyy-MM-dd HH:mm:ss,如果后台定义的POJO相应的属性为Long类型,可以自动转换为时间戳,对此非常好奇,框架是如何实现这一功能的? 框架选型、版本及主要功能 spring boot 2.1.6.RELEASE spring cloud Greenwich.SR3 alibaba fastjson 1.2.60 注意json框架使用的是fastjson 代码演示 为了方便演示,定义一个特别简单的POJO类: public class DateReq { private String dateFormat; private Long timestamp; // 省略getter/setter/toString方法 } 再定义一个简单的Controller方法: @RestController public class DemoController { @PostMapping(value = "/json/demo/info") public

What is the most efficient way to create a lexer?

偶尔善良 提交于 2020-06-11 09:45:07
问题 I am currently trying to learn how to create my own lexical analyser, by hand. I had been using Flex (along with Bison) a lot to practice and learn how it works internally, but I am currently seeing at least 3 different solutions to develop my own. Using a list of REs, going through each and when one matches, simply return the associated token (see python docs about REs) Creating a DFA from REs (as does Flex for example: based on REs, create a big state machine) Creating my own 'state machine

What is the most efficient way to create a lexer?

偶尔善良 提交于 2020-06-11 09:45:01
问题 I am currently trying to learn how to create my own lexical analyser, by hand. I had been using Flex (along with Bison) a lot to practice and learn how it works internally, but I am currently seeing at least 3 different solutions to develop my own. Using a list of REs, going through each and when one matches, simply return the associated token (see python docs about REs) Creating a DFA from REs (as does Flex for example: based on REs, create a big state machine) Creating my own 'state machine

[Java]算术表达式求值之二(中序表达式转后序表达式方案,支持小数)

烈酒焚心 提交于 2020-05-08 09:30:16
Inlet类,入口类,这个类的主要用途是验证用户输入的算术表达式: package com.hy; import java.io.BufferedReader; import java.io.IOException; import java.io.InputStreamReader; // 此类用于把算术表达式送入解析器 public class Inlet { public static void main(String[] args) throws IOException{ // 取得用户输入的表达式 BufferedReader br = new BufferedReader( new InputStreamReader(System.in)); String rawExpression = null ; System.out.print( "请输入算术表达式:" ); rawExpression = br.readLine(); // 得到合法的算术表达式 String expression="" ; for ( int i=0;i<rawExpression.length();i++ ){ // 拿到表达式的每个字符 char c= rawExpression.charAt(i); // System.out.print(c+","); if (Character

CMSC 216 Project #6 Spring

℡╲_俬逩灬. 提交于 2020-05-03 21:11:42
CMSC 216 Project #6 Spring 2019 A Simple Shell Fri May 3, 11:30PM, Tue May 7, 11:30PM 1 Overview In this project, you will write the guts of a shell that will support boolean operations, pipes, and file redirection. The project has two deadlines: Fri May 3, 11:30PM - Your code must pass the following public tests: public00/01/02/06/07/11. That is the only requirement for this deadline. We will not grade the code for style. This first part is worth .5% of your course grade (NOT .5% of this project grade). Notice you can still submit late for this part. Tue May 7, 11:30PM - Final deadline for

分布式 | DBLE 之 SQL 解析

老子叫甜甜 提交于 2020-04-28 19:54:00
作者:路路 热爱技术、乐于分享的技术人,目前主要从事数据库相关技术的研究。 本文来源:原创投稿 *爱可生开源社区出品,原创内容未经授权不得随意使用,转载请联系小编并注明来源。 数据库中间件与数据库有什么区别? 个人认为除了没做数据存储,其他的功能数据库中间件几乎一样不少,比如 SQL 解析、结果集处理、协议实现等。 SQL 解析的定义 今天我们主要来谈谈 SQL 解析,SQL 的全称为 Structured Query Language,即结构化查询语言,既然定义为语言,那其实它和任何其他语言都是平等的。所以对 SQL 的解析完全等同于对任何一门语言的解析,对编程语言的解析这就涉及到编译原理相关知识了。 想要解析一门语言,通常需要具备以下工具: 词法分析器(Lexer):负责解析基本的词法,也就是将字符序列转换为单词(Token)序列的过程。 语法分析器(Parser):将词法分析器解析出的单词(Token)序列,进一步构建成有语义的数据结构,比如抽象语法树。 访问器(Visitor):能够对语法分析器生成的抽象语法树进行遍历,获取需要的信息。 上面的定义可能大家看完还有点懵懂,这里我通过一个实际的例子来让大家有个更具体的认识。 案例解析 原始 SQL 如下: SELECT id, name FROM test WHERE ID > 2 LIMIT 2; 下面我们来解析这个 SQL

How to throw meaningful error on syntax or parsing error [duplicate]

家住魔仙堡 提交于 2020-03-28 07:06:50
问题 This question already has answers here : Is there a way to easily adapt the error messages of ANTLR4? (2 answers) Closed 10 days ago . I have a grammar for parsing jcl The jcl looks like below /*comment line //PR1290@ JOB (10),'ISPW COB SN900E' lexer and parser is working perfectly fine. Suppose instead of // if jcl starts from / currently lexer is throwing 1:0 token recognition error at: '/P' Parser will throw no viable alternative input R1290@ JOB I am looking for throwing error similar to