geoip

ELK技术栈之-Logstash详解

青春壹個敷衍的年華 提交于 2019-12-02 21:54:05
ELK技术栈之-Logstash详解 前言 在第九章节中,我们已经安装好Logstash组件了,并且启动实例测试它的数据输入和输出,但是用的是最简单的控制台标准输入和标准输出,那这节我们就来深入的学习Logstash的详细使用。 常用启动参数 我们在上一节中演示了启动Logstash的实例,其中我们启动的时候给Logstash脚本传入了-e的参数,但实际上,Logstash的启动参数有很多,我们来看一下各个启动参数的作用: -e #立即启动实例,例如: ./logstash -e "input {stdin {}} output {stdout {}}" -f #指定启动实例的配置文件,例如: ./logstash -f config/test.conf -t #测试配置文件的正确性,例如: ./logstash -f config/test.conf -t -l #指定日志文件名称,例如: ./logstash -f config/test.conf -l logs/test.log -w #指定filter线程数量,不指定默认是5,例如: ./logstash-f config/test.conf -w 8 配置文件语法 文件结构 我们刚刚知道,启动参数可以指定一个配置文件,那么接下来就有必要来了解一下配置文件的结构: Logstash通过{}来定义区域,区域内可以定义插件

【Java】通过ip地址获取详细地域信息(不通过API使用本地库)

匿名 (未验证) 提交于 2019-12-02 21:45:52
MaxMind GeoIP2 服务能识别互联网用户的地点位置与其他特征,应用广泛,包括个性化定制内容、诈欺检测、广告定向、网站流量分析、执行规定、地理目标定位、地理围栏定位 (geo-fencing)以及数字版权管理。目前使用 GeoIP 更多是配合Nginx或Apache服务器进行日志分析获取网站访问量地域分布状况。 GeoIP 分为商业版和免费版,免费版比商业版精度差了许多,经测试对于城市定位确实有差距,能否接受看你的精度要求!(老板说免费的可以了,哈哈) 下载GeoIP2的库,这个库是经常更新的,如果数据要求很高的,需要经常更新(我们不高,预计一年一次) 但是好像网站不太稳定,我这边3个都下载好了,可【点击下载】 关于Java如何使用 不用担心,已经有开源库,maven下载一个 <dependency> <groupId>com.maxmind.geoip2</groupId> <artifactId>geoip2</artifactId> <version>2.12.0</version> </dependency> 没有maven,怎么办?不怕,我上传了,可【点击下载】 使用用到的工具类 import com.maxmind.geoip2.DatabaseReader; import com.maxmind.geoip2.model.AsnResponse; import

How to enable dynamic module with an existing NGINX installation

若如初见. 提交于 2019-12-02 17:09:10
Introduction From NGINX version 1.9.11 and upwarts, a new feature is introduced: dynamic modules. With dynamic modules, you can optionally load separate shared object files at runtime as modules – both third-party modules and some native NGINX modules. ( source ) My setup and the problem I have NGINX installed from the mainline (currently 1.9.14) so it is capable to use dynamic modules. It has also the module I want dynamicly enabled: nginx -V nginx version: nginx/1.9.14 built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1) built with OpenSSL 1.0.1f 6 Jan 2014 TLS SNI support enabled configure

Exception while accessing maxmind's GeoIP-country.mmdb database through hive

一个人想着一个人 提交于 2019-12-02 10:09:42
I have a custom hive UDF to access Maxmind's GeoIP-country.mmdb database that is added to the hive resources through "add file pqr.mmdb". The compiled UDF is added as "add jar abc.jar" When I run a hive query, behind the scenes the java class tries to access the data in geo database and fails by throwing the following exception. Error: java.lang.ClassNotFoundException: com.maxmind.db.Reader$FileMode at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net

Geo redirect user just once using php

非 Y 不嫁゛ 提交于 2019-12-02 03:54:44
问题 i can use this script easily when users land on site.com/redirect.php they get redirected to appropriate TLD according to geo IP but when i add this code in 'index.php' it creates a redirect loop. Can you help me modify it so that it doesn't create loop. right now this 'break' is not helping.. <?php // Next two lines are for Beyond Hosting // Don't forget to change your-domain require_once '/home/your-domain/php/Net/GeoIP.php'; $geoip = Net_GeoIP::getInstance('/home/your-domain/php/Net/GeoIP

Geo redirect user just once using php

ぐ巨炮叔叔 提交于 2019-12-02 01:26:10
i can use this script easily when users land on site.com/redirect.php they get redirected to appropriate TLD according to geo IP but when i add this code in 'index.php' it creates a redirect loop. Can you help me modify it so that it doesn't create loop. right now this 'break' is not helping.. <?php // Next two lines are for Beyond Hosting // Don't forget to change your-domain require_once '/home/your-domain/php/Net/GeoIP.php'; $geoip = Net_GeoIP::getInstance('/home/your-domain/php/Net/GeoIP.dat'); // Next two lines are for HostGator require_once 'Net/GeoIP.php'; $geoip = Net_GeoIP:

geocoding an IP and save using model forms

♀尐吖头ヾ 提交于 2019-12-01 11:21:38
问题 i have this thing in my models.py file user = models.ForeignKey('auth.User', unique = True) latitude = models.DecimalField(max_digits=8, decimal_places=6) longitude = models.DecimalField(max_digits=8, decimal_places=6) Availability = models.CharField(max_length=8,choices=STATUS_CHOICES, blank= False, null=False) Status = models.CharField(max_length=50, blank = True, null= True) and in forms.py i have class registerForm(forms.ModelForm): class Meta: model=register fields = ('latitude',

【记录】logstash 的filter 使用

陌路散爱 提交于 2019-12-01 09:02:08
概述 logstash 之所以强大和流行,与其丰富的过滤器插件是分不开的 过滤器提供的并不单单是过滤的功能,还可以对进入过滤器的原始数据进行复杂的逻辑处理,甚至添加独特的新事件到后续流程中 强大的文本解析工具 -- Grok grok 是一个十分强大的 logstash filter 插件,他可以解析任何格式的文本,他是目前 logstash 中解析非结构化日志数据最好的方式 基本用法 Grok 的语法规则是: %{语法 : 语义} “语法”指的就是匹配的模式,例如使用 NUMBER 模式可以匹配出数字,IP 则会匹配出 127.0.0.1 这样的 IP 地址: %{NUMBER:lasttime}%{IP:client} 默认情况下,所有“语义”都被保存成字符串,你也可以添加转换到的数据类型 %{NUMBER:lasttime:int}%{IP:client} 目前转换类型只支持 int 和 float 覆盖 -- overwrite 使用 Grok 的 overwrite 参数也可以覆盖日志中的信息 filter { grok { match => { "message" => "%{SYSLOGBASE} %{DATA:message}" } overwrite => [ "message" ] } } 日志中的 message 字段将会被覆盖 示例 对于下面的log

How to do a geoip lookup in php?

|▌冷眼眸甩不掉的悲伤 提交于 2019-12-01 08:12:05
All tutorials I have seen have pointed towards functions like geoip_record_by_name . I always get this error: Fatal error: Call to undefined function geoip_record_by_name() in /home/<acct>/public_html/geoip.php on line <line> I'm on "shared" hosting, meaning I don't have access to install the PECL geoip extension. What are other (free!) ways to perform geoip lookups? Preferably ones which don't depend on an external service? If you have MySQL server, there are IP databases that you can use and install for free, then do a $_SERVER['REMOTE_ADDR'] and run it against the database data. i.e. (based

Geoip2's python library doesn't work in pySpark's map function

二次信任 提交于 2019-11-30 20:11:39
I'm using geoip2's python library and pySpark to get the geographical address of some IPs. My code is like: geoDBpath = 'somePath/geoDB/GeoLite2-City.mmdb' geoPath = os.path.join(geoDBpath) sc.addFile(geoPath) reader = geoip2.database.Reader(SparkFiles.get(geoPath)) def ip2city(ip): try: city = reader.city(ip).city.name except: city = 'not found' return city I tried print ip2city("128.101.101.101") It works. But when I tried to do this in rdd.map: rdd = sc.parallelize([ip1, ip2, ip3, ip3, ...]) print rdd.map(lambda x: ip2city(x)) It reported Traceback (most recent call last): File "/home