cdc

Sql Server Change Data Capture: Preserving history when adding columns?

只谈情不闲聊 提交于 2019-12-09 17:56:14
问题 When a new column is added to table that is configured for change data capture (cdc), the capture instance table will not have the new column until cdc is disabled and re-enabled for the source table. In the process the existing capture instance is dropped. I thought I could copy existing data out to a temp table and then copy back using the following SQL. However, other CDC meta information, such as the cdc.change_tables.start_lsn, becomes invalid. How can the capture instance history be

What does LSN mean in SQL Server?

不想你离开。 提交于 2019-12-08 16:33:31
问题 What is the meaning of Log Sequence Number? I know that it is of type binary and 10bytes long and it corresponds to the time the transaction happen in DB. But is this a high precision date-time value that is stored in some efficient binary format or is this a function of date-time and something else (for example the serial number of transactions that happen at the same milli second). I did a lot of searching but couldn't find a good answer to this. Can any one explain with a formula or

汽车之家社区从传统商业数据库到开源分布式数据库的架构变迁

╄→гoц情女王★ 提交于 2019-12-05 20:34:57
一、项目介绍 汽车之家社区于 2005 年上线,作为之家最老的业务之一,十四年来沉淀了亿级帖子、十亿级回复数据,目前每天有千万级 DAU、亿级的访问量,接口日均调用量 10亿+次 。期间经历过架构升级重构、技术栈升级等,但其数据始终存放在SQL Server中,随着数据的不断递增,我们在使用SQL Server 数据库方面遇到了很多瓶颈,以至于我们不得不寻找一个新的数据库替换方案。 二、使用SQL Server遇到的瓶颈 随着业务的不断扩大,汽车之家社区的访问量和发表量不断上涨,遇到的数据库问题也越来越多,下面列举两个必须很快要解决掉的问题: 历史上,之家社区回复库采用了分库分表的设计,用以解决SQL Server单表过大的时候性能下降等问题。时至今日,回复库有100+个库、1000+张表(根据帖子ID分库分表)。这本身并没有问题,代码写好了,数据该写哪里写哪里,该读哪里读哪里。但是随着应用的发展、需求的变化,我们发现在实现某些需求时,分库分表的结构难以满足。我们需要数据逻辑上在一张表里。 近些年来,随着业务加速成长,数据量突飞猛进,而硬盘容量是有限的,每台服务器上能扩展的硬盘数量也是有限的。致使每隔一段时间都要增加更大容量的存储服务器来应对,而且这个事情一开始是很复杂的,涉及到很多关联项目,即便到现在我们轻车熟路了,每次换服务器的时候依然需要关注它,并且大容量数据库服务器价格昂贵

关于CDC的研究

孤街浪徒 提交于 2019-12-05 17:32:18
为了实现数据库的实时监控,表数据的改变难以捕获,是我们目前所遇到的问题,而这几天我注意到一个可参考的方案,可以解决这一问题,CDC。 概念原理: CDC又称变更数据捕获(Change Data Capture),开启cdc的源表在插入INSERT、更新UPDATE和删除DELETE活动时会插入数据到日志表中。CDC通过 捕获进程将变更数据捕获到变更表中,通过cdc提供的查询函数,我们可以捕获这部分数据。 1.SQL server 2008版本以上的企业版、开发版和评估版中可用; 2.需要开启代理服务(作业)。 3.CDC需要业务库之外的额外的磁盘空间。 4.CDC的表需要主键或者唯一主键。 正式设置: 一, 开启代理服务: windows环境下,可以直接在服务中找到SQL server代理,设为自动打开即可。 Linux环境下: sudo /opt/mssql/bin/mssql-conf set sqlagent.enabled true sudo docker restart <container ID> 二,创建测试环境: 1. /******* Step1:创建示例数据库*******/ USE master GO IF EXISTS(SELECT name FROM sys.databases WHERE name = 'CDC_DB') DROP DATABASE

file:/// to http:// communication via IFrame

帅比萌擦擦* 提交于 2019-12-05 13:43:18
Maybe some of you could have ran into the same problem i did. Imagine you have a file on your machine: file:///c:\test.html And you have an IFrame inside of this file. You need to indicate whether the IFrame contents are loaded or no. Bacically, what we have here: 1. location, href, or any other property is inaccessible from file:/// to http:/// , or backwards. 2. you can't fire event from the browser window in iframe, or in opposite direction, unfortunately. Does this problem have a solution? P.S.: that's not a hack. it's a real problem. making some interaction from local machine with website

MFC中CDC类及其派生类

佐手、 提交于 2019-12-04 14:04:05
CDC类(设备上下文类)用于绘图 CDC派生类  封装的GDI函数 功能说明 CPaintDC类 BeginPaint EndPaint 标准客户区绘图,窗口刷新时不消失。 只在WM_PAINT消息下使用(OnPaint()) CWindowDC类 GetWindowDC ReleaseDC 非客户区绘图,窗口刷新时不消失, 在WM_NCPAINT消息下使用 CClientDC类 GetDC ReleaseDC 临时客户区绘图,窗口刷新时消失, 在任何情况下都可以使用 CMemDC类 CreateCompatibleDC DeleteDC VC6.0未实现,网上可以找到。 CMemDC类: #ifndef __MEMDC_H__ #define __MEMDC_H__ //Author:www.baojy.com class CMemDC :public CDC { CSize m_size; public: void BitRgn( CRgn &rgn, //目标区域 COLORREF crTrans // 透明色 ) { int i = 0,j=0; rgn.CreateRectRgn(0,0,0,0); while(i<m_size.cx) { j = 0; while(j<m_size.cy) { if(GetPixel(i,j) - crTrans) { CRgn r;

CDC is enabled, but cdc.dbo<table-name>_CT table is not being populated

无人久伴 提交于 2019-12-04 10:20:45
I have enabled CDC using the following steps: exec sys.sp_cdc_enable_db; exec sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'table_name', @role_name = N'CDC_Access', @supports_net_changes = 1; I can see that a CT table has been created in the System Tables ; SQL Server Agent is on, and I can see the cdc.db_name_capture job has been created and is running. However, even though the table_name table is being populated, I never see anything in the CT table. I have other tables that have CDC enabled for them in the same database which are being updated, and CDC is capturing data

Sql Server Change Data Capture: Preserving history when adding columns?

孤街醉人 提交于 2019-12-04 05:52:52
When a new column is added to table that is configured for change data capture (cdc), the capture instance table will not have the new column until cdc is disabled and re-enabled for the source table. In the process the existing capture instance is dropped. I thought I could copy existing data out to a temp table and then copy back using the following SQL. However, other CDC meta information, such as the cdc.change_tables.start_lsn, becomes invalid. How can the capture instance history be preserved, using the same capture instance name, if at all? Thanks, Rich /*Change Data Capture Test -

cdc_acm : failed to set dtr/rts - can not communicate with usb cdc device

落花浮王杯 提交于 2019-12-03 14:50:10
I was trying to enumerate usb cdc device using pic24fj128gb206. Device seems to be enumerated properly. But when I connect my device to Linux PC, I am getting the below warning message from kernel. cdc_acm 1-8.1.6.7:1.0: failed to set dtr/rts And this message will repeat when I try to connect using screen. screen /dev/ttyACM9 115200 And I am not able to communicate with my device from PC [ Ubuntu, 14.04 ] When analysing the data using wireshark, it looks like USB communication is fine until, host issues URB_CONTROL_out and the device responds with URB status as Broken Pipe (-EPIPE ) (-32) Can

SaCa CDC产品简介

三世轮回 提交于 2019-12-02 05:32:29
1.1 产品简介 东软SaCa™ CDC实时增量数据捕获平台(以下简称SaCa™ CDC)是一款高度可扩展、可靠和易于配置的异构实时事务复制软件,能够提供异构环境下数据的实时捕捉和交付数据,以独立于创建信息的应用的方式,可实现跨异构数据源、跨网络、实时的捕获与整合数据,提升组织服务的质量和效率,帮助组织实现更智慧的业务决策和对市场形势的更快响应。 SaCa™ CDC使用基于日志的变更数据捕获,最大程度地减少对源系统的影响,SaCa™ CDC几乎不消耗系统资源,又能处理所需的事务量,而不影响源数据库的性能。 1.2 产品定位 SaCa™ CDC帮助用户从原有系统中,包括遗留的数据源,关系型数据库,文本文件系统等,近乎实时的移植,整合,各种应用程序,独立于数据源系统,对源系统极小的性能损耗,全自动化的运转,利用实时监控、断点恢复,DDL自适应、定时启停、异常告警等功能,同步各个系统中的数据,并利用整合后的数据,构建数据仓库、商业智能、对组织内数据全方位的利用。 SaCa™ CDC将作为原始数据与数据应用的快速通道。 1.3 产品特性 产品特性包括:数据捕获、数据发布、管理和监控。 1.3.1 数据捕获 l 基于数据库归档/在线日志实时捕获与同步变更数据; l 支持不同厂商硬件(主机、存储)之间数据库的数据复制; l 支持实时捕获Oracle,Oracle RAC以及达梦数据库变更数据;

工具导航Map