gridfs

Express file upload with multer and gridfs (corrupted file ?)

故事扮演 提交于 2019-12-06 04:10:09
I'm trying to upload a file (an image), the upload is fine, the file is stored in Mongo and have the same content type and same size as the original file, then when I try to download it, the file is corrupted but keeps the same content type (if I upload a pdf, it is recognized as a pdf, if it is a png, it is also recognized, but I can't open them). I don't understand what is wrong with this, it is pretty simple and standard. For the upload from the client, I use angular ng-upload, for the download, it is a simple GET request to the route defined in the code. EDIT : The file is well uploaded on

How can I specify a GridFS bucket?

北慕城南 提交于 2019-12-06 03:20:19
This is my express.js code to upload and download files to GridFS: var fs = require("fs"); var gridStream = require("gridfs-stream"); var mongoose = require("mongoose"); exports.init = function(app, db) { var grid = gridStream(db, mongoose.mongo); app.post("/UploadFile", function(request, response) { var file = request.files.UploadedFile; var meta = request.param("Meta"); var name = request.param("Name"); var stream = grid.createWriteStream( { filename: name, metadata: meta }); fs.createReadStream(file.path) .on("end", function() { response.send({ Success: true }); }) .on("Error", function

Cleaning orphaned files out of GridFS

流过昼夜 提交于 2019-12-06 02:10:16
I have a collection referencing GridFS files, generally 1-2 files per record. The collections are reasonably large - about 705k records in the parent collection, and 790k GridFS files. Over time, there have become a number of orphaned GridFS files - the parent records were deleted, but the referenced files weren't. I'm now attempting to clean the orphaned files out of the GridFS collection. The problem with an approach like suggested here is that combining the 700k records into a single large list of ids results in a Python list that's about 4mb in memory - passing that into a $nin query in

GridFS的应用

感情迁移 提交于 2019-12-06 00:13:39
GridFS介绍 G r i d F S 是 M o n g o D B 提 供 的 用 于 持 久 化 存 储 文 件 的 模 块 , C M S 使 用 M o n g o D B 存 储 数 据 , 使 用 G r i d F S 可 以 快 速 集 成 开 发 。 工作原理: 在 G r i d F S 存 储 文 件 是 将 文 件 分 块 存 储 , 文 件 会 按 照 2 5 6 K B 的 大 小 分 割 成 多 个 块 进 行 存 储 , G r i d F S 使 用 两 个 集 合 ( c o l l e c t i o n ) 存 储 文 件 , 一 个 集 合 是 c h u n k s , 用 于 存 储 文 件 的 二 进 制 数 据 ; 一 个 集 合 是 fi l e s , 用 于 存 储 文 件 的 元 数 据 信 息 ( 文 件 名 称 、 块 大 小 、 上 传 时 间 等 信 息 ) 。 从GridFS中读取文件要对文件的各个块进行组装、合并。 1、存文件 测试代码如下 1 @Test 2 public void testGridFs() throws FileNotFoundException { 3 //要存储的文件 4 File file = new File("d:/index_banner.html"); 5 //定义输入流 6

MongoDB GridFS \"illegal chunk format' exception

戏子无情 提交于 2019-12-05 07:16:49
I've been writing an app in Node.js that stores images in MongoDB's GridFS file system. I've uploaded images through the app, and the images appear to be stored correctly: $ mongofiles -v -d speaker-karaoke get howard-basement-100x115.jpg Tue Jul 17 12:14:16 creating new connection to:127.0.0.1 Tue Jul 17 12:14:16 BackgroundJob starting: ConnectBG Tue Jul 17 12:14:16 connected connection! connected to: 127.0.0.1 done write to: howard-basement-100x115.jpg This grabbed the .jpg out of MongoDB and I was able to open it no problem, so it looks like what I'm uploading is being stored correctly.

Storing a file in mongodb using node.js

风格不统一 提交于 2019-12-05 04:48:17
问题 Hi i need to store a file in mongodb using node.js, the file is placed in my desktop i will have to store it in my mongodb database.I came across something called gridfs but am not sure how to proceed further.Any help will be much appreciated 回答1: While I don't recommend storing big files in Mongo, though it's possible, smaller files would be better. Simply read the file's text (if it's a text file), or binary (if it's in a binary format i.e executable). You can use the fs library to read the

redis、memcached、mongdb

有些话、适合烂在心里 提交于 2019-12-05 04:44:10
>>Memcached Memcached的优点: Memcached可以利用多核优势,单实例吞吐量极高,可以达到几十万QPS(取决于key、value的字节大小以及服务器硬件性能,日常环境中QPS高峰大约在4-6w左右)。适用于最大程度扛量。 支持直接配置为session handle。 Memcached的局限性: 只支持简单的key/value数据结构,不像Redis可以支持丰富的数据类型。 无法进行持久化,数据不能备份,只能用于缓存使用,且重启后数据全部丢失。 无法进行数据同步,不能将MC中的数据迁移到其他MC实例中。 Memcached内存分配采用Slab Allocation机制管理内存,value大小分布差异较大时会造成内存利用率降低,并引发低利用率时依然出现踢出等问题。需要用户注重value设计。 >>Redis Redis的优点: 支持多种数据结构,如 string(字符串)、 list(双向链表)、dict(hash表)、set(集合)、zset(排序set)、hyperloglog(基数估算) 支持持久化操作,可以进行aof及rdb数据持久化到磁盘,从而进行数据备份或数据恢复等操作,较好的防止数据丢失的手段。 支持通过Replication进行数据复制,通过master-slave机制,可以实时进行数据的同步复制,支持多级复制和增量复制,master

Redis、Memcache和MongoDB的区别

江枫思渺然 提交于 2019-12-04 19:47:06
Redis、Memcache和MongoDB的区别 https://www.cnblogs.com/tuyile006/p/6382062.html >>Memcached Memcached的优点: Memcached可以利用多核优势,单实例吞吐量极高,可以达到几十万QPS(取决于key、value的字节大小以及服务器硬件性能,日常环境中QPS高峰大约在4-6w左右)。适用于最大程度扛量。 支持直接配置为session handle。 Memcached的局限性: 只支持简单的key/value数据结构,不像Redis可以支持丰富的数据类型。 无法进行持久化,数据不能备份,只能用于缓存使用,且重启后数据全部丢失。 无法进行数据同步,不能将MC中的数据迁移到其他MC实例中。 Memcached内存分配采用Slab Allocation机制管理内存,value大小分布差异较大时会造成内存利用率降低,并引发低利用率时依然出现踢出等问题。需要用户注重value设计。 >>Redis Redis的优点: 支持多种数据结构,如 string(字符串)、 list(双向链表)、dict(hash表)、set(集合)、zset(排序set)、hyperloglog(基数估算) 支持持久化操作,可以进行aof及rdb数据持久化到磁盘,从而进行数据备份或数据恢复等操作,较好的防止数据丢失的手段。

MongoDB GridFS VS Directly disk IO

时间秒杀一切 提交于 2019-12-04 15:50:08
问题 Use MongoDB GridFS store images and images stored directly on disk What are the advantages? 回答1: Main advantage in my opinion is easy files distribution across multiple servers when system start growing (sharding, replication). Easy improvement reads/writes speed. No need to care where to put new file. Files systems become slow with big amount of small files. 来源: https://stackoverflow.com/questions/6212990/mongodb-gridfs-vs-directly-disk-io

Is it possible to get the md5 hash value of a file being downloaded from a server?

社会主义新天地 提交于 2019-12-04 02:33:21
问题 Is it possible to get the md5 hash value of a file using mongo grid fs being downloaded from a server before storing the file to mongo databse. Now using the code below it stores direcly the data to mongo with the md5 , filename , contenttype etc because I used pipe. Now I want to check the md5 hash of the file first before storing it to the database , I want to get the md5 value of a file first cause i wanna use it to avoid duplicates. Now I did try to log the fileStorage. but i dont think