Find duplicate urls in mongodb [duplicate]

问题

I have a DB with news articles, and I am trying to do a little DB cleaning. I want to find all duplicate documents, and the best way i think to accomplish this by using the url field. My documents are structured as follows:

{
    _id:
    author:
    title:
    description:
    url:
    urlToImage:
    publishedAt:
    content:
    summarization:
    source_id:
}

Any help is greatly appreciated

回答1:

Assuming a collection documents with name (using name instead of url) field consisting duplicate values. I have two aggregations which return some output which can be used to do further processing. I hope you will find this useful.

{ _id: 1, name: "jack" },
{ _id: 2, name: "john" },
{ _id: 3, name: "jim" },
{ _id: 4, name: "john" }
{ _id: 5, name: "john" }, 
{ _id: 6, name: "jim" }

Note that "john" has 3 occurrances and "jim" has 2.

(1) This aggregation returns the names which have duplicates (more than one occurance):

db.collection.aggregate( [
  { 
      $group: { 
          _id: "$name", 
          count: { $sum: 1 }
      } 
  },
  { 
      $group: { 
          _id: "duplicate_names", 
          names: { $push: { $cond: [ { $gt: [ "$count", 1 ] }, "$_id", "$DUMMY" ] } }
      }
  }
] )

The output:

{ "_id" : "duplicate_names", "names" : [ "john", "jim" ] }

(2) The following aggregation just returns the _id field values for the duplicate documents. For example, the name "jim" has _idvalues 3 and 6. The output has only the id's for the duplicate documents, i.e., 6.

db.colection.aggregate( [
  { 
      $group: { 
          _id: "$name", 
          count: { $sum: 1 }, 
          ids: { $push: "$_id" } 
      } 
  },
  { 
      $group: { 
          _id: "duplicate_ids",  
          ids: { $push: { $slice: [ "$ids", 1, 9999 ] } } 
      } 
  },
  { 
      $project: { 
          ids: { 
              $reduce: {
                  input: "$ids", 
                  initialValue: [ ],
                  in: { $concatArrays: [ "$$this", "$$value" ] }
              }
          }
      } 
  }
] )

The output:

{ "_id" : duplicate_ids", "ids" : [ 6, 4, 5 ] }

来源：https://stackoverflow.com/questions/61062508/find-duplicate-urls-in-mongodb

标签

database

mongodb