mirror of
https://github.com/feathersjs/feathers.git
synced 2026-02-01 17:37:38 +00:00
feat(mongodb) aggregation pipeline queries (#2840)
* feat(mongodb) aggregation pipeline queries
Makes queries using the MongoDB Aggregation Pipeline, by default, unless either of these conditions is met:
- `params.mongodb` is provided. This object is for the `collection.find` method, and won’t work for `collection.aggregate`.
- `params.query.$limit` is set to `0`. The aggregation pipeline will not accept 0 as a value for the `$limit` stage.
## Updates to `service.find`
The `find` method has been updated so that queries not meeting the two criteria, above, are created using the aggregation pipeline, which is an array of the document objects representing the stages of the request. Here’s an example query:
```ts
app.service(‘todos’).find({
query: {
$sort: { name: 1 },
$skip: 1,
$limit: 4
}
})
// the internal aggregation request looks like this
const query = collection.aggregate([
{ $match: {} },
{ $sort: { name: 1 } },
{ $skip: 1 },
{ $limit: 4 },
…params.pipeline || []
])
```
## New `pipeline` param
Notice that it’s now possible to pass `params.pipeline`. This means you can set params.pipeline as an array of additional stages in the aggregation pipeline and they will be executed during the same request. This means that it’s now possible to perform `$lookup` and `$unwind` stages alongside the normal Feathers queries.
## Feathers Query with `params.pipeline`
This example shows how to populate the `user` onto a query for `todos`, assuming that each `todo` contains a `userId` property.
```ts
const result = await app.service('todos').find({
query: { $sort: { name: 1 } },
pipeline: [
{
$lookup: {
from: ‘users’,
localField: 'userId',
foreignField: '_id',
as: ‘user’
}
},
{ $unwind: { path: '$user’ } }
],
paginate: false
})
```
In the above example, the `query` is added to the pipeline, first (under the hood) then additional stages are added in the `pipeline` option:
- The `$lookup` stage creates an array called `user` which contains any matches in `todo.userId`, so if `userId` were an array of ids, any matches would be in the `users` array. However, in this example, the `userId` is a single id, so…
- The `$unwind` stage turns the array into a single object.
All stages of the pipeline happen directly on the MongoDB server.
## Custom `.aggregate` requests
It’s still possible to make `.aggregate` requests without the Feathers syntax by directly accessing `service.model`:
```ts
const result = await app.service(‘todos’).model.aggregate([
// …stages go here
]).toArray()
```
* refactor(mongodb): decouple find logic, getModel
Splits the `find` method into two utilities for converting params into MongoDB requests:
- `asFindQuery`
- `asAggregateQuery`
Each of the above methods returns the raw MongoDB request object, without having called `.toArray`, so it’s likely also possible to directly use MongoDB cursors on the returned object, though I’ve not added a test for such a use case.
This PR also optimizes the `find` method to parallelize the `count` query (for pagination) and the `results` query. This makes it so we don’t have to wait for the count before we start querying the results.
One more thing: this PR also combines non-DRY code that was found in every method into a single `getModel` method.
* feat(mongodb) control stages with $feathers stage
A custom `$feathers` stage has been added to our wrapper for the MongoDB aggregation pipeline. It allows you to specify where in the pipeline the Feathers-related stages get injected. This means you can now specify for some pipeline operations to run BEFORE the ones related to the Feathers query.
The `$feathers` stage is an object optionally containing a `handleStages` function. The `handleStages` function receives the Feathers stages and allows them to be modified. It’s a feature that will only be used in complex, modular/layered queries, since it would usually be simpler to just modify the incoming query before passing it to `service.find`.
* refactor(mongodb) rename utils and combine methods
- rename `asFindQuery` to `findRaw`
- rename `asAggregateQuery` to `aggregateRaw`
- combine the `makePipeline` method into the `aggregateRaw` method,
* docs(mongodb) document aggregation queries
* feat(mongodb) remove handleStages from aggregation
This commit is contained in:
Marshall Thompson
GitHub
parent
fe47d3d744
commit
cb70e0778e
@ -11,7 +11,8 @@ outline: deep
|
||||
|
||||
</Badges>
|
||||
|
||||
A [Feathers](https://feathersjs.com) service adapter MongoDB.
|
||||
Support for MongoDB is provided in Feathers via the `@feathersjs/mongodb` database adapter which uses the [MongoDB Client for Node.js](https://www.npmjs.com/package/mongodb). The adapter uses the [MongoDB Aggregation Framework](https://www.mongodb.com/docs/manual/aggregation/), internally, and enables using Feathers' friendly syntax with the full power of [aggregation operators](https://www.mongodb.com/docs/manual/meta/aggregation-quick-reference/). The adapter automatically uses the [MongoDB Query API](https://www.mongodb.com/docs/drivers/node/current/quick-reference/) when you need features like [Collation](https://www.mongodb.com/docs/drivers/node/current/fundamentals/collations/).
|
||||
|
||||
|
||||
```bash
|
||||
$ npm install --save @feathersjs/mongodb
|
||||
@ -23,9 +24,418 @@ The MongoDB adapter implements the [common database adapter API](./common) and [
|
||||
|
||||
</BlockQuote>
|
||||
|
||||
## Setup
|
||||
|
||||
There are two typical setup steps for using `@feathersjs/mongodb` in an application:
|
||||
|
||||
- connect to the database and
|
||||
- setup schemas for types and validation.
|
||||
|
||||
### Connect to the Database
|
||||
|
||||
Before using `@feathersjs/mongodb`, you'll need to create a connection to the database. This example connects to a MongoDB database similar to how the CLI-generated app connects. It uses `app.get('mongodb')` to read the connection string from `@feathersjs/configuration`. The connection string would be something similar to `mongodb://localhost:27017/my-app-dev` for local development or one provided by your database host.
|
||||
|
||||
Once the connection attempt has been started, the code uses `app.set('monodbClient', mongoClient)` to store the connection promise back into the config, which allows it to be looked up when initializing individual services.
|
||||
|
||||
```ts
|
||||
import { MongoClient } from 'mongodb'
|
||||
import type { Db } from 'mongodb'
|
||||
import type { Application } from './declarations'
|
||||
|
||||
export const mongodb = (app: Application) => {
|
||||
const connection = app.get('mongodb')
|
||||
const database = new URL(connection).pathname.substring(1)
|
||||
const mongoClient = MongoClient.connect(connection)
|
||||
.then(client => client.db(database))
|
||||
app.set('mongodbClient', mongoClient)
|
||||
}
|
||||
|
||||
declare module './declarations' {
|
||||
interface Configuration {
|
||||
mongodbClient: Promise<Db>
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Setup the Schema & Types
|
||||
|
||||
To take full advantage of the new TypeScript features in Feathers v5, we can create schema for our service's data types. This example shows how to use `@feathersjs/typebox` to create schemas and types for data and query types. This is the same as generated by the CLI, but the resolvers have been removed for brevity.
|
||||
|
||||
```ts
|
||||
import { Type, querySyntax } from '@feathersjs/typebox'
|
||||
import type { Static } from '@feathersjs/typebox'
|
||||
|
||||
// Main data model schema
|
||||
export const messagesSchema = Type.Object(
|
||||
{
|
||||
_id: Type.String(),
|
||||
text: Type.String(),
|
||||
},
|
||||
{ $id: 'Messages', additionalProperties: false },
|
||||
)
|
||||
export type Messages = Static<typeof messagesSchema>
|
||||
|
||||
// Schema for creating new entries
|
||||
export const messagesDataSchema = Type.Pick(messagesSchema, ['name'],
|
||||
{ $id: 'MessagesData', additionalProperties: false },
|
||||
)
|
||||
export type MessagesData = Static<typeof messagesDataSchema>
|
||||
|
||||
// Schema for allowed query properties
|
||||
export const messagesQueryProperties = Type.Pick(messagesSchema, ['_id', 'name'], { additionalProperties: false })
|
||||
export const messagesQuerySchema = querySyntax(messagesQueryProperties)
|
||||
export type MessagesQuery = Static<typeof messagesQuerySchema>
|
||||
```
|
||||
|
||||
### Schemas vs MongoDB Validation
|
||||
|
||||
In Feathers v5 (Dove) we added support for Feathers Schema, which performs validation and provides TypeScript types. Recent versions of MongoDB include support for JSON Schema validation at the database server. Most applications will benefit from using Feathers Schema for the following reasons.
|
||||
|
||||
- Feathers Schema's TypeBox integration makes JSON Schema so much easier to read and write.
|
||||
- You get TypeScript types for free once you've defined your validation rules, using `TypeBox` or `json-schema-to-ts`
|
||||
- All configuration is done in code, reducing the time to prototype/setup/launch. With MongoDB's built-in validation, you essentially add another "DevOps" step before you can use the database.
|
||||
- Support for JSON Schema draft 7. MongoDB's validation is based on version 4.
|
||||
- Feathers Schema don't have to wait for a round-trip to the database to validate the data.
|
||||
- Feathers Schema can be used in the browser or on the server.
|
||||
|
||||
MongoDB's built-in validation does have built-in support for `bsonType` to force data to be stored as a specific BSON type once it passes validation. There's nothing keeping you from using both solutions together. It's not a use case that's documented, here.
|
||||
|
||||
## API
|
||||
|
||||
### `service([options])`
|
||||
### `service(options)`
|
||||
|
||||
Returns a new service instance initialized with the given options. The following example extends the `MongoDBService` class using the schema examples from earlier on this page. It then uses the `mongodbClient` from the app configuration and provides it to the `Model` option, which is passed to the new `MessagesService`.
|
||||
|
||||
```ts
|
||||
import type { Params } from '@feathersjs/feathers'
|
||||
import { MongoDBService } from '@feathersjs/mongodb'
|
||||
import type { MongoDBAdapterParams, MongoDBAdapterOptions } from '@feathersjs/mongodb'
|
||||
|
||||
import type { Application } from '../../declarations'
|
||||
import type { Messages, MessagesData, MessagesQuery } from './messages.schema'
|
||||
|
||||
export interface MessagesParams extends MongoDBAdapterParams<MessagesQuery> {}
|
||||
|
||||
export class MessagesService<ServiceParams extends Params = MessagesParams>
|
||||
extends MongoDBService<Messages, MessagesData, ServiceParams> {}
|
||||
|
||||
export const messages = (app: Application) => {
|
||||
const options: MongoDBAdapterOptions = {
|
||||
paginate: app.get('paginate'),
|
||||
Model: app.get('mongodbClient').then((db) => db.collection('messages')),
|
||||
}
|
||||
app.use('messages', new MessagesService(options), {
|
||||
methods: ['find', 'get', 'create', 'update', 'patch', 'remove'],
|
||||
events: [],
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
Here's an overview of the `options` object:
|
||||
|
||||
**Options:**
|
||||
|
||||
- `Model {Promise<MongoDBCollection>}` (**required**) - The MongoDB collection instance
|
||||
- `id {string}` (*optional*, default: `'_id'`) - The name of the id field property. By design, MongoDB will always add an `_id` property.
|
||||
- `disableObjectify {boolean}` (*optional*, default `false`) - This will disable the objectify of the id field if you want to use normal strings
|
||||
- `events {string[]}` (*optional*) - A list of [custom service events](/api/events.html#custom-events) sent by this service
|
||||
- `paginate {Object}` (*optional*) - A [pagination object](/api/databases/common.html#pagination) containing a `default` and `max` page size
|
||||
- `filters {Object}` (*optional*) - An object of additional filter parameters to allow (e..g `{ $customQueryOperator: true }`). See [Filters](/api/databases/querying.md#filters)
|
||||
- `operators {string[]}` (*optional*) - A list of additional query parameters to allow (e..g `[ '$regex', '$geoNear' ]`) See [Operators](/api/databases/querying.md#operators)
|
||||
- `multi {string[]|true}` (*optional*) - Allow `create` with arrays and `update` and `remove` with `id` `null` to change multiple items. Can be `true` for all methods or an array of allowed methods (e.g. `[ 'remove', 'create' ]`)
|
||||
- `useEstimatedDocumentCount {boolean}` (*optional*, default `false`) - If `true` document counting will rely on `estimatedDocumentCount` instead of `countDocuments`
|
||||
|
||||
### `aggregateRaw(params)`
|
||||
|
||||
The `find` method has been split into separate utilities for converting params into different types of MongoDB requests. By default, requests are processed by this method and are run through the MongoDB Aggregation Pipeline. This method returns a raw MongoDB Cursor object, which can be used to perform custom pagination or in custom server scripts, if desired.
|
||||
|
||||
### `findRaw(params)`
|
||||
|
||||
The `find` method has been split into separate utilities for converting params into different types of MongoDB requests. When `params.mongodb` is used, the `findRaw` method is used to retrieve data using `params.mongodb` as the `FindOptions` object. This method returns a raw MongoDB Cursor object, which can be used to perform custom pagination or in custom server scripts, if desired.
|
||||
|
||||
### `makeFeathersPipeline(params)`
|
||||
|
||||
`makeFeathersPipeline` takes a set of Feathers params and converts them to a pipeline array, ready to pass to `collection.aggregate`. This utility comprises the bulk of the `aggregateRaw` functionality, but does not use `params.pipeline`.
|
||||
|
||||
### Custom Params
|
||||
|
||||
The `@feathersjs/mongodb` adapter utilizes two custom params which control adapter-specific features: `params.pipeline` and `params.mongodb`.
|
||||
|
||||
#### params.pipeline
|
||||
|
||||
This is a test
|
||||
|
||||
#### params.mongodb
|
||||
|
||||
When making a [service method](/api/services.md) call, `params` can contain an `mongodb` property (for example, `{upsert: true}`) which allows modifying the options used to run the MongoDB query.
|
||||
|
||||
The adapter will automatically switch to use the MongoClient's`collection.find` method when you use `params.mongodb`.
|
||||
|
||||
## Aggregation Pipeline
|
||||
|
||||
In Feathers v5 Dove, we added support for the full power of MongoDB's Aggregation Framework and blends it seamlessly with the familiar Feathers Query syntax. All `find` queries now use the Aggregation Framework, by default.
|
||||
|
||||
The Aggregation Framework is accessed through the mongoClient's `collection.aggregate` method, which accepts an array of "stages". Each stage contains an operator which describes an operation to apply to the previous step's data. Each stage applies the operation to the results of the previous step. It’s now possible to perform any of the [Aggregation Stages](https://www.mongodb.com/docs/upcoming/reference/operator/aggregation-pipeline/) like `$lookup` and `$unwind`, integration with the normal Feathers queries.
|
||||
|
||||
Here's how it works with the operators that match the Feathers Query syntax. Let's convert the following Feathers query:
|
||||
|
||||
```ts
|
||||
const query = {
|
||||
text: { $regex: 'feathersjs', $options: 'igm' },
|
||||
$sort: { createdAt: -1 },
|
||||
$skip: 0,
|
||||
$limit: 10,
|
||||
}
|
||||
```
|
||||
|
||||
The above query looks like this when converted to aggregation pipeline stages:
|
||||
|
||||
```ts
|
||||
[
|
||||
// returns the set of records containing the word "feathersjs"
|
||||
{ $match: { text: { $regex: 'feathersjs', $options: 'igm' } } },
|
||||
// Sorts the results of the previous step by newest messages, first.
|
||||
{ $sort: { createdAt: -1 } },
|
||||
// Skips the first 20 records of the previous step
|
||||
{ $skip: 20 },
|
||||
// returns the next 10 records
|
||||
{ $limit: 10 },
|
||||
]
|
||||
```
|
||||
|
||||
### Pipeline Queries
|
||||
|
||||
You can use the `params.pipeline` array to append additional stages to the query. This next example uses the `$lookup` operator together with the `$unwind` operator to populate a `user` attribute onto each message based on the message's `userId` property.
|
||||
|
||||
```ts
|
||||
const result = await app.service('messages').find({
|
||||
query: { $sort: { name: 1 } },
|
||||
pipeline: [
|
||||
{
|
||||
$lookup: {
|
||||
from: 'users',
|
||||
localField: 'userId',
|
||||
foreignField: '_id',
|
||||
as: 'user'
|
||||
}
|
||||
},
|
||||
{ $unwind: { path: '$user' } }
|
||||
],
|
||||
paginate: false
|
||||
})
|
||||
```
|
||||
|
||||
### Aggregation Stages
|
||||
|
||||
In the example, above, the `query` is added to the pipeline, first. Then additional stages are added in the `pipeline` option:
|
||||
|
||||
- The `$lookup` stage creates an array called `user` which contains any matches in `message.userId`, so if `userId` were an array of ids, any matches would be in the `users` array. However, in this example, the `userId` is a single id, so...
|
||||
- The `$unwind` stage turns the array into a single `user` object.
|
||||
|
||||
The above is like doing a join, but without the data transforming overhead like you'd get with an SQL JOIN. If you have properly applied index to your MongoDB collections, the operation will typically execute extremely fast for a reasonable amount of data.
|
||||
|
||||
A couple of other notable query stages:
|
||||
|
||||
- `$graphLookup` lets you recursively pull in a tree of data from a single collection.
|
||||
- `$search` lets you do full-text search on fields
|
||||
|
||||
All stages of the pipeline happen directly on the MongoDB server.
|
||||
|
||||
Read through the full list of supported stages [in the MongoDB documentation](https://www.mongodb.com/docs/upcoming/reference/operator/aggregation-pipeline/).
|
||||
|
||||
### The `$feathers` Stage
|
||||
|
||||
The previous section showed how to append stages to a query using `params.pipeline`. Well, `params.pipeline` also supports a custom `$feathers` operator/stage which allows you to specify exactly where in the pipeline the Feathers Query gets injected.
|
||||
|
||||
### Example: Proxy Permissions
|
||||
|
||||
Imagine a scenario where you want to query the `pages` a user can edit by referencing a `permissions` collection to find out which pages the user can actually edit. Each record in the `permissions` record has a `userId` and a `pageId`. So we need to find and return only the pages to which the user has access by calling `GET /pages` from the client.
|
||||
|
||||
We could put the following query in a hook to pull the correct `pages` from the database in a single query THROUGH the permissions collection. Remember, the request is coming in on the `pages` service, but we're going to query for pages `through` the permissions collection. Assume we've already authenticated the user, so the user will be found at `context.params.user`.
|
||||
|
||||
```ts
|
||||
// Assume this query on the client
|
||||
const pages = await app.service('pages').find({ query: {} })
|
||||
|
||||
// And put this query in a hook to populate pages "through" the permissions collection
|
||||
const result = await app.service('permissions').find({
|
||||
query: {},
|
||||
pipeline: [
|
||||
// query all permissions records which apply to the current user
|
||||
{
|
||||
$match: { userId: context.params.user._id }
|
||||
},
|
||||
// populate the pageId onto each `permission` record, as an array containing one page
|
||||
{
|
||||
$lookup: {
|
||||
from: 'pages',
|
||||
localField: 'pageId',
|
||||
foreignField: '_id',
|
||||
as: 'page',
|
||||
},
|
||||
},
|
||||
// convert the `page` array into an object, so now we have an array of permissions with permission.page on each.
|
||||
{
|
||||
$unwind: { path: '$page' }
|
||||
},
|
||||
// Add a permissionId to each page
|
||||
{
|
||||
$addFields: {
|
||||
'page.permissionId': '$_id'
|
||||
}
|
||||
},
|
||||
// discard the permission and only keep the populated `page`, and bring it top level in the array
|
||||
{
|
||||
$replaceRoot: { newRoot: '$page' }
|
||||
},
|
||||
// apply the feathers query stages to the aggregation pipeline.
|
||||
// now the query will apply to the pages, since we made the pages top level in the previous step.
|
||||
{
|
||||
$feathers: {}
|
||||
},
|
||||
],
|
||||
paginate: false
|
||||
})
|
||||
```
|
||||
|
||||
Notice the `$feathers` stage in the above example. It will apply the query to that stage in the pipeline, which allows the query to apply to pages even though we had to make the query through the `permissions` service.
|
||||
|
||||
If we were to express the above query with JavaScript, the final result would the same as with the following example:
|
||||
|
||||
```ts
|
||||
// perform a db query to get the permissions
|
||||
const permissions = await context.app.service('permissions').find({
|
||||
query: {
|
||||
userId: context.params.user._id
|
||||
},
|
||||
paginate: false,
|
||||
})
|
||||
// make a list of pageIds
|
||||
const pageIds = permissions.map((permission) => permission.pageId)
|
||||
// perform a db query to get the pages with matching `_id`
|
||||
const pages = await context.app.service('pages').find({
|
||||
query: {
|
||||
_id: {
|
||||
$in: pageIds
|
||||
}
|
||||
},
|
||||
paginate: false
|
||||
})
|
||||
// key the permissions by pageId for easy lookup
|
||||
const permissionsByPageId = permissions.reduce((byId, current) => {
|
||||
byId[current.pageId] = current
|
||||
return byId
|
||||
}, {})
|
||||
// Add the permissionId to each `page` record.
|
||||
const pagesWithPermissionId = pages.map((page) => {
|
||||
page.permissionId = permissionByPageId[page._id]._id
|
||||
return page
|
||||
})
|
||||
// And now apply the original query, whatever the client may have sent, to the pages.
|
||||
// It might require another database query
|
||||
```
|
||||
|
||||
Both examples look a bit complex, but te one using aggregation stages will be much quicker because all stages run in the database server. It will also be quicker because it all happens in a single database query!
|
||||
|
||||
One more obstacle for using JavaScript this way is that if the user's query changed (from the front end), we would likely be required to edit multiple different parts of the JS logic in order to correctly display results. With the pipeline example, above, the query is very cleanly applied.
|
||||
|
||||
## Transactions
|
||||
|
||||
You can utilize [MongoDB Transactions](https://docs.mongodb.com/manual/core/transactions/) by passing a `session` with the `params.mongodb`:
|
||||
|
||||
```js
|
||||
import { ObjectID } from 'mongodb'
|
||||
|
||||
export default async app => {
|
||||
app.use('/fooBarService', {
|
||||
async create(data) {
|
||||
// assumes you have access to the mongoClient via your app state
|
||||
let session = app.mongoClient.startSession()
|
||||
try {
|
||||
await session.withTransaction(async () => {
|
||||
let fooID = new ObjectID()
|
||||
let barID = new ObjectID()
|
||||
app.service('fooService').create(
|
||||
{
|
||||
...data,
|
||||
_id: fooID,
|
||||
bar: barID,
|
||||
},
|
||||
{ mongodb: { session } },
|
||||
)
|
||||
app.service('barService').create(
|
||||
{
|
||||
...data,
|
||||
_id: barID
|
||||
foo: fooID
|
||||
},
|
||||
{ mongodb: { session } },
|
||||
)
|
||||
})
|
||||
} finally {
|
||||
await session.endSession()
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## Collation
|
||||
|
||||
This adapter includes support for [collation and case insensitive indexes available in MongoDB v3.4](https://docs.mongodb.com/manual/release-notes/3.4/#collation-and-case-insensitive-indexes). Collation parameters may be passed using the special `collation` parameter to the `find()`, `remove()` and `patch()` methods.
|
||||
|
||||
### Example: Patch records with case-insensitive alphabetical ordering
|
||||
|
||||
The example below would patch all student records with grades of `'c'` or `'C'` and above (a natural language ordering). Without collations this would not be as simple, since the comparison `{ $gt: 'c' }` would not include uppercase grades of `'C'` because the code point of `'C'` is less than that of `'c'`.
|
||||
|
||||
```js
|
||||
const patch = { shouldStudyMore: true };
|
||||
const query = { grade: { $gte: 'c' } };
|
||||
const collation = { locale: 'en', strength: 1 };
|
||||
students.patch(null, patch, { query, collation }).then( ... );
|
||||
```
|
||||
|
||||
### Example: Find records with a case-insensitive search
|
||||
|
||||
Similar to the above example, this would find students with a grade of `'c'` or greater, in a case-insensitive manner.
|
||||
|
||||
```js
|
||||
const query = { grade: { $gte: 'c' } };
|
||||
const collation = { locale: 'en', strength: 1 };
|
||||
students.find({ query, collation }).then( ... );
|
||||
```
|
||||
|
||||
For more information on MongoDB's collation feature, visit the [collation reference page](https://docs.mongodb.com/manual/reference/collation/).
|
||||
|
||||
## Querying
|
||||
|
||||
Additionally to the [common querying mechanism](https://docs.feathersjs.com/api/databases/querying.html) this adapter also supports [MongoDB's query syntax](https://docs.mongodb.com/v3.2/tutorial/query-documents/) and the `update` method also supports MongoDB [update operators](https://docs.mongodb.com/v3.2/reference/operator/update/).
|
||||
|
||||
> **Important:** External query values through HTTP URLs may have to be converted to the same type stored in MongoDB in a before [hook](https://docs.feathersjs.com/api/hooks.html) otherwise no matches will be found. Websocket requests will maintain the correct format if it is supported by JSON (ObjectIDs and dates still have to be converted).
|
||||
|
||||
For example, an `age` (which is a number) a hook like this can be used:
|
||||
|
||||
```js
|
||||
const ObjectID = require('mongodb').ObjectID
|
||||
|
||||
app.service('users').hooks({
|
||||
before: {
|
||||
find(context) {
|
||||
const { query = {} } = context.params
|
||||
|
||||
if (query.age !== undefined) {
|
||||
query.age = parseInt(query.age, 10)
|
||||
}
|
||||
|
||||
context.params.query = query
|
||||
|
||||
return Promise.resolve(context)
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
Which will allows queries like `/users?_id=507f1f77bcf86cd799439011&age=25`.
|
||||
|
||||
## Validating MongoDB Data
|
||||
|
||||
@ -155,7 +565,7 @@ export const connectionsResultSchema = schema(
|
||||
)
|
||||
```
|
||||
|
||||
## Common Pitfalls
|
||||
## Common Mistakes
|
||||
|
||||
Here are a couple of errors you might run into while using validators.
|
||||
|
||||
@ -173,206 +583,3 @@ You'll see an error like `Error: unknown format "date-time" ignored in schema at
|
||||
|
||||
- You're attempting to use a formatter not built into AJV.
|
||||
- You fail to [Pass the Custom AJV Instance to every `schema`](#pass-the-custom-ajv-instance-to-schema). If you're using a custom AJV instance, be sure to provide it to **every** place where you call `schema()`.
|
||||
|
||||
## API
|
||||
|
||||
### `service(options)`
|
||||
|
||||
Returns a new service instance initialized with the given options. `Model` has to be a MongoDB collection.
|
||||
|
||||
```js
|
||||
const MongoClient = require('mongodb').MongoClient
|
||||
const service = require('feathers-mongodb')
|
||||
|
||||
MongoClient.connect('mongodb://localhost:27017/feathers').then((client) => {
|
||||
app.use(
|
||||
'/messages',
|
||||
service({
|
||||
Model: client.db('feathers').collection('messages')
|
||||
})
|
||||
)
|
||||
app.use('/messages', service({ Model, id, events, paginate }))
|
||||
})
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
- `Model` (**required**) - The MongoDB collection instance
|
||||
- `id` (_optional_, default: `'_id'`) - The name of the id field property. By design, MongoDB will always add an `_id` property.
|
||||
- `disableObjectify` (_optional_, default `false`) - This will disable the objectify of the id field if you want to use normal strings
|
||||
- `events` (_optional_) - A list of [custom service events](https://docs.feathersjs.com/api/events.html#custom-events) sent by this service
|
||||
- `paginate` (_optional_) - A [pagination object](https://docs.feathersjs.com/api/databases/common.html#pagination) containing a `default` and `max` page size
|
||||
- `whitelist` (_optional_) - A list of additional query parameters to allow (e..g `[ '$regex', '$geoNear' ]`)
|
||||
- `multi` (_optional_) - Allow `create` with arrays and `update` and `remove` with `id` `null` to change multiple items. Can be `true` for all methods or an array of allowed methods (e.g. `[ 'remove', 'create' ]`)
|
||||
- `useEstimatedDocumentCount` (_optional_, default `false`) - If `true` document counting will rely on `estimatedDocumentCount` instead of `countDocuments`
|
||||
|
||||
### params.mongodb
|
||||
|
||||
When making a [service method](https://docs.feathersjs.com/api/services.html) call, `params` can contain an `mongodb` property (for example, `{upsert: true}`) which allows to modify the options used to run the MongoDB query.
|
||||
|
||||
#### Transactions
|
||||
|
||||
You can utilized a [MongoDB Transactions](https://docs.mongodb.com/manual/core/transactions/) by passing a `session` with the `params.mongodb`:
|
||||
|
||||
```js
|
||||
import { ObjectID } from 'mongodb'
|
||||
|
||||
export default async app => {
|
||||
app.use('/fooBarService', {
|
||||
async create(data) {
|
||||
// assumes you have access to the mongoClient via your app state
|
||||
let session = app.mongoClient.startSession()
|
||||
try {
|
||||
await session.withTransaction(async () => {
|
||||
let fooID = new ObjectID()
|
||||
let barID = new ObjectID()
|
||||
app.service('fooService').create(
|
||||
{
|
||||
...data,
|
||||
_id: fooID,
|
||||
bar: barID,
|
||||
},
|
||||
{ mongodb: { session } },
|
||||
)
|
||||
app.service('barService').create(
|
||||
{
|
||||
...data,
|
||||
_id: barID
|
||||
foo: fooID
|
||||
},
|
||||
{ mongodb: { session } },
|
||||
)
|
||||
})
|
||||
} finally {
|
||||
await session.endSession()
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
Here is an example of a Feathers server with a `messages` endpoint that writes to the `feathers` database and the `messages` collection.
|
||||
|
||||
```
|
||||
$ npm install @feathersjs/feathers @feathersjs/errors @feathersjs/express @feathersjs/socketio feathers-mongodb mongodb
|
||||
```
|
||||
|
||||
In `app.js`:
|
||||
|
||||
```js
|
||||
const feathers = require('@feathersjs/feathers')
|
||||
const express = require('@feathersjs/express')
|
||||
const socketio = require('@feathersjs/socketio')
|
||||
|
||||
const MongoClient = require('mongodb').MongoClient
|
||||
const service = require('feathers-mongodb')
|
||||
|
||||
// Create an Express compatible Feathers application instance.
|
||||
const app = express(feathers())
|
||||
// Turn on JSON parser for REST services
|
||||
app.use(express.json())
|
||||
// Turn on URL-encoded parser for REST services
|
||||
app.use(express.urlencoded({ extended: true }))
|
||||
// Enable REST services
|
||||
app.configure(express.rest())
|
||||
// Enable Socket.io
|
||||
app.configure(socketio())
|
||||
|
||||
// Connect to the db, create and register a Feathers service.
|
||||
app.use(
|
||||
'/messages',
|
||||
service({
|
||||
paginate: {
|
||||
default: 2,
|
||||
max: 4
|
||||
}
|
||||
})
|
||||
)
|
||||
|
||||
// A basic error handler, just like Express
|
||||
app.use(express.errorHandler())
|
||||
|
||||
// Connect to your MongoDB instance(s)
|
||||
MongoClient.connect('mongodb://localhost:27017/feathers')
|
||||
.then(function (client) {
|
||||
// Set the model now that we are connected
|
||||
app.service('messages').Model = client.db('feathers').collection('messages')
|
||||
|
||||
// Now that we are connected, create a dummy Message
|
||||
app
|
||||
.service('messages')
|
||||
.create({
|
||||
text: 'Message created on server'
|
||||
})
|
||||
.then((message) => console.log('Created message', message))
|
||||
})
|
||||
.catch((error) => console.error(error))
|
||||
|
||||
// Start the server.
|
||||
const port = 3030
|
||||
|
||||
app.listen(port, () => {
|
||||
console.log(`Feathers server listening on port ${port}`)
|
||||
})
|
||||
```
|
||||
|
||||
## Querying
|
||||
|
||||
Additionally to the [common querying mechanism](https://docs.feathersjs.com/api/databases/querying.html) this adapter also supports [MongoDB's query syntax](https://docs.mongodb.com/v3.2/tutorial/query-documents/) and the `update` method also supports MongoDB [update operators](https://docs.mongodb.com/v3.2/reference/operator/update/).
|
||||
|
||||
> **Important:** External query values through HTTP URLs may have to be converted to the same type stored in MongoDB in a before [hook](https://docs.feathersjs.com/api/hooks.html) otherwise no matches will be found. Websocket requests will maintain the correct format if it is supported by JSON (ObjectIDs and dates still have to be converted).
|
||||
|
||||
For example, an `age` (which is a number) a hook like this can be used:
|
||||
|
||||
```js
|
||||
const ObjectID = require('mongodb').ObjectID
|
||||
|
||||
app.service('users').hooks({
|
||||
before: {
|
||||
find(context) {
|
||||
const { query = {} } = context.params
|
||||
|
||||
if (query.age !== undefined) {
|
||||
query.age = parseInt(query.age, 10)
|
||||
}
|
||||
|
||||
context.params.query = query
|
||||
|
||||
return Promise.resolve(context)
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
Which will allows queries like `/users?_id=507f1f77bcf86cd799439011&age=25`.
|
||||
|
||||
## Collation Support
|
||||
|
||||
This adapter includes support for [collation and case insensitive indexes available in MongoDB v3.4](https://docs.mongodb.com/manual/release-notes/3.4/#collation-and-case-insensitive-indexes). Collation parameters may be passed using the special `collation` parameter to the `find()`, `remove()` and `patch()` methods.
|
||||
|
||||
### Example: Patch records with case-insensitive alphabetical ordering
|
||||
|
||||
The example below would patch all student records with grades of `'c'` or `'C'` and above (a natural language ordering). Without collations this would not be as simple, since the comparison `{ $gt: 'c' }` would not include uppercase grades of `'C'` because the code point of `'C'` is less than that of `'c'`.
|
||||
|
||||
```js
|
||||
const patch = { shouldStudyMore: true };
|
||||
const query = { grade: { $gte: 'c' } };
|
||||
const collation = { locale: 'en', strength: 1 };
|
||||
students.patch(null, patch, { query, collation }).then( ... );
|
||||
```
|
||||
|
||||
### Example: Find records with a case-insensitive search
|
||||
|
||||
Similar to the above example, this would find students with a grade of `'c'` or greater, in a case-insensitive manner.
|
||||
|
||||
```js
|
||||
const query = { grade: { $gte: 'c' } };
|
||||
const collation = { locale: 'en', strength: 1 };
|
||||
students.find({ query, collation }).then( ... );
|
||||
```
|
||||
|
||||
For more information on MongoDB's collation feature, visit the [collation reference page](https://docs.mongodb.com/manual/reference/collation/).
|
||||
|
||||
## Search
|
||||
|
||||
@ -6,7 +6,8 @@ import {
|
||||
InsertOneOptions,
|
||||
DeleteOptions,
|
||||
CountDocumentsOptions,
|
||||
ReplaceOptions
|
||||
ReplaceOptions,
|
||||
Document
|
||||
} from 'mongodb'
|
||||
import { NotFound } from '@feathersjs/errors'
|
||||
import { _ } from '@feathersjs/commons'
|
||||
@ -29,6 +30,7 @@ export interface MongoDBAdapterOptions extends AdapterServiceOptions {
|
||||
|
||||
export interface MongoDBAdapterParams<Q = AdapterQuery>
|
||||
extends AdapterParams<Q, Partial<MongoDBAdapterOptions>> {
|
||||
pipeline?: Document[]
|
||||
mongodb?:
|
||||
| BulkWriteOptions
|
||||
| FindOptions
|
||||
@ -86,6 +88,68 @@ export class MongoDbAdapter<
|
||||
}
|
||||
}
|
||||
|
||||
getModel(params: P) {
|
||||
const { Model } = this.getOptions(params)
|
||||
return Promise.resolve(Model)
|
||||
}
|
||||
|
||||
async findRaw(params: P) {
|
||||
const { filters, query } = this.filterQuery(null, params)
|
||||
const model = await this.getModel(params)
|
||||
const q = model.find(query, { ...params.mongodb })
|
||||
|
||||
if (filters.$select !== undefined) {
|
||||
q.project(this.getSelect(filters.$select))
|
||||
}
|
||||
|
||||
if (filters.$sort !== undefined) {
|
||||
q.sort(filters.$sort)
|
||||
}
|
||||
|
||||
if (filters.$skip !== undefined) {
|
||||
q.skip(filters.$skip)
|
||||
}
|
||||
|
||||
if (filters.$limit !== undefined) {
|
||||
q.limit(filters.$limit)
|
||||
}
|
||||
|
||||
return q
|
||||
}
|
||||
|
||||
async aggregateRaw(params: P) {
|
||||
const model = await this.getModel(params)
|
||||
const pipeline = params.pipeline || []
|
||||
const index = pipeline.findIndex((stage: Document) => stage.$feathers)
|
||||
const before = index >= 0 ? pipeline.slice(0, index) : []
|
||||
const feathersPipeline = this.makeFeathersPipeline(params)
|
||||
const after = index >= 0 ? pipeline.slice(index + 1) : pipeline
|
||||
|
||||
return model.aggregate([...before, ...feathersPipeline, ...after])
|
||||
}
|
||||
|
||||
makeFeathersPipeline(params: P) {
|
||||
const { filters, query } = this.filterQuery(null, params)
|
||||
const pipeline: Document[] = [{ $match: query }]
|
||||
|
||||
if (filters.$select !== undefined) {
|
||||
pipeline.push({ $project: this.getSelect(filters.$select) })
|
||||
}
|
||||
|
||||
if (filters.$sort !== undefined) {
|
||||
pipeline.push({ $sort: filters.$sort })
|
||||
}
|
||||
|
||||
if (filters.$skip !== undefined) {
|
||||
pipeline.push({ $skip: filters.$skip })
|
||||
}
|
||||
|
||||
if (filters.$limit !== undefined) {
|
||||
pipeline.push({ $limit: filters.$limit })
|
||||
}
|
||||
return pipeline
|
||||
}
|
||||
|
||||
getSelect(select: string[] | { [key: string]: number }) {
|
||||
if (Array.isArray(select)) {
|
||||
return select.reduce<{ [key: string]: number }>(
|
||||
@ -121,7 +185,6 @@ export class MongoDbAdapter<
|
||||
}
|
||||
|
||||
async $get(id: Id, params: P = {} as P): Promise<T> {
|
||||
const { Model } = this.getOptions(params)
|
||||
const {
|
||||
query,
|
||||
filters: { $select }
|
||||
@ -139,7 +202,7 @@ export class MongoDbAdapter<
|
||||
...projection
|
||||
}
|
||||
|
||||
return Promise.resolve(Model)
|
||||
return this.getModel(params)
|
||||
.then((model) => model.findOne(query, findOptions))
|
||||
.then((data) => {
|
||||
if (data == null) {
|
||||
@ -155,44 +218,35 @@ export class MongoDbAdapter<
|
||||
async $find(params?: P & { paginate: false }): Promise<T[]>
|
||||
async $find(params?: P): Promise<Paginated<T> | T[]>
|
||||
async $find(params: P = {} as P): Promise<Paginated<T> | T[]> {
|
||||
const { paginate, useEstimatedDocumentCount } = this.getOptions(params)
|
||||
const { filters, query } = this.filterQuery(null, params)
|
||||
const { paginate, Model, useEstimatedDocumentCount } = this.getOptions(params)
|
||||
const findOptions = { ...params.mongodb }
|
||||
const model = await Promise.resolve(Model)
|
||||
const q = model.find(query, findOptions)
|
||||
const useAggregation = !params.mongodb && filters.$limit !== 0
|
||||
|
||||
if (filters.$select !== undefined) {
|
||||
q.project(this.getSelect(filters.$select))
|
||||
const countDocuments = async () => {
|
||||
if (paginate && paginate.default) {
|
||||
const model = await this.getModel(params)
|
||||
if (useEstimatedDocumentCount && typeof model.estimatedDocumentCount === 'function') {
|
||||
return model.estimatedDocumentCount()
|
||||
} else {
|
||||
return model.countDocuments(query, { ...params.mongodb })
|
||||
}
|
||||
}
|
||||
return Promise.resolve(0)
|
||||
}
|
||||
|
||||
if (filters.$sort !== undefined) {
|
||||
q.sort(filters.$sort)
|
||||
}
|
||||
const [request, total] = await Promise.all([
|
||||
useAggregation ? this.aggregateRaw(params) : this.findRaw(params),
|
||||
countDocuments()
|
||||
])
|
||||
|
||||
if (filters.$limit !== undefined) {
|
||||
q.limit(filters.$limit)
|
||||
}
|
||||
|
||||
if (filters.$skip !== undefined) {
|
||||
q.skip(filters.$skip)
|
||||
}
|
||||
|
||||
const runQuery = async (total: number) => ({
|
||||
const page = {
|
||||
total,
|
||||
limit: filters.$limit,
|
||||
skip: filters.$skip || 0,
|
||||
data: filters.$limit === 0 ? [] : ((await q.toArray()) as any as T[])
|
||||
})
|
||||
|
||||
if (paginate && paginate.default) {
|
||||
if (useEstimatedDocumentCount && typeof model.estimatedDocumentCount === 'function') {
|
||||
return model.estimatedDocumentCount().then(runQuery)
|
||||
}
|
||||
|
||||
return model.countDocuments(query, findOptions).then(runQuery)
|
||||
data: filters.$limit === 0 ? [] : ((await request.toArray()) as any as T[])
|
||||
}
|
||||
|
||||
return runQuery(0).then((page) => page.data)
|
||||
return paginate && paginate.default ? page : page.data
|
||||
}
|
||||
|
||||
async $create(data: D, params?: P): Promise<T>
|
||||
@ -200,8 +254,7 @@ export class MongoDbAdapter<
|
||||
async $create(data: D | D[], _params?: P): Promise<T | T[]>
|
||||
async $create(data: D | D[], params: P = {} as P): Promise<T | T[]> {
|
||||
const writeOptions = params.mongodb
|
||||
const { Model } = this.getOptions(params)
|
||||
const model = await Promise.resolve(Model)
|
||||
const model = await this.getModel(params)
|
||||
const setId = (item: any) => {
|
||||
const entry = Object.assign({}, item)
|
||||
|
||||
@ -236,8 +289,7 @@ export class MongoDbAdapter<
|
||||
async $patch(id: NullableId, data: Partial<D>, _params?: P): Promise<T | T[]>
|
||||
async $patch(id: NullableId, _data: Partial<D>, params: P = {} as P): Promise<T | T[]> {
|
||||
const data = this.normalizeId(id, _data)
|
||||
const { Model } = this.getOptions(params)
|
||||
const model = await Promise.resolve(Model)
|
||||
const model = await this.getModel(params)
|
||||
const {
|
||||
query,
|
||||
filters: { $select }
|
||||
@ -282,8 +334,7 @@ export class MongoDbAdapter<
|
||||
}
|
||||
|
||||
async $update(id: Id, data: D, params: P = {} as P): Promise<T> {
|
||||
const { Model } = this.getOptions(params)
|
||||
const model = await Promise.resolve(Model)
|
||||
const model = await this.getModel(params)
|
||||
const { query } = this.filterQuery(id, params)
|
||||
const replaceOptions = { ...params.mongodb }
|
||||
|
||||
@ -296,8 +347,7 @@ export class MongoDbAdapter<
|
||||
async $remove(id: Id, params?: P): Promise<T>
|
||||
async $remove(id: NullableId, _params?: P): Promise<T | T[]>
|
||||
async $remove(id: NullableId, params: P = {} as P): Promise<T | T[]> {
|
||||
const { Model } = this.getOptions(params)
|
||||
const model = await Promise.resolve(Model)
|
||||
const model = await this.getModel(params)
|
||||
const {
|
||||
query,
|
||||
filters: { $select }
|
||||
|
||||
@ -97,10 +97,17 @@ describe('Feathers MongoDB Service', () => {
|
||||
friends: string
|
||||
}
|
||||
}
|
||||
type Todo = {
|
||||
_id: string
|
||||
name: string
|
||||
userId: string
|
||||
person?: Person
|
||||
}
|
||||
|
||||
type ServiceTypes = {
|
||||
people: MongoDBService<Person>
|
||||
'people-customid': MongoDBService<Person>
|
||||
todos: MongoDBService<Todo>
|
||||
}
|
||||
|
||||
const app = feathers<ServiceTypes>()
|
||||
@ -319,7 +326,7 @@ describe('Feathers MongoDB Service', () => {
|
||||
}
|
||||
)
|
||||
|
||||
assert.strictEqual(result[0].friends.length, 1)
|
||||
assert.strictEqual(result[0].friends?.length, 1)
|
||||
|
||||
const patched = await peopleService.patch(
|
||||
null,
|
||||
@ -329,7 +336,7 @@ describe('Feathers MongoDB Service', () => {
|
||||
{ query: { name: { $gt: 'AAA' } } }
|
||||
)
|
||||
|
||||
assert.strictEqual(patched[0].friends.length, 2)
|
||||
assert.strictEqual(patched[0].friends?.length, 2)
|
||||
})
|
||||
|
||||
it('overrides default index selection using hint param if present', async () => {
|
||||
@ -351,6 +358,79 @@ describe('Feathers MongoDB Service', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('Aggregation', () => {
|
||||
let bob: any
|
||||
let alice: any
|
||||
let doug: any
|
||||
|
||||
before(async () => {
|
||||
app.use(
|
||||
'todos',
|
||||
new MongoDBService({
|
||||
Model: db.collection('todos'),
|
||||
events: ['testing']
|
||||
})
|
||||
)
|
||||
bob = await app.service('people').create({ name: 'Bob', age: 25 })
|
||||
alice = await app.service('people').create({ name: 'Alice', age: 19 })
|
||||
doug = await app.service('people').create({ name: 'Doug', age: 32 })
|
||||
|
||||
// Create a task for each person
|
||||
await app.service('todos').create({ name: 'Bob do dishes', userId: bob._id })
|
||||
await app.service('todos').create({ name: 'Bob do laundry', userId: bob._id })
|
||||
await app.service('todos').create({ name: 'Alice do dishes', userId: alice._id })
|
||||
await app.service('todos').create({ name: 'Doug do dishes', userId: doug._id })
|
||||
})
|
||||
|
||||
after(async () => {
|
||||
db.collection('people').deleteMany({})
|
||||
db.collection('todos').deleteMany({})
|
||||
})
|
||||
|
||||
it('assumes the feathers stage runs before all if it is not explicitly provided in pipeline', async () => {
|
||||
const result = await app.service('todos').find({
|
||||
query: { name: /dishes/, $sort: { name: 1 } },
|
||||
pipeline: [
|
||||
{
|
||||
$lookup: {
|
||||
from: 'people',
|
||||
localField: 'userId',
|
||||
foreignField: '_id',
|
||||
as: 'person'
|
||||
}
|
||||
},
|
||||
{ $unwind: { path: '$person' } }
|
||||
],
|
||||
paginate: false
|
||||
})
|
||||
assert.deepEqual(result[0].person, alice)
|
||||
assert.deepEqual(result[1].person, bob)
|
||||
assert.deepEqual(result[2].person, doug)
|
||||
})
|
||||
|
||||
it('can prepend stages by explicitly placing the feathers stage', async () => {
|
||||
const result = await app.service('todos').find({
|
||||
query: { $sort: { name: 1 } },
|
||||
pipeline: [
|
||||
{ $match: { name: 'Bob do dishes' } },
|
||||
{ $feathers: {} },
|
||||
{
|
||||
$lookup: {
|
||||
from: 'people',
|
||||
localField: 'userId',
|
||||
foreignField: '_id',
|
||||
as: 'person'
|
||||
}
|
||||
},
|
||||
{ $unwind: { path: '$person' } }
|
||||
],
|
||||
paginate: false
|
||||
})
|
||||
assert.deepEqual(result[0].person, bob)
|
||||
assert.equal(result.length, 1)
|
||||
})
|
||||
})
|
||||
|
||||
testSuite(app, errors, 'people', '_id')
|
||||
testSuite(app, errors, 'people-customid', 'customid')
|
||||
})
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user