100 Commits

Author SHA1 Message Date
William
2d6248f08b
Remove additional space 2018-07-12 14:26:02 -07:00
William
849d3b206f
Change error detection
Use code provided by the errors providerError.
2018-07-12 14:24:56 -07:00
William
880ba0b788
#5110 null error on undefined SSM variables
Ignore null errors to allow resolution instead of rejection on undefined SSM variables.
2018-07-11 08:44:58 -07:00
Erik Erikson
534dda6285 Fix #4973
Do a better job of extracting the current variable syntax for use in deep variable generation.
2018-05-18 00:04:37 -07:00
Erik Erikson
63045fd206 Fix #4946
Use the variable syntax present in the given variable string, extracting it from the given string that may contain embedded custom variables (i.e. `${{self:var.${{self:var.foo}}.bar}}`).
2018-05-02 11:52:24 -07:00
Takahiro Horike
a9e6a79d01
Merge pull request #4743 from svdgraaf/feature/pass-serverless-variable
Pass serverless variable when calling function in referenced file
2018-04-03 05:04:08 +09:00
Erik Erikson
71c6db9687 Re-write self:service.[...] to self:serviceObject.[...]
Handle `self:service.[...]` references properly.  The prior represents what the user typed into their configuration.  In service, we move all this from `service` to `serviceObject`, therefore, by rewriting `self:service.[...]` to `self:serviceObject.[...]` user's will get the result they expect.
Use else if, given exclusivity of conditions
2018-03-27 15:34:34 -07:00
Frank Schmid
5a4d00c9ad
Merge pull request #4800 from bsdkurt/master
Continue recursion for #4687
2018-03-06 11:35:17 +01:00
Kurt Miller
3f5a041678 Continue recursion for #4687 2018-03-04 08:25:19 -05:00
Erik Erikson
0ee0a1e239 Only disable and restore dependent service resolution methods once. Otherwise, subsequent removals may cache the previous replacements. If they restore last then they will restore with the replacements, breaking standard usage.
Add tests for this guarantee.
2018-02-24 09:34:50 -08:00
Erik Erikson
91a10940eb Add further comments noting code entanglement 2018-02-23 10:09:18 -08:00
Erik Erikson
bf5a8c9fd4 Fix 4744 and associated
The framework's Service class modifies a user's service, mutating a given `service: { name: 'string' }` to `service: 'string'` but doesn't account for this in the variables system.  This is the basis for #4744.

While working this area, I discovered that the Service class also does this to the provider, accepting `provider: 'aws'` and replacing it with `provider: { name: 'aws' }`, causing the natural `${self:provider}` to fail.

Catching either 'self:service.name' or 'self:provider' and replacing them with the mutated reference solves this user confusion.
2018-02-22 17:17:55 -08:00
Erik Erikson
dc3a4aa6af Fix print, clean pre-population, fix cyclic bug
Fix `print`
The print command is highly linked to the `Variables` and `Service` codebases, keep those in sync and leave reminders about the link.  Made these explicit and separately implemented to avoid complexity.
Additionally, the print command re-populates an object with the *very similar* content as the previously pre-populated service (just not augmented as just mentioned).  This can lead to cross contamination between the two.  As such, all caches must be cleared per unique invocation of service/object/property population.
Add tests for some expected but previously unverified behaviors.

Clean pre-population
The previous implementation worked okay but was unnecessary and would have been a maintenance problem.  Instead, just knock out the population of variables depending on those config dependent services and use the standard means of resolution.

Fix cyclic bug (resulting from running print against a self-referencing serverless.yml)
The caching of values could lead to a cyclic object remaining in the caches for variable population.  This causes crashes and pain.  Solved by the cache cleaning logic.
2018-02-22 16:30:04 -08:00
Erik Erikson
8c7db120ae Add cases for all dependent services, and a variety of deep variable (DV) cases
Solve for these cases.  The bug was that DVs can render to DVs when a value is being obtained from them (i.e. they are being de-referenced in getValueFromDeep).  This is particularly problematic in the case that the original DV being rendered has a deep reference into the rendered DV.  In that case the DV returned by rendering must be replaced by yet another DV.

While accomplishing the above, various bits of cleanup were implemented and I added some commentary around getDeeperValue for future engineers.

This case is particularly relevant if you have:

`serverless.yml`:
```
service: serverless-hello-world

provider:
  name: aws
  runtime: nodejs6.10
  stage: dev
  environment:
    SECRET: ${self:custom.secrets.SECRET}

functions:
  helloWorld:
    handler: handler.helloWorld
    events:
      - http:
          path: hello-world
          method: get
          cors: true

custom:
  stage: ${opt:stage, self:provider.stage}
  secrets: ${file(secrets.${self:custom.stage}.yml)}
```
AND
`secrets.dev.yml`:
```
SECRETS: secrets
```

Populating this service should result in the following sequence of rendering phases:

#########################
# PHASE 1
#
# ${self:custom.secrets.SECRET}
#   <- ${deep:0.SECRET}
#   deep[0] = ${file(secrets.${self:custom.stage}.yml)}
#
# ${opt:stage, self:provider.stage}
#   <- 'dev'
#
# ${file(secrets.${self:custom.stage}.yml)}
#   <- ${file(secrets.${deep:1}.yml)}
#   deep[1] = ${opt:stage, self:provider.stage}
#
# RESULT
#
# service: serverless-hello-world
#
# provider:
#   name: aws
#   runtime: nodejs6.10
#   stage: dev
#   environment:
#     SECRET: ${deep:0.SECRET}
#
# functions:
#   helloWorld:
#     handler: handler.helloWorld
#     events:
#       - http:
#           path: hello-world
#           method: get
#           cors: true
#
# custom:
#   stage: dev
#   secrets: ${file(secrets.${deep:1}.yml)}
#########################

#########################
# PHASE 2
#
# ${deep:0.SECRET}
#   <- this.populateValue('${file(secrets.${self:custom.stage}.yml)}') => '${file(secrets.${deep:1}.yml)}' => '${deep:2}' => '${deep:2.SECRET}'
#   deep[2] = ${file(secrets.${deep:1}.yml)}
#
# ${file(secrets.${deep:1}.yml)}
#   <- this.populateValue('${file(secrets.${deep:1}.yml)}') => '${file(secrets.dev.yml)}' => '${deep:3}'
#   deep[3] = ${file(secrets.dev.yml)}
#
# RESULT
#
# service: serverless-hello-world
#
# provider:
#   name: aws
#   runtime: nodejs6.10
#   stage: dev
#   environment:
#     SECRET: ${deep:2.SECRET}
#
# functions:
#   helloWorld:
#     handler: handler.helloWorld
#     events:
#       - http:
#           path: hello-world
#           method: get
#           cors: true
#
# custom:
#   stage: dev
#   secrets: ${deep:3}
#########################

#########################
# PHASE 3
#
# ${deep:2.SECRET}
#   <- this.populateValue('${file(secrets.${deep:1}.yml)}') => '${file(secrets.dev.yml)}' => '${deep:3}' => '${deep:3.SECRET}'
#   deep[3] is already set to ${file(secrets.dev.yml)}
#
# ${deep:3}
#   <- this.populateValue('${file(secrets.dev.yml)}') => { SECRET: 'secret' }
#
# RESULT
#
# service: serverless-hello-world
#
# provider:
#   name: aws
#   runtime: nodejs6.10
#   stage: dev
#   environment:
#     SECRET: ${deep:3.SECRET}
#
# functions:
#   helloWorld:
#     handler: handler.helloWorld
#     events:
#       - http:
#           path: hello-world
#           method: get
#           cors: true
#
# custom:
#   stage: dev
#   secrets:
#     SECRET: secret
#########################

#########################
# PHASE 4
#
# ${deep:3}
#   <- this.populateValue('${file(secrets.dev.yml)}') => this.getDeeperValue(['SECRET'], { SECRET: 'secret' }) => 'secret'
#
# RESULT
#
# service: serverless-hello-world
#
# provider:
#   name: aws
#   runtime: nodejs6.10
#   stage: dev
#   environment:
#     SECRET: secret
#
# functions:
#   helloWorld:
#     handler: handler.helloWorld
#     events:
#       - http:
#           path: hello-world
#           method: get
#           cors: true
#
# custom:
#   stage: dev
#   secrets:
#     SECRET: secret
#########################
2018-02-20 20:57:24 -08:00
Erik Erikson
a35bcc01cd Deep values can be overwrites too! Fix issue reported by @laardee (thank you!)
Simple oversight that deep variables can reference overwrites too.  This fix acknowledges this oversight by using the standard logic for using overwrite instead of getValueFromSource in this case.
2018-02-20 12:21:26 -08:00
Erik Erikson
4400ffc9e4 fix #4311 & #4734, separate PromiseTracker, minutiae
#4311
  see prepopulateService - attempts to pre-populate the region and stage settings necessary for making a request to AWS, rejecting and dependencies thereon it runs into during that process
  see the `deep` variable work.  this was a knock-on effect of providing pre-population.  it actually represents an obscure class of bugs where the recursive population previously in getDeepValue caused the caller not to be in charge of population choices (thus fixing for pre-population) but also caused potential deadlocks resulting from getDeepValue creating circular dependencies.

#4734
  see splitByComma - a regex to do the splitting but ignore quoted commas was getting very deep and involved.  Instead, identify quoted string boundaries, then identify commas (including white space around them) and take those commas not contained by quotes as the locations for splitting the given string.  Trim the string as well (removing leading and trailing white space).
  add sophistication to the overwrite syntax, allowing for whitespace and repetitions (for robustness)
  add sophistication to the string ref syntax, allowing for use in identifying multiple quoted strings in a variable (i.e. for overwrites)

#NotCreated
  fix a bug I created earlier in the branch that caused reporting to be less informative (see renderMatches)

separate PromiseTracker
  move this class into its own file for the purpose of increasing testability and offering reuse

minutiae
  filter the properties given to populateVariables so as to avoid attempting resolution on non-variables.  Efficiency and noise reduction change.  Also cleans up the code (e.g. see populateObject and its use)
  cleaning of overwrite as a side effect
  offer variable cleaning as a idiom
  reorder the Variables.js `require`s to be in alphabetical order
2018-02-16 17:13:03 -08:00
Sander van de Graaf
8abcee66a4
Pass the serverless variable as parameter to the receiving function 2018-02-15 15:59:45 +01:00
Erik Erikson
3606589ff2 Do not double-reject errors
Calling .then here creates a new promise.  One that is not returned and generating a rejection within that promise can lead to unhandledRejection events being inappropriately raised.
As such, don't declare unneeded variables.
2018-02-06 16:35:16 -08:00
Erik Erikson
c18eb70a28 Eliminate Deadlocks, Report Hung Promises (#4687)
Add a "PromiseTracker" to the Variables class.  It provides tracking and reporting of pending variable resolutions during service populations.
Decompose populateObject and populateProperty into smaller pieces that are easier to understand.
Additionally, remove the immediate recursive resolution of variables, recursing only at the Object or Property entry point (when changes have been made).  This avoids a problem where the resolved value, prior to being replaced in the containing object or value refers to itself and thereby is waiting on its own resolution before it will resolve itself.
Return the given value in warnIfNotFound so that it can chain.
2018-02-02 19:29:19 -08:00
Craig R Webster
271b45009d Fix typo in 'syntax' 2018-01-03 19:15:41 +08:00
Frank Schmid
81c896bc12
Removed reintroduction of stage+region in request. Added options.
Delete bucket was still using them

Hopefully all :)

Further test fixes.

.... worked too long yesterday

Fixed Variable tests

Remove not used parameters from request() and add options with warning
2017-12-11 12:33:35 +01:00
Erik Erikson
8c1fe6dfcc Refactor, simplify, correct, add tests
Increase name consistency
Improve names to be-what-they-are
Reduce cyclomatic complexity
Include overrides population fix
Add variable methods documentation
Add more tests around significant variable patterns/risky cases
2017-12-06 15:59:07 -08:00
Frank Schmid
bce1264d7b
Fixed merge 2017-12-07 00:35:13 +01:00
Frank Schmid
8f70c13a9c
Merge branch 'master' into circular-vars 2017-12-07 00:22:55 +01:00
Frank Schmid
2e09606f13
Merge remote-tracking branch 'remotes/upstream/master' into circular-vars 2017-12-06 13:20:51 +01:00
Frank Schmid
8ff661835e
Merge branch 'master' into populate-default-objects 2017-12-06 12:34:50 +01:00
Frank Schmid
8cd604cd41
Wrap cached value into promise (addresses issue mentioned in #4509) 2017-12-05 12:39:48 +01:00
Frank Schmid
5526802a1b
Added request cache and queue to AWS provider
Make sure that requests are throttled. Use a queue for the requests.

Added new expected parameter to request() call in variables test

Added request cache tests
Check that request is indeed called 1000 times

Cache promises, not values

Use request cache for AWS variable requests

Use fromCallback instead of "new Promise" anti-pattern.

Added request cache to AWS provider
2017-12-05 12:39:39 +01:00
Frank Schmid
37340467ed
Populate objects if default syntax targets an object 2017-12-04 18:08:39 +01:00
e-e-e
f90c6d2782 return resolver promise; fix lint 2017-12-03 21:25:22 +11:00
e-e-e
57d898d39e replace promise in cache with cached result once resolved. 2017-12-02 12:03:33 +11:00
Benjamin Forster
6e6fea3fa2 update ssm cache to generic variable cache. 2017-12-01 10:33:30 +11:00
Benjamin Forster
e780a59b66 fix linter 2017-11-29 17:21:41 +11:00
Benjamin Forster
de22ae698b add cache to ssm variables 2017-11-29 17:10:05 +11:00
Erik Erikson
143a2aa65b Account for duplicate copies without populating more than once. As a result, populate in place. 2017-11-17 17:30:15 -08:00
Eslam A. Hefnawy
bebcdf3057 added support for circular references in variable system 2017-11-17 00:00:57 +01:00
Farley
80c9442794 Fixing linting, finally? 2017-11-06 13:25:38 +01:00
Farley
2080ce9ec1 Fixing linting 2017-11-06 13:19:23 +01:00
Farley
5b786e8914 Fixing linting 2017-11-06 13:14:57 +01:00
Farley
113e7b93a9 Fixing #4435 2017-11-06 12:43:13 +01:00
Basile Trujilllo [L0gIn]
c6d204bd88 Remove realpathSync second arg to fix unit tests 2017-10-17 14:51:25 +02:00
Basile Trujilllo [L0gIn]
08d62af899 Resolve file path before read file 2017-10-17 11:33:23 +02:00
Loren Gordon
da782b85f4 Escapes the hyphen in the ssm variable syntax
Fixes #4248
2017-09-12 13:57:52 -04:00
Loren Gordon
b00ad4e773 Removes escaping of double-quotes around string value 2017-09-07 07:32:06 -04:00
Loren Gordon
5a7d14af09 Adds overwrite syntax support to SSM variable syntax
Check for the expected error message in the case the SSM parameter
does not exist and return `undefined`. Any other (unexpected) error
message still throws an exception.
2017-09-01 15:10:00 -04:00
Loren Gordon
a266d4f224 Supports ssm: variable references
Adds support for specifying SSM parameters as serverless variables.
Basic syntax is `${ssm:<foo>}`. To support decryption of SecureString
parameters, the extended syntax is `${ssm:<foo>~true|false}`. The `~`
is a delimiter; `true|false` is evaluated and passed to the
`WithDecryption` option of the SSM API.
2017-09-01 12:54:04 -04:00
Toby Hede
33431cb45a Allow double-quoted and single-quoted variables
Empty string is now valid
2017-08-20 11:05:15 +10:00
Toby Hede
2b3181c362 Handle both single and double quoted strings 2017-08-18 09:13:01 +10:00
Toby Hede
a1c8a37b9a Allow use of string as default values in variables 2017-08-18 09:07:29 +10:00
Rafal Wilinski
fea41a0ddf Fix Variables.s3RefSyntax typo 2017-08-11 20:01:09 +02:00