This fixes a serious bug in install.sh where it ignored externaly set env variable values such as `SENTRY_IMAGE` in favor of the ones defined in `.env`, essentially making all our e2e tests usless.
We need `docker-compose ps -a` for CI so we were already using 1.24.1, this aligns the rest with that.
For Docker, there are a bunch of network-related fixes in 19.03.12 and prior (DNS fallback and IPv6 advertising) that we'd like to have to see if they are going to fix some reported connectivity issues w/ onpremise.
Fixes https://github.com/getsentry/onpremise/issues/672
I split this PR up into 4 commits. The first one is the bare minimum for the issue. The rest are just consistency corrections that we neckbeards at irc://chat.freenode.net/%23bash would always make.
This is for the onpremise release on Sept 15th.
The new migration system has a migration to handle recreating the
transaction table if the old one is present, we no longer need to do
this in `install.sh`.
This continues on the ideas from #607. By "downtime" here I mean "not accepting events" - web, smtp and background processes are out of scope.
This PR adds a `--minimize-downtime` option to install.sh. This options changes the behaviour of the script by:
1. keeping nginx and relay running until the very end,
2. disabling cleanup on exit and failure,
3. explicitly reloading nginx configuration,
4. and starting the whole cluster at the end.
The results are promising: no downtime if relay version doesn't change, and only a second when it does. So far this was only tested with a curl loop, so I'm still not sure if Relay flushes the events to Sentry before getting recreated by `$dc up`.
This is a long-needed test that tests the whole pipeline from Nginx, Relay, to Kafka, and Snuba. The final missing piece is testing the Symbolicator integration.
This PR is also a follow up to #576 as it didn't solve the Relay issues fully (the earlier fix was a coincidence or is not as reliable as it seemed).
Fixes#486 (finally?).
This patch adds `INTERNAL_IPS` definition to `sentry.conf.py` by sniffing the network from eth0 and relies on this for trusted Relays instead of the ALLOWLISTED PKs. This removes the necessity of syncing Relay PKs to `sentry.conf.py`.
This PR needs getsentry/sentry#19798 to work.
I would like to be able to customize the configuration for my Sentry 10 symbolicator instance, which this change allows me to easily do.
See related: https://github.com/getsentry/symbolicator/issues/245
Co-authored-by: Burak Yigit Kaya <ben@byk.im>
Hi,
I've been through quite a few different ways of implementing this fix and settled on creating a variable to store the output of checking whether the zookeeper copy target folder exists and copying the snapshot file based on the copy target folder existing. I've ran quite a few manual tests for each option as well. Currently the PR sits on Option 3 from the below options.
**Option 1**
Judging from the [Jira issue](https://issues.apache.org/jira/browse/ZOOKEEPER-3056), it seems like the work around for zookeeper upgrades could be omitted entirely since the issue relates to upgrades from v3.4.10 to v3.5.4. I've tested removing the zookeeper workaround entirely and that install ran smoothly on a clean install of Sentry (no existing data) as well as an install of Sentry that currently has very minimal amount of log entries (roughly 100 log entries). Could we possibly remove the workaround entirely?
**Option 2**
The second option was to simply add a check to the currently [existing line](https://github.com/getsentry/onpremise/blob/master/install.sh#L178) of whether the copy target folder exists and perform the snapshot file copy only if the copy target folder exists. This is the least amount of code and possibly the simpler fix while also setting the `ZOOKEEPER_SNAPSHOT_TRUST_EMPTY` env var to `true`, however, some unnecessary calculations will be done to determine the `ZOOKEEPER_LOG_FILE_COUNT` and `ZOOKEEPER_SNAPSHOT_FILE_COUNT`.
**Option 3**
I've created a variable to store whether the copy target folder exists and proceed with the zookeeper upgrade workaround only if the copy target folder exists. This means that if the copy target folder does not exist, the env var `ZOOKEEPER_SNAPSHOT_TRUST_EMPTY` will not be set.
Fixes#547.
Co-authored-by: chamirb <chamirb@globalkinetic.com>
Tested this in a Ubuntu VM. The output of `date` is not too pretty but at least localized (so D/M vs M/D is not confusing)
## What is the TSDB migration?
We're effectively deprecating all TSDB backends but `sentry.tsdb.redissnuba.RedisSnubaTSDB`. We cannot reasonably support any other backend due to the fact that we would have to reimplement each of the backends in Relay, which is written in a different language. Also, like with deprecating mysql support, we don't really have the capacity to support things we do not use ourselves.
## Migration
`install.sh` should rewrite your configuration automatically and define a cutover date such that no data is lost. Before the cutover date, data is written to two backends at once, Redis and Snuba, and read from one, Redis. After the cutover date, event-related metrics will be read from Snuba which matches what we have on sentry.io.
## Manual migration guide for TSDB
In case `install.sh` is unable to migrate your files you will be given basic instructions on the console that essentially tell you to completely delete all TSDB config and paste the new, standard one. If for some reason you cannot say goodbye to your existing TSDB config, please create a new issue in this repo and cc @untitaker on it.
This PR disables checking for the SSE4.2 flag, which may not be present if the server is running in a VM even if SSE4.2 is in fact available to use (ClickHouse/ClickHouse#20). The KVM check was obtained from a [comment](https://github.com/ClickHouse/ClickHouse/issues/20#issuecomment-226849297) in that issue.
If SSE4.2 is not actually available in the VM, then the installation script may fail in an unpredictable way. Perhaps we can add a keyword argument to the script to let the user decide if they want to skip the check.
This patch does two things:
1. Separate creating of Relay credentials from syncing them to Sentry config
2. Has a more flexible Relay credentials check and sync: look for the quoted public key in the config file, if it is there, assume this must be correctly set as it is very unlikely to have that random key in a different context with quotes around. The second one is to allow having other whitelisted relay keys by using an append method when adding the new key.
Update exit -1 to exit 1 in line with best practices
Fixes premature exit of the script when checking for sse42 support. The
script would exit before displaying the reason for failure to the user.
With getsentry/snuba#709 merged, we no longer need to randomly try forever creating Kafka topics, which were a prerequisite for creating Clickhouse tables. This patch also removes the preceding (and obsolete) `docker-compose up` statement which was used to "speed up" the `snuba bootstrap` command in the hopes to fix it.
Since we are [moving the eventstream into a proper db migration](https://github.com/getsentry/sentry/pull/16226), we need Snuba and its friends ready at the time of `sentry upgrade` command. This patch does exactly that.
Co-authored-by: Lyn Nagara <lyn.nagara@gmail.com>
This change makes the install script create a new install log file, `sentry_install_log-<ISO_TIMESTAMP>.txt`, for each run and records all the output there for future reference.
