As discussed, removing the upgrade test since it doesn't provide that much utility as we are already testing upgrades from restoring docker volumes from cache
We need to care about more than just src/sentry/migrations. We will need to account for files in src/sentry/**/migrations/*
Taken from afd7469818/.github/file-filters.yml (L90)
In this version, there's a new `--pull` argument for `docker compose run` which we will start leveraging, especially with `sentry-admin` command.
Should come with a slight speed boost.
Just starting up services for Snuba or Sentry migrations takes up to a minute sometimes and we do this even when there are no migrations, just because one of the Sentry or Snuba migrations change. This patch splits the caches up so only the necessary one runs, saving further time. It also uses the `LATEST_TAG` as the cache key for upgrade tests as the image versions or data will never change for a certain tag once it is release.
We should do the backup/restore tests _after_ we do the basic tests. This is both more efficient as we avoid an extra up/down cycle and more meaningful as we will back up and restore an actually used system.
A bit hard to measure directly as this also moves the initial `docker compose up -w` into the test suite but a random run without this patch took about 10m 49s to finish for the testing part whereas with the patch it came down to 9m 10s so **almost 2 minutes faster**!
Instead of using direct image hashes, only use hashes from migrations folders for each respective image for cache key generation. Should increase cache hit rate significantly as we don't have migrations much.
Also swaps the key order from `sentry-snuba` to `snuba-senry` assuming Snuba has less frequent migration additions.
This is an initial transitionary patch before landing #3516. Once we land this, we will update users of the old action to use this one and remove that repo. Then land #3516 safely.
Great thing is, with this patch and the subsequent update to getsentry/action-self-hosted-e2e-tests to use this one, all the repos would be using the Docker Volume caching we introduced in #3488.
Docker Compose is much more robust nowadays compared to the past where we had to maintain tests for both v1 and v2 and then a specific version of v2. Hence, we are removing tests for the older versions of Docker Compose with this patch. We also remove the separate tests for customizations and w/o customizations as the one with customizations should cover the one without them anyway. This reduces the CI workload to 25% of what it was
This patch caches all DB volumes based on the sentry and snuba images to avoid doing the same migrations over and over for every test run.
This shaved off a whole minute from "Install self-hosted" jobs (so ~20% speed increase).
Left side: cached re-run -- Right side: no-cache initial run
I think we split these actions in the past due to some lack of options for them to work together properly. Right now looks like `bootstrap` would automatically migrate and propagates the `force` flag.
`geoipupdate` is not used by any other service nor it is needed for any
service to run. Moreover, it is a one-shot command, causing
`docker compose up --wait` to fail when it exits with a non-zero status.
This happens when one has not yet set up their credentials and they may
choose to never do this.
This PR removes `geoipupdate` from the `docker-compose.yml` file and
moves the command directly into the geoip related script. One may run
this whenever they want to update their GeoIP database.
This PR needs an accompanying docs change.
Our default fallback, `_assets`, assumes we use a CDN which is not the case on self-hosted. This patch adds a stop-gap fix for front-end URLs asking for this path.
Should fix#3479 and #3470.
The backup-method described in the [documentation](https://develop.sentry.dev/self-hosted/backup/#backup) `./scripts/backup.sh` creates a backup file at `sentry/backup.json` which should be ignored by git.
### Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
As Sentry continues to evolve, effective resource management becomes crucial for maintaining performance and stability. This update includes configurations that will help optimize Redis's memory usage, ensuring that the system runs efficiently under varying loads.
**Key Changes:**
- **Added `maxmemory` Directive**: Configured Redis to limit its memory usage to a specified size. This prevents excessive memory consumption and helps maintain system stability.
- **Set `maxmemory-policy` to `allkeys-lru`**: This policy allows Redis to evict the least recently used keys when it reaches the memory limit, ensuring that frequently accessed data remains available while older, less-used data is removed.
