aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2022-03-11LLP: fix templating problem for login pageHEADmasterKelley Spoon
Since the login page is using the view created by django.contrib.auth, we need to ensure that the "site_name" context variable is set. Change-Id: I5e5160ffd54383718b9942d06871611cb038e818 Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org> Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40739
2022-03-11llp: prevent error if a request group doesn't existKelley Spoon
When we check for group membership in LDAP, we should log the fact it doesn't exist, but proceed on without an missing key error. Change-Id: I2bbee729f8cf144624c8c01af7b76f49f2b8be35 Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org> Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40740
2022-03-11llp: install build reqs for python-ldapKelley Spoon
We're using python-ldap=3.3.1 now, which must be build from source since the latest supported version is 2.2.4. Let's apt-get the build requirements in the unit-test.sh script so Jenkins can be useful instead of always erroring on the ldap build failure. Change-Id: I582d2d1480e185b705153f50367ea51e4b61d953 Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org> Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40741
2022-02-14template: use CSS for table cellsMarcin Juszkiewicz
<td valign="top"> is obsolete. Same with <td align="center"> Change-Id: I35c0f57585a5550e40b1c4ae52e0fc561a2bc9b5 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40533 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2022-02-14template: generate more valid HTMLMarcin Juszkiewicz
- DOCTYPE is mandatory for HTML 5 - type/language parameters for <script> <style> are obsolete Change-Id: I7a01b650eb723b3fb5437826decdbf5cf2ac55b4 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40534 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2022-02-02group_auth_ldap: fix an error with user group authorizationKelley Spoon
There is a bug in the ldap group authorization code where we use the full django username for authentication (which is the full email), but only the UID (first.lastname) is stored in the group membership table. We should also take this time to just try to look up the UID in the groups table instead of trying to build a list of group memberships for the user and compare that to the required groups. Change-Id: I41209fb8745a6225f3e7344910dc89c19d336a76 Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org> Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40448
2022-02-01settings_production: add in LDAP settingsKelley Spoon
This change adds in the requires AUTH_LDAP settings to allow llp to use auth_ldap_backend. The LDAP bind password is set in the host's secrets. Change-Id: Ifcf8ef6d79ae8ba7d5e904a696b60a8bb9dd9eae Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org> Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40443
2022-01-31llp: remove crowd and openid support in favor of ldap authKelley Spoon
In order to remove the final piece of Crowd usage in our infrastructure, let's migrate LLP to using ldap for both authentication and group look ups. We also need to update the views, urls, and tests to no longer rely upon crowd/openid and instead mock up the LDAP lookups. Change-Id: I254d7bbffff97ffa8ebe5f5be72e763e77a6c45d Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org> Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40275
2021-12-18llp: add s3 lambda to tag files for expiryKelley Spoon
This change adds in a lambda function and the terraform to upload and enable it that will check if an uploaded (created) file is in a protected path. If not, the S3 object will be tagged for eventual deletion via a lifecycle rule when it's lifespan expires. Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org> Change-Id: I34ade1f38311a269c479d6373495a789de96ab15 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40186
2021-12-18staticfiles: rename root static dir and add missing filesKelley Spoon
By adding 'static' to the gitignore file, the pattern match was also applied to 'license-protected-downloads/static' and the missing admin, jquery, and css files were left out of the repo. Let's rename the top level directory where the collected files are to be put to 'staticroot' so that it can safely be added to the gitignore file and restore the missing files. Change-Id: Ie051df5c26ab83df96af5a8453648b0db9f3cbcd Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40226 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2021-12-18staticfiles: get control of the static files once againKelley Spoon
Fix staticfiles so that they are correctly generated by 'collectstatic' regardless of where or how this app is deployed. In this change, let's concentrate all static files in 'license_protected_downloads/static' so that we can clear out STATIC_ROOT ('static') and rebuild it reliably with 'collectstatic'. We also restore the contents of the 'license_protected_downloads/static' directory, and update the unit-test.sh script to call 'collectstatic' on a freshly installed STATIC_ROOT before it runs. Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org> Change-Id: Ice720b4794a48540c3753f4a6efba027cbc29040 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40206
2021-11-18settings.py: add beautifulsoup warning suppression and fix static dirKelley Spoon
This change adds in code to supress the beautifulsoup warning from the application log in addition to the previous commit that removes it from the user space commands. Also, the STATICFILES_DIRS is incorrectly set to be the same as STATIC_ROOT. Instead let's point it at PROJECT_ROOT to make sure the right files get picked up and we avoid any nasty loops when running collectstatic Change-Id: I5a2c592cfd3c625053e785c62863e6f6ff0af59f Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40044 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2021-11-11manage.py: silence beautifulsoup warningsKelley Spoon
BeautifulSoup produces and UserWarning when imported reminding the user that the version we're using is EOL and that we should be upgrading it soon. This warning is producing output that is causing crontab to email when there are no errors or problems to report. Let's silence it until we can finally update the software. Change-Id: Ic45819c0f22ff91b2f2b5138433d69428cbeec95 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40042 Reviewed-by: Riku Voipio <riku.voipio@linaro.org>
2021-06-17gitignore: ignore a couple of files created in productionKelley Spoon
We should add allowed_hosts.txt and local_settings.py to gitignore as they are created when run in production and we get warnings about untracked files on a git status. Change-Id: I2944817848fff25bf1e4e6594d4a4f9868dcbb1f Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38934 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2021-06-17staticfiles: handle staticfiles correctlyKelley Spoon
Let's use the static() helper in our templates to correctly generate a URL for our static files. We also need to add our STATIC_URL to the urlpatterns list, and straighten out the settings to ensure the static root is found and loaded correctly. Change-Id: I933bcf9c9eeec7300dc0f3935b07179446f7b621 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38933 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2021-06-17staticfiles: remove unecessary directories and remove symlinked iconsKelley Spoon
Somehow the css and js directories got moved out of the static directory and up into the root of the project, and then were committed with the minified file names. Let's clean this junk out of the way to it doesn't continue to confuse the issue. Change-Id: I234cebb2ba6f743160f8261ed863b17134c0e305 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38932 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2021-06-07report_process: support alternate timestamp formatKelley Spoon
Previously the cron job to process the download report CSV file was choking because the timestamp in the CSV didn't have the fractional milisecond field on it. This change will catch the ValueError that gets raised, and tries an alternate timestamp so that processing can continue. Change-Id: Ied59f40eebc5e2be692aba5f87872f7e652e0e9d Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38859 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2021-05-19settings.py: add in testplan and testreport changesKelley Spoon
A while back a request was made to include a 'Test Plan' and 'Test Report' tab for some of our projects that would feature the contents of the respective files. Apparently the change was made in place on the server, but was never migrated into the LLP repo. Let's get caught up. Change-Id: I5e49fcd41211966065edbc01191ad607d84e00c5 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38759 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2021-05-19gitignore: add static files directoryKelley Spoon
Let's fix the gitignore for the static files directory as git seems to think a few png's are now version controlled. Change-Id: I3470b508b58f0d6ef6fb383e1ce95c7178305c54 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38760 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2021-05-19Whitelist: Add android.linaror.orgBenjamin Copeland
Change-Id: I2546c45b4eda485d453bf8201ff65ee34e0d0e56 Signed-off-by: Benjamin Copeland <ben.copeland@linaro.org> Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38758 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2021-03-31llp: add STATIC_ROOT to STATIC_DIRSKelley Spoon
testdata.lo was failing to show static files (ie, icons in pages). Let's add in the STATIC_ROOT to STATIC_DIRS explicitly in order to ensure that the files are correctly found and served. Change-Id: Iffb8b489cd421ac3fb8344e0c51d542e3aa7118f Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38333 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2021-01-13Whitelist: Add missing AOSP slaveBenjamin Copeland
Change-Id: I596f3cdea9c83148568031007c740c84b2708fae Signed-off-by: Benjamin Copeland <ben.copeland@linaro.org> Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/37509
2020-12-01Add timestamps to the report CSVKelley Spoon
Currently, our process_report script assumes that a Download occured on the day the process_report script was run. This isn't correct, so we need to keep track of the actual datetime when the download occurred in the CSV as an extra field. This attempts to handle it in a way that is backwards compatible with previous CSV files that don't include a timestamp column. Change-Id: I5249c434a2777f09fb1e2010b812a91255517b07 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/36874 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-10-29llp: fix links to deeper pathsRiku Voipio
With the original code, the re match is: ('https://snapshots.linaro.org/openembedded/schneider/linaro-dunfell-4.19/soca9', 'latest/dip', 'MD5SUMS.txt') Correct it by matching only exactly to "latest" ('https://snapshots.linaro.org/openembedded/schneider/linaro-dunfell-4.19/soca9', 'latest', 'dip/MD5SUMS.txt') Hence reconstructing the url will succeed again fixes LSS-1804 Change-Id: I73bb5b6c6655e0f3590facc3e4302dc7aadb9294 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/36676 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2020-08-19fix direct url referencesRiku Voipio
Fixes direct reference like: http://localhost:8080/components/kernel/linaro-edk2/latest/x.txt and thus also wildcards on latest: http://localhost:8080/components/kernel/linaro-edk2/latest/startup-qemu-*.nsh Signed-off-by: Riku Voipio <riku.voipio@linaro.org> Change-Id: Ia95dbaa3f715dfe604f01b43776dbba9f26147ea Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/36170 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-08-18Fix testsuiteRiku Voipio
1. requirements pin auth plugin version The new version requires Python 3 2. Revert the S3 testsuite changes. These need to implemented properly in it's own changeset 3. views.py: don't call s3_replace_latest if we are not configured to use S3 Signed-off-by: Riku Voipio <riku.voipio@linaro.org> Change-Id: I7723155a1dbfc242857618e64ecf5852ea4ed22c Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/36180 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2020-08-04fix bug introduced in new link_latest codeKelley Spoon
The previous commit would truncate anything after "/latest" if an .s3_link_from file was found, which made it impossible to reference a file through the "latest" version directly. This change also includes updates to the default settings.py to include S3 settings that will trigger API v3 testing. Enabling the v3 unit-tests exposed some problems in our tests, so those have been skipped with a "FIXME" tag until they can be addressed. Change-Id: I1c801f86bbf11bc281a44d755c5767febc98f507 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/36054 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-07-28LLP: move latest/ link resolving to runtimeRiku Voipio
Instead of copying all items in link-latest operation, simply parse the latest link on fly when user is browsing Signed-off-by: Riku Voipio <riku.voipio@linaro.org> Change-Id: Ifeb2b9e95e1ddb0d40ec2d066ca5b1edb4fb1b53 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35939 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2020-06-30llp: add missing nodes to whitelist configKelley Spoon
This adds in x86_64-15 and x86_64-swarm to the list of machines allowed to post/get to the server with no authentication. Change-Id: Ia40873c7bebecdae31042d673898c8d40f748952 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35796 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-06-01s3_flatten.py: add missing import for sleep()Kelley Spoon
Somehow missed importing sleep() from time. Let's do that in case the httplib error ever occurs again. Change-Id: Ifb6a039a54433111c2e278b2666f906405baa424 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35430 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-05-26s3_flatten.py: fix a silly typo that was causing an exceptionKelley Spoon
There was a missing format character for an error log message that was triggering a valueerror exception. Change-Id: Iea0569991d3914c256c269bdc7bc14a8f25b2ab2 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35367 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2020-05-14s3_flatten: fix inifinte loopKelley Spoon
Turns out if you use a while loop to do retries, you should also set the exit for a success. Change-Id: I593c717459e259487d804bd3b616b5b3049ff462 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35188 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-05-14s3_flatten.py: wrap the entire handle_bucket() method in retry logicKelley Spoon
It turns out that boto will attempt to lazily load details for objects in a bucket. As a result, it is not sufficient to simply wrap the list_objects() calls. In order to prevent even more complexity by attempting to predict where try/excepts retries are needed, let's just wrap the handle_bucket() method itself and use the wrapper to safely clean up objects. Change-Id: I257f9390dbb6f59e83511e50594824306ad9baf8 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35184 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-05-14s3_flatten.py: add retries for boto3 bucket operationsKelley Spoon
It turns out that this script can occasionally error out if the AWS API doesn't respond with proper HTTP. When this happens, the error isn't caught and the script ends prematurely and can leave large amounts of unnecessary data in the bucket. This wraps S3 bucket operations (specifically delete_keys and list_objectS) in a try statement that will attempt up to 3 retries after sleeping for 30 seconds. Change-Id: I3f4c945937b20635166cad093630bfbca9026a78 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35063 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2020-05-02static file: change locationKelley Spoon
Due to reorganizing the docker images to have their own independent mount points, the path to the static files has changed. Change-Id: I1e406c3bdecb2b7720a78c6bb307a394a9bc0fdd Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35007 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-05-02whitenoise: lock version to 4.1.4Kelley Spoon
WhiteNoise 5.0 needs to be downgraded and locked to 4.1.4 for backwards compat as 5.0 is using the py3 urllib API. Change-Id: I654055a210f63192a8c069787eb1169e02c0b25f Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35005 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-05-02Revert "gunicorn: version lock to 19.7.1"Kelley Spoon
This reverts commit 3d97c57b6e6710d6931e92495eb026d4b99849b0. Reason for revert: gunicorn is handled by OS package manager and should not be done here. Change-Id: I5bd46ff3d1282e4771a71da874bc1bba55246f9d Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34498 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-05-02Revert "gevent: add gevent as a requirement"Kelley Spoon
This reverts commit c1633ecc22806843950cf985c5dfc837a46b5593. Reason for revert: gevent is handled by the OS package manager, shouldn't be installed here. Change-Id: I0a860b65cde24610e5a229fd02386e7e1c3acb3b Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34497 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-05-02gevent: add gevent as a requirementKelley Spoon
By adding in gunicorn, gevent was picked up as a dependency. Change-Id: I984e1c70563349107df8b1b550fdb870b2bc6e8b Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35004 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-05-02gunicorn: version lock to 19.7.1Kelley Spoon
Need to version lock to 19.7.1 due to python2 support. 19.9.0 explicitly calls python3, which can't find django installed for python2.7 Change-Id: Ic1205c0b651afcc422f9bcc1798b148a09f4e2f2 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35003 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-05-02Whitenoise: Version lockBenjamin Copeland
We need version lock whitenoise due to python2 support. :( Change-Id: Ibb1edd633b8b0447430c23fc4bd64eb88bc83e25 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34511 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-03-06Whitelist: Add aarch64-08Benjamin Copeland
Change-Id: I8b5a67101e503eca96f22647df6900232da024a3 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34487 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2020-03-05Revert "llp: make delete for link_latest aware of bucket versioning"Kelley Spoon
This reverts commit 526f40cb9bc4062f9272077a53642ed27fdac18c. Reason for revert: This change is causing a 403 from AWS s3 API when users attempt to run link_latest. It's possible there is a permissions problem on the credentials being used. Change-Id: I9ace60d0fcde998a05b7ae74c43d4c76cca42d4d Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34491 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2020-02-26Whitelist: Add armhf-01Benjamin Copeland
Change-Id: Ic8b77933e35bea2c5c723b6010203a13c0242910 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34378 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2020-01-22llp: make delete for link_latest aware of bucket versioningKelley Spoon
Currently LLP will call Bucket.list() on a bucket to generate a list of keys to be deleted from the target directory of the link_latest() API function. This can cause a problem if the underlying bucket is using S3 versioning as the S3.Key information returned by Bucket.list() does not include version information. S3 will interpret this as a request to insert deletemarkers at the top of all keys in the list rather than actually removing them, and this can dramatically increase the object count of the bucket. This change will check to see if a bucket is using versioning, and if so use the Bucket.list_versions() call in order to make sure all S3 objects in the target directory are removed. Change-Id: Ifb14aaa7b4166c83741abcd98339d36e72b5ee5c Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33579 Reviewed-by: Riku Voipio <riku.voipio@linaro.org> Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2019-12-12s3_flatten: fix prefix defaultKelley Spoon
The default prefix was set to "snapshots/aarch64laptops" as part of testing out the script. Let's set that properly to "snapshots/" Change-Id: I84521c4a43497a1911eb4df743a35d57e8062740 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33581 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2019-11-27s3_flatten: break bucket listing up by delimitersKelley Spoon
Boto's bucket.list_version() function has the option to specify a delimiter for the key names that are returned. Delimiters will treat any occurance of the character as if it was a new directory and will not "descend" into it. This change uses delimiters to traverse through all s3 file keys in the given --prefix, and will clean up the objects table before descending into a subdirectory in order to prevent wasting memory storing s3 key data for objects that are no longer going to be accessed. Change-Id: I294937c99c60e89a87439789b9031f50e2e5cefc Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33359 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2019-11-12s3_flatten: add s3_flatten scriptKelley Spoon
This script is intended to replace the s3_purge command. It simplifies the logic by breaking the clean up process into 3 parts: 1 - "flatten" all file objects to have no more than 1 version object and optionally 1 deletemarker regardless of prefix 2 - delete any expired files from prefixes that have not been excluded via S3_PURGE_EXCLUDES 3 - place deletemarkers on files that are older than MARK_DAYS that have not been excluded via S3_PURGE_EXCLUDES The other difference with this script is that it attempts to determine what objects to delete as it iterates through the versioned bucket listing, and will collect multiple delete operations into a single multidelete request. Change-Id: I25a227f574504bb4e828f29670aec366e87952c9 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33268 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
2019-10-29Docker: Fix broken production commandBenjamin Copeland
We need to mount the secrets.py file for the "S3" mode to work. Change-Id: I78817f961200f8e171d6f8587625b80473f5ca19 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33094 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
2019-10-15s3_purge: rewrite to use Bucket.list_versions()Kelley Spoon
The current s3_purge script uses the Bucket.list() method to get a listing of all s3 objects to work on. The list() method unfortunately filters out files that have been marked for deleted, and this creates a race condition where s3_purge is unable to see the files in the list that it needs to delete in a --forcedelete run. This change uses the list_versions() method to get a list of everything. It groups keys by filename, and uses the first key in the list (the lastest version of the file) to determine when deciding what action to take. This should allow for the script to run slightly faster as it process the files as all of its keys come in instead of waiting for the list() method to complete. This change drops the needs for the --clean-releases option as the method for deleting files is working on the s3 object's name and the scope of the operation should have been limited by the --prefix option already (default 'snapshots/'). Change-Id: If30f0dd9c7a6155442ac87bf44ef2f609802bf3c Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33011 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>