Age | Commit message (Collapse) | Author |
|
In ldap, the 'linaro' group was deprecated and
replaced by the 'everybody-flat' group.
Since it was hardcoded here, we need to update
the group name in order to allow linaro
employees access to protected urls.
Change-Id: I6b340728ff26faa64a29fd2384d9ebe321dcff36
Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org>
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/44724
|
|
Since the login page is using the view created
by django.contrib.auth, we need to ensure that
the "site_name" context variable is set.
Change-Id: I5e5160ffd54383718b9942d06871611cb038e818
Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org>
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40739
|
|
When we check for group membership in LDAP, we should
log the fact it doesn't exist, but proceed on without
an missing key error.
Change-Id: I2bbee729f8cf144624c8c01af7b76f49f2b8be35
Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org>
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40740
|
|
We're using python-ldap=3.3.1 now, which must
be build from source since the latest supported
version is 2.2.4.
Let's apt-get the build requirements in the
unit-test.sh script so Jenkins can be useful
instead of always erroring on the ldap build failure.
Change-Id: I582d2d1480e185b705153f50367ea51e4b61d953
Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org>
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40741
|
|
<td valign="top"> is obsolete. Same with <td align="center">
Change-Id: I35c0f57585a5550e40b1c4ae52e0fc561a2bc9b5
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40533
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
- DOCTYPE is mandatory for HTML 5
- type/language parameters for <script> <style> are obsolete
Change-Id: I7a01b650eb723b3fb5437826decdbf5cf2ac55b4
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40534
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
There is a bug in the ldap group authorization code
where we use the full django username for authentication
(which is the full email), but only the UID (first.lastname)
is stored in the group membership table.
We should also take this time to just try to look up
the UID in the groups table instead of trying to build
a list of group memberships for the user and compare
that to the required groups.
Change-Id: I41209fb8745a6225f3e7344910dc89c19d336a76
Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org>
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40448
|
|
This change adds in the requires AUTH_LDAP settings to
allow llp to use auth_ldap_backend. The LDAP bind password
is set in the host's secrets.
Change-Id: Ifcf8ef6d79ae8ba7d5e904a696b60a8bb9dd9eae
Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org>
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40443
|
|
In order to remove the final piece of Crowd usage in our
infrastructure, let's migrate LLP to using ldap for both
authentication and group look ups.
We also need to update the views, urls, and tests to
no longer rely upon crowd/openid and instead mock up
the LDAP lookups.
Change-Id: I254d7bbffff97ffa8ebe5f5be72e763e77a6c45d
Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org>
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40275
|
|
This change adds in a lambda function and the
terraform to upload and enable it that will check
if an uploaded (created) file is in a protected
path. If not, the S3 object will be tagged for
eventual deletion via a lifecycle rule when it's
lifespan expires.
Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org>
Change-Id: I34ade1f38311a269c479d6373495a789de96ab15
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40186
|
|
By adding 'static' to the gitignore file, the pattern match was
also applied to 'license-protected-downloads/static' and the
missing admin, jquery, and css files were left out of the
repo.
Let's rename the top level directory where the collected files are
to be put to 'staticroot' so that it can safely be added to
the gitignore file and restore the missing files.
Change-Id: Ie051df5c26ab83df96af5a8453648b0db9f3cbcd
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40226
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Fix staticfiles so that they are correctly generated by
'collectstatic' regardless of where or how this app
is deployed.
In this change, let's concentrate all static files in
'license_protected_downloads/static' so that we can
clear out STATIC_ROOT ('static') and rebuild it reliably
with 'collectstatic'.
We also restore the contents of the
'license_protected_downloads/static' directory, and update
the unit-test.sh script to call 'collectstatic' on a freshly
installed STATIC_ROOT before it runs.
Signed-off-by: Kelley Spoon <kelley.spoon@linaro.org>
Change-Id: Ice720b4794a48540c3753f4a6efba027cbc29040
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40206
|
|
This change adds in code to supress the beautifulsoup warning from the
application log in addition to the previous commit that removes it from
the user space commands.
Also, the STATICFILES_DIRS is incorrectly set to be the same as STATIC_ROOT.
Instead let's point it at PROJECT_ROOT to make sure the right files get
picked up and we avoid any nasty loops when running collectstatic
Change-Id: I5a2c592cfd3c625053e785c62863e6f6ff0af59f
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40044
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
BeautifulSoup produces and UserWarning when imported reminding the user
that the version we're using is EOL and that we should be upgrading it
soon.
This warning is producing output that is causing crontab to email when
there are no errors or problems to report. Let's silence it until
we can finally update the software.
Change-Id: Ic45819c0f22ff91b2f2b5138433d69428cbeec95
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/40042
Reviewed-by: Riku Voipio <riku.voipio@linaro.org>
|
|
We should add allowed_hosts.txt and local_settings.py to gitignore
as they are created when run in production and we get
warnings about untracked files on a git status.
Change-Id: I2944817848fff25bf1e4e6594d4a4f9868dcbb1f
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38934
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Let's use the static() helper in our templates to correctly
generate a URL for our static files. We also need to add
our STATIC_URL to the urlpatterns list, and straighten out
the settings to ensure the static root is found and loaded
correctly.
Change-Id: I933bcf9c9eeec7300dc0f3935b07179446f7b621
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38933
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Somehow the css and js directories got moved out of the static
directory and up into the root of the project, and then were
committed with the minified file names. Let's clean this junk
out of the way to it doesn't continue to confuse the issue.
Change-Id: I234cebb2ba6f743160f8261ed863b17134c0e305
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38932
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Previously the cron job to process the download report CSV file
was choking because the timestamp in the CSV didn't have the
fractional milisecond field on it. This change will catch
the ValueError that gets raised, and tries an alternate timestamp
so that processing can continue.
Change-Id: Ied59f40eebc5e2be692aba5f87872f7e652e0e9d
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38859
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
A while back a request was made to include a 'Test Plan'
and 'Test Report' tab for some of our projects that would
feature the contents of the respective files. Apparently
the change was made in place on the server, but was never
migrated into the LLP repo.
Let's get caught up.
Change-Id: I5e49fcd41211966065edbc01191ad607d84e00c5
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38759
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Let's fix the gitignore for the static files directory as git
seems to think a few png's are now version controlled.
Change-Id: I3470b508b58f0d6ef6fb383e1ce95c7178305c54
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38760
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Change-Id: I2546c45b4eda485d453bf8201ff65ee34e0d0e56
Signed-off-by: Benjamin Copeland <ben.copeland@linaro.org>
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38758
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
testdata.lo was failing to show static files (ie, icons in pages). Let's
add in the STATIC_ROOT to STATIC_DIRS explicitly in order to
ensure that the files are correctly found and served.
Change-Id: Iffb8b489cd421ac3fb8344e0c51d542e3aa7118f
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/38333
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
Change-Id: I596f3cdea9c83148568031007c740c84b2708fae
Signed-off-by: Benjamin Copeland <ben.copeland@linaro.org>
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/37509
|
|
Currently, our process_report script assumes that a Download occured
on the day the process_report script was run. This isn't correct,
so we need to keep track of the actual datetime when the download
occurred in the CSV as an extra field.
This attempts to handle it in a way that is backwards compatible with
previous CSV files that don't include a timestamp column.
Change-Id: I5249c434a2777f09fb1e2010b812a91255517b07
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/36874
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
With the original code, the re match is:
('https://snapshots.linaro.org/openembedded/schneider/linaro-dunfell-4.19/soca9', 'latest/dip', 'MD5SUMS.txt')
Correct it by matching only exactly to "latest"
('https://snapshots.linaro.org/openembedded/schneider/linaro-dunfell-4.19/soca9', 'latest', 'dip/MD5SUMS.txt')
Hence reconstructing the url will succeed again
fixes LSS-1804
Change-Id: I73bb5b6c6655e0f3590facc3e4302dc7aadb9294
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/36676
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
Fixes direct reference like:
http://localhost:8080/components/kernel/linaro-edk2/latest/x.txt
and thus also wildcards on latest:
http://localhost:8080/components/kernel/linaro-edk2/latest/startup-qemu-*.nsh
Signed-off-by: Riku Voipio <riku.voipio@linaro.org>
Change-Id: Ia95dbaa3f715dfe604f01b43776dbba9f26147ea
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/36170
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
1. requirements pin auth plugin version
The new version requires Python 3
2. Revert the S3 testsuite changes. These need to
implemented properly in it's own changeset
3. views.py: don't call s3_replace_latest if we
are not configured to use S3
Signed-off-by: Riku Voipio <riku.voipio@linaro.org>
Change-Id: I7723155a1dbfc242857618e64ecf5852ea4ed22c
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/36180
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
The previous commit would truncate anything after "/latest" if an
.s3_link_from file was found, which made it impossible to reference
a file through the "latest" version directly.
This change also includes updates to the default settings.py to
include S3 settings that will trigger API v3 testing. Enabling
the v3 unit-tests exposed some problems in our tests, so those have
been skipped with a "FIXME" tag until they can be addressed.
Change-Id: I1c801f86bbf11bc281a44d755c5767febc98f507
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/36054
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Instead of copying all items in link-latest operation,
simply parse the latest link on fly when user is browsing
Signed-off-by: Riku Voipio <riku.voipio@linaro.org>
Change-Id: Ifeb2b9e95e1ddb0d40ec2d066ca5b1edb4fb1b53
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35939
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
This adds in x86_64-15 and x86_64-swarm to the list of machines
allowed to post/get to the server with no authentication.
Change-Id: Ia40873c7bebecdae31042d673898c8d40f748952
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35796
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Somehow missed importing sleep() from time. Let's do that
in case the httplib error ever occurs again.
Change-Id: Ifb6a039a54433111c2e278b2666f906405baa424
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35430
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
There was a missing format character for an error log message
that was triggering a valueerror exception.
Change-Id: Iea0569991d3914c256c269bdc7bc14a8f25b2ab2
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35367
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
Turns out if you use a while loop to do retries, you should
also set the exit for a success.
Change-Id: I593c717459e259487d804bd3b616b5b3049ff462
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35188
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
It turns out that boto will attempt to lazily load details for objects
in a bucket. As a result, it is not sufficient to simply wrap the
list_objects() calls. In order to prevent even more complexity
by attempting to predict where try/excepts retries are needed,
let's just wrap the handle_bucket() method itself and use the
wrapper to safely clean up objects.
Change-Id: I257f9390dbb6f59e83511e50594824306ad9baf8
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35184
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
It turns out that this script can occasionally error out
if the AWS API doesn't respond with proper HTTP. When
this happens, the error isn't caught and the script ends
prematurely and can leave large amounts of unnecessary
data in the bucket.
This wraps S3 bucket operations (specifically delete_keys
and list_objectS) in a try statement that will attempt
up to 3 retries after sleeping for 30 seconds.
Change-Id: I3f4c945937b20635166cad093630bfbca9026a78
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35063
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
Due to reorganizing the docker images to have their
own independent mount points, the path to the static
files has changed.
Change-Id: I1e406c3bdecb2b7720a78c6bb307a394a9bc0fdd
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35007
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
WhiteNoise 5.0 needs to be downgraded and locked to
4.1.4 for backwards compat as 5.0 is using the
py3 urllib API.
Change-Id: I654055a210f63192a8c069787eb1169e02c0b25f
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35005
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
This reverts commit 3d97c57b6e6710d6931e92495eb026d4b99849b0.
Reason for revert: gunicorn is handled by OS package manager and should not be done here.
Change-Id: I5bd46ff3d1282e4771a71da874bc1bba55246f9d
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34498
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
This reverts commit c1633ecc22806843950cf985c5dfc837a46b5593.
Reason for revert: gevent is handled by the OS package manager, shouldn't be installed here.
Change-Id: I0a860b65cde24610e5a229fd02386e7e1c3acb3b
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34497
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
By adding in gunicorn, gevent was picked up as a dependency.
Change-Id: I984e1c70563349107df8b1b550fdb870b2bc6e8b
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35004
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Need to version lock to 19.7.1 due to python2 support. 19.9.0
explicitly calls python3, which can't find django installed
for python2.7
Change-Id: Ic1205c0b651afcc422f9bcc1798b148a09f4e2f2
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/35003
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
We need version lock whitenoise due to python2 support. :(
Change-Id: Ibb1edd633b8b0447430c23fc4bd64eb88bc83e25
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34511
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Change-Id: I8b5a67101e503eca96f22647df6900232da024a3
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34487
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
This reverts commit 526f40cb9bc4062f9272077a53642ed27fdac18c.
Reason for revert: This change is causing a 403 from AWS s3 API when users attempt to run link_latest. It's possible there is a permissions problem on the credentials being used.
Change-Id: I9ace60d0fcde998a05b7ae74c43d4c76cca42d4d
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34491
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Change-Id: Ic8b77933e35bea2c5c723b6010203a13c0242910
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/34378
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
Currently LLP will call Bucket.list() on a bucket to
generate a list of keys to be deleted from the target
directory of the link_latest() API function.
This can cause a problem if the underlying bucket is
using S3 versioning as the S3.Key information returned
by Bucket.list() does not include version information.
S3 will interpret this as a request to insert deletemarkers
at the top of all keys in the list rather than actually
removing them, and this can dramatically increase the object
count of the bucket.
This change will check to see if a bucket is using versioning,
and if so use the Bucket.list_versions() call in order to make
sure all S3 objects in the target directory are removed.
Change-Id: Ifb14aaa7b4166c83741abcd98339d36e72b5ee5c
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33579
Reviewed-by: Riku Voipio <riku.voipio@linaro.org>
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
The default prefix was set to "snapshots/aarch64laptops" as part
of testing out the script. Let's set that properly to "snapshots/"
Change-Id: I84521c4a43497a1911eb4df743a35d57e8062740
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33581
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|
|
Boto's bucket.list_version() function has the option to specify
a delimiter for the key names that are returned. Delimiters
will treat any occurance of the character as if it was a new
directory and will not "descend" into it.
This change uses delimiters to traverse through all s3 file keys
in the given --prefix, and will clean up the objects table before
descending into a subdirectory in order to prevent wasting memory
storing s3 key data for objects that are no longer going to be
accessed.
Change-Id: I294937c99c60e89a87439789b9031f50e2e5cefc
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33359
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
This script is intended to replace the s3_purge command.
It simplifies the logic by breaking the clean up process
into 3 parts:
1 - "flatten" all file objects to have no more than 1
version object and optionally 1 deletemarker regardless
of prefix
2 - delete any expired files from prefixes that have not been
excluded via S3_PURGE_EXCLUDES
3 - place deletemarkers on files that are older than MARK_DAYS
that have not been excluded via S3_PURGE_EXCLUDES
The other difference with this script is that it attempts to
determine what objects to delete as it iterates through the
versioned bucket listing, and will collect multiple delete
operations into a single multidelete request.
Change-Id: I25a227f574504bb4e828f29670aec366e87952c9
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33268
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
|
|
We need to mount the secrets.py file for the "S3" mode to work.
Change-Id: I78817f961200f8e171d6f8587625b80473f5ca19
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33094
Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>
|