1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
|
Linaro downloads license protection
===================================
Linaro builds sometimes contain "binary blobs"—pieces of binary-only code
which enable extra features like accelerated graphics or multimedia.
These pieces are distributed under a separate license, and downloading
images or collections containing them requires some sort of license
protection.
This code provides such license protection on the hosting web server:
other parts of infrastructure need to properly integrate with it (see
eg. android-build.linaro.org section). At the moment, it's implemented as
a click-through license acceptance on the server side with Apache+Django.
To ensure and prove this keeps working, we are also using automated tests
with this code.
Setup
-----
This tree is the base for snapshots.linaro.org and releases.linaro.org www
roots as served by Apache. It uses Django and mod_xsendfile.
Currently, if BUILD-INFO.txt is found in a directory it is parsed and
information from that file is used to apply appropriate protection to the
requested file, otherwise old-style EULA handling is used: if special EULA
file is found then it is used for selecting license and theme for requested
file, otherwise all directories/files containing either 'origen' or 'snowball'
in the URL path are protected with appropriate license (Samsung or ST-E)
click-through.
Dependencies
............
libapache2-mod-xsendfile >= 0.10
libapache2-mod-python
python-django >= 1.3.1
python-django-openid-auth
Testing:
python 2.7, testrepository (>=0.0.6), python-html2text,
python-subunit, python-beautifulsoup, python-mock
Deployment steps
................
- Checkout trunk branch into /srv/releases.linaro.org/linaro-license-protection
- Create directory for database: /srv/releases.linaro.org/db
- Create database, from /srv/releases.linaro.org/linaro-license-protection run
python ./manage.py syncdb --noinput
- Create symlink for the application in /srv/snapshots.linaro.org:
ln -s /srv/releases.linaro.org/linaro-license-protection \
/srv/snapshots.linaro.org/linaro-license-protection
- Create directory for database: /srv/snapshots.linaro.org/db
- Create database, from /srv/snapshots.linaro.org/linaro-license-protection run
python ./manage.py syncdb --noinput
- Put apache2 configs from branch lp:~linaro-infrastructure/linaro-license-protection/apache-configs
in /etc/apache2/sites-available
- Enable sites: a2ensite releases.linaro.org snapshots.linaro.org
Technical details
-----------------
releases.linaro.org
...................
Root directory for releases.linaro.org includes downloadable artifacts served
by Apache and Django application.
It allows a few hard-coded hosts to bypass the click-through protection,
by their IPs:
* android-build.linaro.org (50.17.250.69)
* validation.linaro.org (213.123.120.124)
Currently hosted on mombin.canonical.com.
snapshots.linaro.org
....................
Same basic set-up as releases.linaro.org.
Further, to allow android-build.linaro.org to push to snapshots.linaro.org
www area directly, we set-up two different users on the system with SSH keys:
* linaro-android-build-publish:
chrooted to /srv/snapshots.linaro.org/www/android/ and allows sftp
access to push files over; home directory
* linaro-android-build-publish-trigger:
limited to executing only a single command through
/etc/ssh/user-authorized-keys/linaro-android-build-publish-trigger
command="/home/android-build-linaro-trigger/scripts
/trigger-android-build-linaro.sh ${SSH_ORIGINAL_COMMAND#* }"
(this passes the arguments received from the remote end as well)
This script does a sudo to 'android-build-linaro' and then runs
/home/android-build-linaro/scripts/jenkins-post-sftp.sh
script which moves files from /android/.tmp/ into appropriate
build directory. It expects "build_name/build_number" as command
line parameters.
android-build.linaro.org
........................
Runs Jenkins and uses SFTP plugin to access the above two users. Private
keys live in
/home/ubuntu/snapshots-sync2/linaro-android-build-publish —
linaro-android-build-publish
/home/ubuntu/snapshots-sync2/linaro-android-build-publish-trigger —
linaro-android-build-publish-trigger
To ensure serialization of steps, and allow LAVA submission, these happen as
build steps, and not as publishing steps.
Build-Info support
------------------
BUILD-INFO.txt format
.....................
Paragraphs
..........
* Header paragraph
* Files paragraph
Header paragraph (once)
.......................
The following fields may be present in a header paragraph:
* Format-Version: (required)
Version of the BUILD-INFO format.
Files paragraph (repeatable)
............................
The following fields may be present in a files paragraph:
* Files-Pattern: (required, must be the first field in the paragraph)
Comma separated filename patterns that are specified using a simplified
shell glob syntax. Will be used to identify protected files.
* Build-Name: (optional)
Would be unique over all the builds of the same type. For now used as
a placeholder and will be ignored. To be added later by build services?
* License-Type: (required)
open - Open builds. No license page is displayed.
protected - EULA protected builds. If 'OpenID-Launchpad-Teams' is defined
then OpenID protection is used, otherwise simple Accept/Decline license
page is displayed before accessing protected files.
* Theme: (required only if License-Type is 'protected')
Acceptable values are: stericsson, samsung.
Theme name for selecting proper theming on download and license pages.
* OpenID-Launchpad-Teams: (optional)
LP team names, members of which are allowed to access protected files. No
OpenID protection if absent.
* Collect-User-Data: (optional)
Acceptable values are: yes, no.
Defaults to 'no' if not present. If the field is set to 'yes' then
Name and E-Mail (some other fields?) fields are asked to be filled. If it
will be needed in the future it could be expanded with
'Collect-User-Data-Fields' field later or any other approach.
* License-Text: (required only if License-Type is 'protected')
EULA full text included in BUILD-INFO.txt
- If line begins with space character ' ' it is treated as a part of last found
variable which turns as multiline variable and is appended to it.
- Field names are case insensitive.
- Fields order doesn't matter, except 'Files-Pattern'
BUILD-INFO.txt example:
.......................
Format-Version: 0.1
Files-Pattern: *.img, *.tar.bz2
Build-Name: landing-snowball
License-Type: protected
Theme: stericsson
License-Text: EULA full text displayed to user.
You should accept to download file.
Files-Pattern: *.txt
Bulid-Name: landing-snowball
License-Type: open
License protection script takes information from the BUILD-INFO.txt file placed
in the same directory as artifacts, finds 'Files-Pattern' block corresponding
to requested file and applies rules from that block to download procedure.
If no BUILD-INFO.txt is found it falls back to per-file/per-directory EULA
protection.
Tests
-----
Testing infrastructure is based on 'testrepository' and requires at least
Python and Apache2:
To run tests for publisher:
$ testr init
$ testr run
To run tests for Django application:
$ python manage.py test
Test plans
..........
To run the test plans from testplans/ subdirectory against
snapshots.linaro.org and releases.linaro.org, execute the following:
$ testr init
$ testr run testplans.test_suite
These tests require an internet connection.
|
